Bingzhe Li
David J. Lilja
Abstract
Neural networks are becoming prevalent in many areas, such as pattern recognition and medical diagnosis. Stochastic computing is one potential solution for neural networks implemented in low-power back-end devices such as solar-powered devices and Internet of Things (IoT) devices. In this article, we investigate a new architecture of stochastic neural networks with a hardware-oriented approximate activation function. The newly proposed approximate activation function can be hidden in the proposed architecture and thus reduce the whole hardware cost. Additionally, to further reduce the hardware cost of the stochastic implementation, a new hybrid stochastic multiplier is proposed. It contains OR gates and a binary parallel counter, which aims to reduce the number of inputs of the binary parallel counter. The experimental results indicate the newly proposed approximate architecture without hybrid stochastic multipliers achieves more than 25%, 60%, and 3x reduction compared to previous stochastic neural networks, and more than 30x, 30x, and 52% reduction compared to conventional binary neural networks, in terms of area, power, and energy, respectively, while maintaining the similar error rates compared to the conventional neural networks. Furthermore, the stochastic implementation with hybrid stochastic multipliers further reduces area about 18% to 80%, power from 15% to 113.1%, and energy about 15% to 131%, respectively.
- Armin Alaghi and John P. Hayes. 2013. Survey of stochastic computing. ACM Transactions on Embedded Computing Systems (TECS) 12, 2s (2013), 92. Google Scholar
Digital Library
- Bradley D. Brown and Howard C. Card. 2001. Stochastic neural computation. I. Computational elements. IEEE Transactions on Computers 50, 9 (2001), 891--905. Google ScholarDigital Library
- Ronan Collobert and Samy Bengio. 2004. Links between perceptrons, MLPs and SVMs. In Proceedings of the 21st International Conference on Machine Learning. ACM, 23. Google ScholarDigital Library
- Jeffery A. Dickson, Robert D. McLeod, and H. C. Card. 1993. Stochastic arithmetic implementations of neural networks with in situ learning. In Proceedings of the 1993 IEEE International Conference on Neural Networks. IEEE, 711--716.Google Scholar
- Brian R. Gaines et al. 1969. Stochastic computing systems. Advances in Information Systems Science 2, 2 (1969), 37--172.Google ScholarCross Ref
- Geoffrey E. Hinton and Ruslan R. Salakhutdinov. 2006. Reducing the dimensionality of data with neural networks. Science 313, 5786 (2006), 504--507.Google Scholar
- Yuan Ji, Feng Ran, Cong Ma, and David J. Lilja. 2015. A hardware implementation of a radial basis function neural network using stochastic logic. In Proceedings of the 2015 Design, Automation 8 Test in Europe Conference 8 Exhibition. EDA Consortium, 880--883. Google ScholarDigital Library
- Seul Jung and Sung su Kim. 2007. Hardware implementation of a real-time neural network controller with a DSP and an FPGA for nonlinear systems. IEEE Transactions on Industrial Electronics 54, 1 (2007), 265--271.Google ScholarCross Ref
- Murat Karabatak and M. Cevdet Ince. 2009. An expert system for detection of breast cancer based on association rules and neural network. Expert Systems with Applications 36, 2 (2009), 3465--3469. Google ScholarDigital Library
- Kyounghoon Kim, Jungki Kim, Joonsang Yu, Jungwoo Seo, Jongeun Lee, and Kiyoung Choi. 2016. Dynamic energy-accuracy trade-off using stochastic computing in deep neural networks. In Proceedings of the 53rd Annual Design Automation Conference. ACM, 124. Google ScholarDigital Library
- Kyounghoon Kim, Jongeun Lee, and Kiyoung Choi. 2016. An energy-efficient random number generator for stochastic circuits. In Proceedings of the 2016 21st Asia and South Pacific Design Automation Conference (ASP-DAC). IEEE, 256--261.Google ScholarDigital Library
- Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-based learning applied to document recognition. Proceedings of the IEEE 86, 11 (1998), 2278--2324.Google ScholarCross Ref
- Yann LeCun and Corinna Cortes. 2010. MNIST handwritten digit database. AT8T Labs {Online}. Retrieved from http://yann.lecun.com/exdb/mnist.Google Scholar
- Vincent T. Lee, Armin Alaghi, John P. Hayes, Visvesh Sathe, and Luis Ceze. 2017. Energy-efficient hybrid stochastic-binary neural networks for near-sensor computing. In Proceedings of the 2017 Design, Automation 8 Test in Europe Conference 8 Exhibition (DATE). IEEE, 13--18. Google ScholarDigital Library
- Bingzhe Li, M. Hassan Najafi, and David J. Lilja. 2015. An FPGA implementation of a restricted Boltzmann machine classifier using stochastic bit streams. In 2015 IEEE 26th International Conference on Application-specific Systems, Architectures and Processors (ASAP). IEEE, 68--69.Google Scholar
- Bingzhe Li, M. Hassan Najafi, and David J. Lilja. 2016. Using stochastic computing to reduce the hardware requirements for a restricted Boltzmann machine classifier. In Proceedings of the 2016 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays. ACM, 36--41. Google ScholarDigital Library
- B. Li, M. H. Najafi, B. Yuan, and D. J. Lilja. 2018. Quantized neural networks with new stochastic multipliers. In Proceedings of the 2018 19th International Symposium on Quality Electronic Design (ISQED). 376--382.Google Scholar
- Bingzhe Li, Yaobin Qin, Bo Yuan, and David J. Lilja. 2017. Neural network classifiers using stochastic computing with a hardware-oriented approximate activation function. In Proceedings of the IEEE International Conference on Computer Design (ICCD’17). IEEE, 97--104.Google Scholar
- Ji Li, Zihao Yuan, Zhe Li, Caiwen Ding, Ao Ren, Qinru Qiu, Jeffrey Draper, and Yanzhi Wang. 2017. Hardware-driven nonlinear activation for stochastic computing based deep convolutional neural networks. arXiv preprint arXiv:1703.04135.Google Scholar
- Peng Li, David J. Lilja, Weikang Qian, Kia Bazargan, and Marc D. Riedel. 2014. Computation on stochastic bit streams digital image processing case studies. IEEE Transactions on Very Large Scale Integration (VLSI) Systems 22, 3 (2014), 449--462. Google ScholarDigital Library
- Peng Li, David J. Lilja, Weikang Qian, Marc D. Riedel, and Kia Bazargan. 2012. Logical computation on stochastic bit streams with linear finite state machines. IEEE Transactions on Computers 63, 6 (2012), 1474--1486. Google ScholarDigital Library
- Peng Li, Weikang Qian, Marc D. Riedel, Kia Bazargan, and David J. Lilja. 2012. The synthesis of linear finite state machine-based stochastic computational elements. In 2012 17th Asia and South Pacific Design Automation Conference (ASP-DAC). IEEE, 757--762.Google Scholar
- Zhe Li, Ao Ren, Ji Li, Qinru Qiu, Yanzhi Wang, and Bo Yuan. 2016. DSCNN: Hardware-oriented optimization for stochastic computing based deep convolutional neural networks. In Proceedings of the IEEE 34th International Conference on Computer Design (ICCD’16). IEEE, 678--681.Google ScholarCross Ref
- Zhe Li, Ao Ren, Ji Li, Qinru Qiu, Bo Yuan, Jeffrey Draper, and Yanzhi Wang. 2017. Structural design optimization for deep convolutional neural networks using stochastic computing. In Proceedings of the 2017 Design, Automation 8 Test in Europe Conference 8 Exhibition (DATE). IEEE, 250--253. Google ScholarDigital Library
- M. Hassan Najafi, Peng Li, David J. Lilja, Weikang Qian, Kia Bazargan, and Marc Riedel. 2017. A reconfigurable architecture with sequential logic-based stochastic computing. ACM Journal on Emerging Technologies in Computing Systems (JETC) 13, 4 (2017), 57. Google ScholarDigital Library
- M. Hassan Najafi, David J. Lilja, Marc Riedel, and Kia Bazargan. 2016. Polysynchronous stochastic circuits. In Proceedings of the 2016 21st Asia and South Pacific Design Automation Conference (ASP-DAC). IEEE, 492--498.Google ScholarDigital Library
- Amos R. Omondi and Jagath Chandana Rajapakse. 2006. FPGA Implementations of Neural Networks. Vol. 365. Springer. Google ScholarDigital Library
- Weikang Qian and Marc D. Riedel. 2010. Synthesizing logical computation on stochastic bit streams. Submitted to Communications of the ACM (2010).Google Scholar
- Weikang Qian, Chen Wang, Peng Li, David J. Lilja, Kia Bazargan, and Marc D. Riedel. 2012. An efficient implementation of numerical integration using logical computation on stochastic bit streams. In Proceedings of the IEEE/ACM International Conference on Computer-Aided Design (ICCAD’12). IEEE, 156--162. Google ScholarDigital Library
- David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. 2016. Mastering the game of Go with deep neural networks and tree search. Nature 529, 7587 (2016), 484--489.Google Scholar
- Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556.Google Scholar
- James E. Stine, Ivan Castellanos, Michael Wood, Jeff Henson, Fred Love, W. Rhett Davis, Paul D. Franzon, Michael Bucher, Sunil Basavarajaiah, Julie Oh, et al. 2007. FreePDK: An open-source variation-aware design kit. In Proceedings of the IEEE International Conference on Microelectronic Systems Education (MSE’07). IEEE, 173--174. Google ScholarDigital Library
- Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015. Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1--9.Google ScholarCross Ref
Index Terms
- Neural Network Classifiers Using a Hardware-Based Approximate Activation Function with a Hybrid Stochastic Multiplier
Recommendations
Low-Cost Stochastic Hybrid Multiplier for Quantized Neural Networks
Special Issue on HALO for Energy-Constrained On-Chip Machine LearningWith increased interests of neural networks, hardware implementations of neural networks have been investigated. Researchers pursue low hardware cost by using different technologies such as stochastic computing (SC) and quantization. More specifically, ...
A hardware implementation of a radial basis function neural network using stochastic logic
DATE '15: Proceedings of the 2015 Design, Automation & Test in Europe Conference & ExhibitionHardware implementations of artificial neural networks typically require significant amounts of hardware resources. This paper proposes a novel radial basis function artificial neural network using stochastic computing elements, which greatly reduces ...
Reconfigurable hardware for neural networks: binary versus stochastic
This paper is focused on hardware implementation of neural networks. We propose a reconfigurable, low-cost and readily available hardware architecture for an artificial neuron. For this purpose, we use field-programmable gate arrays i.e. FPGAs. As the ...
Comments