Physical Neural Networks with Hybrid Activation Functions in Deep Learning

Sheng-Chung Chen^1*, Zhen-Wei Sun², Shih-Chuan Hung², Sheng-Ting Huang², Der-Hsien Lien²

¹前瞻半導體研究所, 國立陽明交通大學, 新竹市, Taiwan
²電子所, 國立陽明交通大學, 新竹市, Taiwan

* Presenter:Sheng-Chung Chen, email:sheng8705@gmail.com

Deep learning is increasingly integrated into daily life, powering technologies from image recognition to weather prediction. However, its development is constrained by the substantial energy required for computation. To address this challenge, we propose a physical neural network featuring tunable activation functions to enhance both training and inference efficiency. This physical neural network utilizes transistors as activation functions, leveraging their nonlinear current-voltage characteristics to enhance the accuracy of deep learning models. By modulating the drain voltage of the transistor, we can adjust its nonlinear behavior, allowing it to emulate common activation functions such as ReLU, Sigmoid, and Softmax. This flexibility enables the use of various activation functions across different hidden layers, which optimizes the adaptability of the physical neural network and minimizes the epoch of training for deep learning. Furthermore, since the activation functions are implemented through transistors rather than GPUs, the energy consumption of the physical neural network is reduced, resulting in a more energy-efficient approach to deep learning. In conclusion, deep learning utilizing physical neural networks is more potential than conventional deep learning for applications in edge computing, autonomous vehicles, and robotics.

Keywords: Physical Neural Network, Activation Function, Deep Learning