Programmable tanh-, ELU-, sigmoid-, and Sin-based nonlinear activation functions for neuromorphic photonics
- Author
- Christos Pappas, Stefanos Kovaios, Miltiadis Moralis-Pegios, Apostolos Tsakyridis, George Giamougiannis, Manos Kirtas, Joris Van Kerrebrouck (UGent) , Gertjan Coudyzer (UGent) , Xin Yin (UGent) , Nikolaos Passalis, Anastasios Tefas and Nikos Pleros
- Organization
- Abstract
- We demonstrate a programmable analog opto-electronic (OE) circuit that can be configured to provide a range of nonlinear activation functions for incoherent neuromorphic photonic circuits at up to 10 Gbaud line-rates. We present a set of well-known activation functions that are typically used to train DL models including tanh-, sigmoid-, ReLU- and inverted ReLU-like activations, introducing also a series of novel photonic nonlinear functions that are referred to as Rectified Sine Squared (ReSin), Sine Squared with Exponential tail (ExpSin) and Double Sine Squared. Experimental validation for all these activation functions is performed at 10 Gbaud operation. The ability of the mathematically modelled photonic activation functions to train Deep Neural Networks (DNNs) has been verified through their employment in Deep Learning (DL) models for MNIST and CIFAR10 classification purposes, comparing their performance against corresponding NNs that utilize an ideal ReLU activation function.
- Keywords
- NETWORKS, Nonlinear activation function, opto-electronic activation function, opto-electro-optic activation function, programmable analog device
Downloads
-
(...).pdf
- full text (Published version)
- |
- UGent only
- |
- |
- 1.90 MB
Citation
Please use this url to cite or link to this publication: http://hdl.handle.net/1854/LU-01H665715H91QCBBKJVKGS89W5
- MLA
- Pappas, Christos, et al. “Programmable Tanh-, ELU-, Sigmoid-, and Sin-Based Nonlinear Activation Functions for Neuromorphic Photonics.” IEEE JOURNAL OF SELECTED TOPICS IN QUANTUM ELECTRONICS, vol. 29, no. 6, 2023, doi:10.1109/JSTQE.2023.3277118.
- APA
- Pappas, C., Kovaios, S., Moralis-Pegios, M., Tsakyridis, A., Giamougiannis, G., Kirtas, M., … Pleros, N. (2023). Programmable tanh-, ELU-, sigmoid-, and Sin-based nonlinear activation functions for neuromorphic photonics. IEEE JOURNAL OF SELECTED TOPICS IN QUANTUM ELECTRONICS, 29(6). https://doi.org/10.1109/JSTQE.2023.3277118
- Chicago author-date
- Pappas, Christos, Stefanos Kovaios, Miltiadis Moralis-Pegios, Apostolos Tsakyridis, George Giamougiannis, Manos Kirtas, Joris Van Kerrebrouck, et al. 2023. “Programmable Tanh-, ELU-, Sigmoid-, and Sin-Based Nonlinear Activation Functions for Neuromorphic Photonics.” IEEE JOURNAL OF SELECTED TOPICS IN QUANTUM ELECTRONICS 29 (6). https://doi.org/10.1109/JSTQE.2023.3277118.
- Chicago author-date (all authors)
- Pappas, Christos, Stefanos Kovaios, Miltiadis Moralis-Pegios, Apostolos Tsakyridis, George Giamougiannis, Manos Kirtas, Joris Van Kerrebrouck, Gertjan Coudyzer, Xin Yin, Nikolaos Passalis, Anastasios Tefas, and Nikos Pleros. 2023. “Programmable Tanh-, ELU-, Sigmoid-, and Sin-Based Nonlinear Activation Functions for Neuromorphic Photonics.” IEEE JOURNAL OF SELECTED TOPICS IN QUANTUM ELECTRONICS 29 (6). doi:10.1109/JSTQE.2023.3277118.
- Vancouver
- 1.Pappas C, Kovaios S, Moralis-Pegios M, Tsakyridis A, Giamougiannis G, Kirtas M, et al. Programmable tanh-, ELU-, sigmoid-, and Sin-based nonlinear activation functions for neuromorphic photonics. IEEE JOURNAL OF SELECTED TOPICS IN QUANTUM ELECTRONICS. 2023;29(6).
- IEEE
- [1]C. Pappas et al., “Programmable tanh-, ELU-, sigmoid-, and Sin-based nonlinear activation functions for neuromorphic photonics,” IEEE JOURNAL OF SELECTED TOPICS IN QUANTUM ELECTRONICS, vol. 29, no. 6, 2023.
@article{01H665715H91QCBBKJVKGS89W5,
abstract = {{We demonstrate a programmable analog opto-electronic (OE) circuit that can be configured to provide a range of nonlinear activation functions for incoherent neuromorphic photonic circuits at up to 10 Gbaud line-rates. We present a set of well-known activation functions that are typically used to train DL models including tanh-, sigmoid-, ReLU- and inverted ReLU-like activations, introducing also a series of novel photonic nonlinear functions that are referred to as Rectified Sine Squared (ReSin), Sine Squared with Exponential tail (ExpSin) and Double Sine Squared. Experimental validation for all these activation functions is performed at 10 Gbaud operation. The ability of the mathematically modelled photonic activation functions to train Deep Neural Networks (DNNs) has been verified through their employment in Deep Learning (DL) models for MNIST and CIFAR10 classification purposes, comparing their performance against corresponding NNs that utilize an ideal ReLU activation function.}},
articleno = {{6101210}},
author = {{Pappas, Christos and Kovaios, Stefanos and Moralis-Pegios, Miltiadis and Tsakyridis, Apostolos and Giamougiannis, George and Kirtas, Manos and Van Kerrebrouck, Joris and Coudyzer, Gertjan and Yin, Xin and Passalis, Nikolaos and Tefas, Anastasios and Pleros, Nikos}},
issn = {{1077-260X}},
journal = {{IEEE JOURNAL OF SELECTED TOPICS IN QUANTUM ELECTRONICS}},
keywords = {{NETWORKS,Nonlinear activation function,opto-electronic activation function,opto-electro-optic activation function,programmable analog device}},
language = {{eng}},
number = {{6}},
pages = {{10}},
title = {{Programmable tanh-, ELU-, sigmoid-, and Sin-based nonlinear activation functions for neuromorphic photonics}},
url = {{http://doi.org/10.1109/JSTQE.2023.3277118}},
volume = {{29}},
year = {{2023}},
}
- Altmetric
- View in Altmetric
- Web of Science
- Times cited: