Accelerating Hopfield network dynamics : beyond synchronous updates and forward Euler
- Author
- Cédric Goemaere (UGent) , Johannes Deleu (UGent) and Thomas Demeester (UGent)
- Organization
- Project
- Abstract
- The Hopfield network serves as a fundamental energy-based model in machine learning, capturing memory retrieval dynamics through an ordinary differential equation (ODE). The model's output, the equilibrium point of the ODE, is traditionally computed via synchronous updates using the forward Euler method. This paper aims to overcome some of the disadvantages of this approach. We propose a conceptual shift, viewing Hopfield networks as instances of Deep Equilibrium Models (DEQs). The DEQ framework not only allows for the use of specialized solvers, but also leads to new insights on an empirical inference technique that we will refer to as 'even-odd splitting'. Our theoretical analysis of the method uncovers a parallelizable asynchronous update scheme, which should converge roughly twice as fast as the conventional synchronous updates. Empirical evaluations validate these findings, showcasing the advantages of both the DEQ framework and even-odd splitting in digitally simulating energy minimization in Hopfield networks. The code is available at https://github.com/cgoemaere/hopdeq.
- Keywords
- Even-odd splitting, Hopfield network, Deep Equilibrium Model, DISCRETE-TIME
Downloads
-
(...).pdf
- full text (Published version)
- |
- UGent only
- |
- |
- 1.56 MB
Citation
Please use this url to cite or link to this publication: http://hdl.handle.net/1854/LU-01JBVC0AD5KXHCW4CKYWAHREBF
- MLA
- Goemaere, Cédric, et al. “Accelerating Hopfield Network Dynamics : Beyond Synchronous Updates and Forward Euler.” 1ST ECAI WORKSHOP ON MACHINE LEARNING MEETS DIFFERENTIAL EQUATIONS : FROM THEORY TO APPLICATIONS, edited by C Coelho et al., vol. 255, 2024.
- APA
- Goemaere, C., Deleu, J., & Demeester, T. (2024). Accelerating Hopfield network dynamics : beyond synchronous updates and forward Euler. In C. Coelho, B. Zimmering, M. Costa, L. Ferras, & O. Niggemann (Eds.), 1ST ECAI WORKSHOP ON MACHINE LEARNING MEETS DIFFERENTIAL EQUATIONS : FROM THEORY TO APPLICATIONS (Vol. 255).
- Chicago author-date
- Goemaere, Cédric, Johannes Deleu, and Thomas Demeester. 2024. “Accelerating Hopfield Network Dynamics : Beyond Synchronous Updates and Forward Euler.” In 1ST ECAI WORKSHOP ON MACHINE LEARNING MEETS DIFFERENTIAL EQUATIONS : FROM THEORY TO APPLICATIONS, edited by C Coelho, B Zimmering, MFP Costa, LL Ferras, and O Niggemann. Vol. 255.
- Chicago author-date (all authors)
- Goemaere, Cédric, Johannes Deleu, and Thomas Demeester. 2024. “Accelerating Hopfield Network Dynamics : Beyond Synchronous Updates and Forward Euler.” In 1ST ECAI WORKSHOP ON MACHINE LEARNING MEETS DIFFERENTIAL EQUATIONS : FROM THEORY TO APPLICATIONS, ed by. C Coelho, B Zimmering, MFP Costa, LL Ferras, and O Niggemann. Vol. 255.
- Vancouver
- 1.Goemaere C, Deleu J, Demeester T. Accelerating Hopfield network dynamics : beyond synchronous updates and forward Euler. In: Coelho C, Zimmering B, Costa M, Ferras L, Niggemann O, editors. 1ST ECAI WORKSHOP ON MACHINE LEARNING MEETS DIFFERENTIAL EQUATIONS : FROM THEORY TO APPLICATIONS. 2024.
- IEEE
- [1]C. Goemaere, J. Deleu, and T. Demeester, “Accelerating Hopfield network dynamics : beyond synchronous updates and forward Euler,” in 1ST ECAI WORKSHOP ON MACHINE LEARNING MEETS DIFFERENTIAL EQUATIONS : FROM THEORY TO APPLICATIONS, Santiago de Compostela, Spain, 2024, vol. 255.
@inproceedings{01JBVC0AD5KXHCW4CKYWAHREBF,
abstract = {{The Hopfield network serves as a fundamental energy-based model in machine learning, capturing memory retrieval dynamics through an ordinary differential equation (ODE). The model's output, the equilibrium point of the ODE, is traditionally computed via synchronous updates using the forward Euler method. This paper aims to overcome some of the disadvantages of this approach. We propose a conceptual shift, viewing Hopfield networks as instances of Deep Equilibrium Models (DEQs). The DEQ framework not only allows for the use of specialized solvers, but also leads to new insights on an empirical inference technique that we will refer to as 'even-odd splitting'. Our theoretical analysis of the method uncovers a parallelizable asynchronous update scheme, which should converge roughly twice as fast as the conventional synchronous updates. Empirical evaluations validate these findings, showcasing the advantages of both the DEQ framework and even-odd splitting in digitally simulating energy minimization in Hopfield networks. The code is available at https://github.com/cgoemaere/hopdeq.}},
author = {{Goemaere, Cédric and Deleu, Johannes and Demeester, Thomas}},
booktitle = {{1ST ECAI WORKSHOP ON MACHINE LEARNING MEETS DIFFERENTIAL EQUATIONS : FROM THEORY TO APPLICATIONS}},
editor = {{Coelho, C and Zimmering, B and Costa, MFP and Ferras, LL and Niggemann, O}},
issn = {{2640-3498}},
keywords = {{Even-odd splitting,Hopfield network,Deep Equilibrium Model,DISCRETE-TIME}},
language = {{eng}},
location = {{Santiago de Compostela, Spain}},
pages = {{21}},
title = {{Accelerating Hopfield network dynamics : beyond synchronous updates and forward Euler}},
url = {{https://proceedings.mlr.press/v255/goemaere24a.html}},
volume = {{255}},
year = {{2024}},
}