Accelerating hierarchical associative memory : a deep equilibrium approach
- Author
- Cédric Goemaere (UGent) , Johannes Deleu (UGent) and Thomas Demeester (UGent)
- Organization
- Project
- Abstract
- Hierarchical Associative Memory models have recently been proposed as a versatile extension of continuous Hopfield networks. In order to facilitate future research on such models, especially at scale, we focus on increasing their simulation efficiency on digital hardware. In particular, we propose two strategies to speed up memory retrieval in these models, which corresponds to their use at inference, but is equally important during training. First, we show how they can be cast as Deep Equilibrium Models, which allows using faster and more stable solvers. Second, inspired by earlier work, we show that alternating optimization of the even and odd layers accelerates memory retrieval by a factor close to two. Combined, these two techniques allow for a much faster energy minimization, as shown in our proof-of-concept experimental results. The code is available at https://github.com/cgoemaere/hamdeq
Downloads
-
(...).pdf
- full text (Published version)
- |
- UGent only
- |
- |
- 589.86 KB
Citation
Please use this url to cite or link to this publication: http://hdl.handle.net/1854/LU-01HQMZD7C12SD4SPNYCS7MWXH2
- MLA
- Goemaere, Cédric, et al. “Accelerating Hierarchical Associative Memory : A Deep Equilibrium Approach.” Associative Memory & Hopfield Networks in 2023 : NeurIPS 2023 Workshop, Proceedings, 2023.
- APA
- Goemaere, C., Deleu, J., & Demeester, T. (2023). Accelerating hierarchical associative memory : a deep equilibrium approach. Associative Memory & Hopfield Networks in 2023 : NeurIPS 2023 Workshop, Proceedings. Presented at the Associative Memory & Hopfield Networks in 2023 : NeurIPS 2023 workshop, New Orleans, Louisiana.
- Chicago author-date
- Goemaere, Cédric, Johannes Deleu, and Thomas Demeester. 2023. “Accelerating Hierarchical Associative Memory : A Deep Equilibrium Approach.” In Associative Memory & Hopfield Networks in 2023 : NeurIPS 2023 Workshop, Proceedings.
- Chicago author-date (all authors)
- Goemaere, Cédric, Johannes Deleu, and Thomas Demeester. 2023. “Accelerating Hierarchical Associative Memory : A Deep Equilibrium Approach.” In Associative Memory & Hopfield Networks in 2023 : NeurIPS 2023 Workshop, Proceedings.
- Vancouver
- 1.Goemaere C, Deleu J, Demeester T. Accelerating hierarchical associative memory : a deep equilibrium approach. In: Associative Memory & Hopfield Networks in 2023 : NeurIPS 2023 workshop, Proceedings. 2023.
- IEEE
- [1]C. Goemaere, J. Deleu, and T. Demeester, “Accelerating hierarchical associative memory : a deep equilibrium approach,” in Associative Memory & Hopfield Networks in 2023 : NeurIPS 2023 workshop, Proceedings, New Orleans, Louisiana, 2023.
@inproceedings{01HQMZD7C12SD4SPNYCS7MWXH2,
abstract = {{Hierarchical Associative Memory models have recently been proposed as a versatile extension of continuous Hopfield networks. In order to facilitate future research on such models, especially at scale, we focus on increasing their simulation efficiency on digital hardware. In particular, we propose two strategies to speed up memory retrieval in these models, which corresponds to their use at inference, but is equally important during training. First, we show how they can be cast as Deep Equilibrium Models, which allows using faster and more stable solvers. Second, inspired by earlier work, we show that alternating optimization of the even and odd layers accelerates memory retrieval by a factor close to two. Combined, these two techniques allow for a much faster energy minimization, as shown in our proof-of-concept experimental results. The code is available at https://github.com/cgoemaere/hamdeq}},
author = {{Goemaere, Cédric and Deleu, Johannes and Demeester, Thomas}},
booktitle = {{Associative Memory & Hopfield Networks in 2023 : NeurIPS 2023 workshop, Proceedings}},
language = {{eng}},
location = {{New Orleans, Louisiana}},
pages = {{12}},
title = {{Accelerating hierarchical associative memory : a deep equilibrium approach}},
year = {{2023}},
}