KeyCLD : learning constrained Lagrangian dynamics in keypoint coordinates from images
- Author
- Rembert Daems (UGent) , Jeroen Taets (UGent) , Francis wyffels (UGent) and Guillaume Crevecoeur (UGent)
- Organization
- Project
- Abstract
- We present KeyCLD, a framework to learn Lagrangian dynamics from images. Learned keypoints represent semantic landmarks in images and can directly represent state dynamics. We show that interpreting this state as Cartesian coordinates, coupled with explicit holonomic constraints, allows expressing the dynamics with a constrained Lagrangian. KeyCLD is trained unsupervised end-to-end on sequences of images. Our method explicitly models the mass matrix, potential energy and the input matrix, thus allowing energy based control. We demonstrate learning of Lagrangian dynamics from images on the cl_bnmsqnk pendulum, cartpole and acrobot environments. KeyCLD can be learned on these systems, whether they are unactuated, underactuated or fully actuated. Trained models are able to produce long-term video predictions, showing that the dynamics are accurately learned. We compare with Lag-VAE, Lag-caVAE and HGN, and investigate the benefit of the Lagrangian prior and the constraint function. KeyCLD achieves the highest valid prediction time on all benchmarks. Additionally, a very straightforward energy shaping controller is successfully applied on the fully actuated systems.
- Keywords
- Artificial Intelligence, Computer Science Applications, Lagrangian Dynamics, Unsupervised learning, Energy shaping control, Lagrangian, Dynamics, Video, Images
Downloads
-
Publisher version.pdf
- full text (Published version)
- |
- open access
- |
- |
- 3.78 MB
Citation
Please use this url to cite or link to this publication: http://hdl.handle.net/1854/LU-01HKPXG89WTRQNXKEH7NGSDQ9T
- MLA
- Daems, Rembert, et al. “KeyCLD : Learning Constrained Lagrangian Dynamics in Keypoint Coordinates from Images.” NEUROCOMPUTING, vol. 573, 2024, doi:10.1016/j.neucom.2023.127175.
- APA
- Daems, R., Taets, J., wyffels, F., & Crevecoeur, G. (2024). KeyCLD : learning constrained Lagrangian dynamics in keypoint coordinates from images. NEUROCOMPUTING, 573. https://doi.org/10.1016/j.neucom.2023.127175
- Chicago author-date
- Daems, Rembert, Jeroen Taets, Francis wyffels, and Guillaume Crevecoeur. 2024. “KeyCLD : Learning Constrained Lagrangian Dynamics in Keypoint Coordinates from Images.” NEUROCOMPUTING 573. https://doi.org/10.1016/j.neucom.2023.127175.
- Chicago author-date (all authors)
- Daems, Rembert, Jeroen Taets, Francis wyffels, and Guillaume Crevecoeur. 2024. “KeyCLD : Learning Constrained Lagrangian Dynamics in Keypoint Coordinates from Images.” NEUROCOMPUTING 573. doi:10.1016/j.neucom.2023.127175.
- Vancouver
- 1.Daems R, Taets J, wyffels F, Crevecoeur G. KeyCLD : learning constrained Lagrangian dynamics in keypoint coordinates from images. NEUROCOMPUTING. 2024;573.
- IEEE
- [1]R. Daems, J. Taets, F. wyffels, and G. Crevecoeur, “KeyCLD : learning constrained Lagrangian dynamics in keypoint coordinates from images,” NEUROCOMPUTING, vol. 573, 2024.
@article{01HKPXG89WTRQNXKEH7NGSDQ9T,
abstract = {{We present KeyCLD, a framework to learn Lagrangian dynamics from images. Learned keypoints represent semantic landmarks in images and can directly represent state dynamics. We show that interpreting this state as Cartesian coordinates, coupled with explicit holonomic constraints, allows expressing the dynamics with a constrained Lagrangian. KeyCLD is trained unsupervised end-to-end on sequences of images. Our method explicitly models the mass matrix, potential energy and the input matrix, thus allowing energy based control. We demonstrate learning of Lagrangian dynamics from images on the cl_bnmsqnk pendulum, cartpole and acrobot environments. KeyCLD can be learned on these systems, whether they are unactuated, underactuated or fully actuated. Trained models are able to produce long-term video predictions, showing that the dynamics are accurately learned. We compare with Lag-VAE, Lag-caVAE and HGN, and investigate the benefit of the Lagrangian prior and the constraint function. KeyCLD achieves the highest valid prediction time on all benchmarks. Additionally, a very straightforward energy shaping controller is successfully applied on the fully actuated systems.}},
articleno = {{127175}},
author = {{Daems, Rembert and Taets, Jeroen and wyffels, Francis and Crevecoeur, Guillaume}},
issn = {{0925-2312}},
journal = {{NEUROCOMPUTING}},
keywords = {{Artificial Intelligence,Computer Science Applications,Lagrangian Dynamics,Unsupervised learning,Energy shaping control,Lagrangian,Dynamics,Video,Images}},
language = {{eng}},
pages = {{14}},
title = {{KeyCLD : learning constrained Lagrangian dynamics in keypoint coordinates from images}},
url = {{http://doi.org/10.1016/j.neucom.2023.127175}},
volume = {{573}},
year = {{2024}},
}
- Altmetric
- View in Altmetric
- Web of Science
- Times cited: