Control policy correction framework for reinforcement learning-based energy arbitrage strategies
- Author
- Seyedsoroush Karimi Madahi (UGent) , Gargya Gokhale (UGent) , Marie-Sophie Verwee (UGent) , Bert Claessens (UGent) and Chris Develder (UGent)
- Organization
- Abstract
- A continuous rise in the penetration of renewable energy sources, along with the use of the single imbalance pricing, provides a new opportunity for balance responsible parties to reduce their cost through energy arbitrage in the imbalance settlement mechanism. Model-free reinforcement learning (RL) methods are an appropriate choice for solving the energy arbitrage problem due to their outstanding performance in solving complex stochastic sequential problems. However, RL is rarely deployed in real-world applications since its learned policy does not necessarily guarantee safety during the execution phase. In this paper, we propose a new RL-based control framework for batteries to obtain a safe energy arbitrage strategy in the imbalance settlement mechanism. In our proposed control framework, the agent initially aims to optimize the arbitrage revenue. Subsequently, in the post-processing step, we correct (constrain) the learned policy following a knowledge distillation process based on properties that follow human intuition. Our post-processing step is a generic method and is not restricted to the energy arbitrage domain. We use the Belgian imbalance price of 2023 to evaluate the performance of our proposed framework. Furthermore, we deploy our proposed control framework on a real battery to show its capability in the real world.
- Keywords
- Battery energy storage systems, distributional reinforcement learning, energy arbitrage, interpretable reinforcement learning, knowledge distillation, safe reinforcement learning
Downloads
-
(...).pdf
- full text (Published version)
- |
- UGent only
- |
- |
- 2.68 MB
Citation
Please use this url to cite or link to this publication: http://hdl.handle.net/1854/LU-01HZ1TK0GZQBJ2PVS7Q9J91MYA
- MLA
- Karimi Madahi, Seyedsoroush, et al. “Control Policy Correction Framework for Reinforcement Learning-Based Energy Arbitrage Strategies.” PROCEEDINGS OF THE 15TH ACM INTERNATIONAL CONFERENCE ON FUTURE AND SUSTAINABLE ENERGY SYSTEMS, E-ENERGY 2024, Association for Computing Machinery (ACM), 2024, pp. 123–33, doi:10.1145/3632775.3661948.
- APA
- Karimi Madahi, S., Gokhale, G., Verwee, M.-S., Claessens, B., & Develder, C. (2024). Control policy correction framework for reinforcement learning-based energy arbitrage strategies. PROCEEDINGS OF THE 15TH ACM INTERNATIONAL CONFERENCE ON FUTURE AND SUSTAINABLE ENERGY SYSTEMS, E-ENERGY 2024, 123–133. https://doi.org/10.1145/3632775.3661948
- Chicago author-date
- Karimi Madahi, Seyedsoroush, Gargya Gokhale, Marie-Sophie Verwee, Bert Claessens, and Chris Develder. 2024. “Control Policy Correction Framework for Reinforcement Learning-Based Energy Arbitrage Strategies.” In PROCEEDINGS OF THE 15TH ACM INTERNATIONAL CONFERENCE ON FUTURE AND SUSTAINABLE ENERGY SYSTEMS, E-ENERGY 2024, 123–33. Association for Computing Machinery (ACM). https://doi.org/10.1145/3632775.3661948.
- Chicago author-date (all authors)
- Karimi Madahi, Seyedsoroush, Gargya Gokhale, Marie-Sophie Verwee, Bert Claessens, and Chris Develder. 2024. “Control Policy Correction Framework for Reinforcement Learning-Based Energy Arbitrage Strategies.” In PROCEEDINGS OF THE 15TH ACM INTERNATIONAL CONFERENCE ON FUTURE AND SUSTAINABLE ENERGY SYSTEMS, E-ENERGY 2024, 123–133. Association for Computing Machinery (ACM). doi:10.1145/3632775.3661948.
- Vancouver
- 1.Karimi Madahi S, Gokhale G, Verwee M-S, Claessens B, Develder C. Control policy correction framework for reinforcement learning-based energy arbitrage strategies. In: PROCEEDINGS OF THE 15TH ACM INTERNATIONAL CONFERENCE ON FUTURE AND SUSTAINABLE ENERGY SYSTEMS, E-ENERGY 2024. Association for Computing Machinery (ACM); 2024. p. 123–33.
- IEEE
- [1]S. Karimi Madahi, G. Gokhale, M.-S. Verwee, B. Claessens, and C. Develder, “Control policy correction framework for reinforcement learning-based energy arbitrage strategies,” in PROCEEDINGS OF THE 15TH ACM INTERNATIONAL CONFERENCE ON FUTURE AND SUSTAINABLE ENERGY SYSTEMS, E-ENERGY 2024, Singapore, Singapore, 2024, pp. 123–133.
@inproceedings{01HZ1TK0GZQBJ2PVS7Q9J91MYA,
abstract = {{A continuous rise in the penetration of renewable energy sources, along with the use of the single imbalance pricing, provides a new opportunity for balance responsible parties to reduce their cost through energy arbitrage in the imbalance settlement mechanism. Model-free reinforcement learning (RL) methods are an appropriate choice for solving the energy arbitrage problem due to their outstanding performance in solving complex stochastic sequential problems. However, RL is rarely deployed in real-world applications since its learned policy does not necessarily guarantee safety during the execution phase. In this paper, we propose a new RL-based control framework for batteries to obtain a safe energy arbitrage strategy in the imbalance settlement mechanism. In our proposed control framework, the agent initially aims to optimize the arbitrage revenue. Subsequently, in the post-processing step, we correct (constrain) the learned policy following a knowledge distillation process based on properties that follow human intuition. Our post-processing step is a generic method and is not restricted to the energy arbitrage domain. We use the Belgian imbalance price of 2023 to evaluate the performance of our proposed framework. Furthermore, we deploy our proposed control framework on a real battery to show its capability in the real world.}},
author = {{Karimi Madahi, Seyedsoroush and Gokhale, Gargya and Verwee, Marie-Sophie and Claessens, Bert and Develder, Chris}},
booktitle = {{PROCEEDINGS OF THE 15TH ACM INTERNATIONAL CONFERENCE ON FUTURE AND SUSTAINABLE ENERGY SYSTEMS, E-ENERGY 2024}},
isbn = {{9798400704802}},
keywords = {{Battery energy storage systems,distributional reinforcement learning,energy arbitrage,interpretable reinforcement learning,knowledge distillation,safe reinforcement learning}},
language = {{eng}},
location = {{Singapore, Singapore}},
pages = {{123--133}},
publisher = {{Association for Computing Machinery (ACM)}},
title = {{Control policy correction framework for reinforcement learning-based energy arbitrage strategies}},
url = {{http://doi.org/10.1145/3632775.3661948}},
year = {{2024}},
}
- Altmetric
- View in Altmetric
- Web of Science
- Times cited: