Abstract
Airborne wind energy is a lightweight technology that allows power extraction from the wind using airborne devices such as kites and gliders, where the airfoil orientation can be dynamically controlled in order to maximize performance. The dynamical complexity of turbulent aerodynamics makes this optimization problem unapproachable by conventional methods such as classical control theory, which rely on accurate and tractable analytical models of the dynamical system at hand. Here we propose to attack this problem through reinforcement learning, a technique that—by repeated trial-and-error interactions with the environment—learns to associate observations with profitable actions without requiring prior knowledge of the system. We show that in a simulated environment reinforcement learning finds an efficient way to control a kite so that it can tow a vehicle for long distances. The algorithm we use is based on a small set of intuitive observations and its physically transparent interpretation allows to describe the approximately optimal strategy as a simple list of manoeuvring instructions.
Graphical Abstract
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data Availability Statement
Data sharing not applicable to this article as no datasets were generated or analysed during the current study.
References
U. Ahrens, M. Diehl, R. Schmehl, Airborne Wind Energy, Green Energy and Technology (Springer, 2013). https://doi.org/10.1007/978-3-642-39965-7
P. Bechtle, M. Schelbergen, R. Schmehl, U. Zillmann, S. Watson, Renew. Energy 141, 1103 (2019). https://doi.org/10.1016/j.renene.2019.03.118
A. Cherubini, A. Papini, R. Vertechy, M. Fontana, Renew. Sustain. Energy Rev. 51, 1461 (2015). https://doi.org/10.1016/j.rser.2015.07.053
J. Wellicome, J. Wind Eng. Ind. Aerodyn. 20, 111 (1985). https://doi.org/10.1016/0167-6105(85)90015-7
M. Canale, L. Fagiano, M. Milanese, IEEE Trans. Control Syst. Technol. 18(2), 279–293 (2010). https://doi.org/10.1109/TCST.2009.2017933
E. Commission, D.-G. for Research, and innovation, study on challenges in the commercialisation of airborne wind energy systems (Publications Office, 2018). https://doi.org/10.2777/87591
A. Ilzhöfer, B. Houska, M. Diehl, Int. J. Robust Nonlinear Control 17, 1590 (2007). https://doi.org/10.1002/rnc.1210. (onlinelibrary.wiley.com/doi/pdf/10.1002/rnc.1210)
M. Canale, L. Fagiano, M. Ippolito, M. Milanese, Control of tethered airfoils for a new class of wind energy generator, Proceedings of the 45th IEEE Conference on Decision and Control, San Diego, CA, USA, 2006, pp. 4020–4026. https://doi.org/10.1109/CDC.2006.376775
M. Erhard, H. Strauch, Automatic control of pumping cycles for the skysails prototype in airborne wind energy (2018), pp. 189–213
P. Williams, B. Lansdorp, W. Ockels, J. Guid. Control Dyn. 31, 81 (2008). https://doi.org/10.2514/1.30089
S.J. Qin, T.A. Badgwell, in Nonlinear Model Predictive Control, ed. by F. Allgöwer, A. Zheng (Birkhäuser Basel, Basel, 2000), pp. 369–392
I. Argatov, P. Rautakorpi, R. Silvennoinen, Renew. Energy 34, 1525 (2009). https://doi.org/10.1016/j.renene.2008.11.001
B. Houska, M. Diehl, in 2007 European Control Conference, ECC 2007 (2006)
R. Sutton, A. Barto, Reinforcement Learning: An Introduction (MIT Press, 2018)
M.L. Littman, Nature 521, 445 (2015)
D. Ernst, M. Glavic, F. Capitanescu, L. Wehenkel, IEEE Trans. Syst. Man Cybern. Part B Cybern. 39, 517 (2009). https://doi.org/10.1109/TSMCB.2008.2007630
G. Reddy, J. Wong-Ng, A. Celani, T.J. Sejnowski, M. Vergassola, Nature 562, 236 (2018)
M.G. Bellemare, S. Candido, P.S. Castro, J. Gong, M.C. Machado, S. Moitra, S.S. Ponda, Z. Wang, Nature 588, 77 (2020). https://doi.org/10.1038/s41586-020-2939-8
G. Novati, L. Mahadevan, P. Koumoutsakos, Phys. Rev. Fluids 4, 093902 (2019). https://doi.org/10.1103/PhysRevFluids.4.093902
S. Verma, G. Novati, P. Koumoutsakos, Proc. Natl. Acad. Sci. (2018). https://doi.org/10.1073/pnas.1800923115
L. Biferale, F. Bonaccorso, M. Buzzicotti, P. Clark Di Leoni, K. Gustavsson, Chaos Interdiscip. J. Nonlinear Sci. 29, 103138 (2019)
M. Canale, L. Fagiano, M. Milanese, IEEE Trans. Control Syst. Technol. 18, 279 (2010). https://doi.org/10.1109/TCST.2009.2017933
A.T. Patera, J. Comput. Phys. 54, 468 (1984). https://doi.org/10.1016/0021-9991(84)90128-1
V. Avsarkisov, S. Hoyas, M. Oberlack, J. García-Galache, J. Fluid Mech. 751, R1 (2014). https://doi.org/10.1017/jfm.2014.323
K. Arulkumaran, M.P. Deisenroth, M. Brundage, A.A. Bharath, IEEE Signal Process. Mag. 34, 26 (2017). https://doi.org/10.1109/MSP.2017.2743240
Funding
This project has received funding from the European Union Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement N. 956457. We are grateful to Lorenzo Basile and Emanuele Panizon for many profitable discussions and useful comments.
Author information
Authors and Affiliations
Contributions
NO and CL: performed research and wrote the paper. JO and AM: performed research. AC: designed research and wrote the paper.
Corresponding author
Additional information
Quantitative AI in Complex Fluids and Complex Flows: Challenges and Benchmarks. Guest editors: Luca Biferale, Michele Buzzicotti, Massimo Cencini. .
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Orzan, N., Leone, C., Mazzolini, A. et al. Optimizing airborne wind energy with reinforcement learning. Eur. Phys. J. E 46, 2 (2023). https://doi.org/10.1140/epje/s10189-022-00259-2
Received:
Accepted:
Published:
DOI: https://doi.org/10.1140/epje/s10189-022-00259-2