Himanshu Sharma
ABSTRACT
With recent advances in deep learning and large-scale computing, learning-based controls have become increasingly attractive for complex physical systems. Consequently, developing generalized learning-based control software that takes into account the next generation of computing architectures is paramount. Specifically, for the case of complex control, we present the Easily eXtendable Architecture for Reinforcement Learning (EXARL), which aims to support various scientific applications seeking to leverage reinforcement learning (RL) on exascale computing architectures. We demonstrate the efficacy and performance of the EXARL library for the scientific use case of designing a complex control policy to stabilize a power system after experiencing a fault. We use a parallel augmented random search method developed within EXARL and present its preliminary validation and performance stabilization of a fault for the IEEE 39-bus system.
- Shrirang Abhyankar, Renke Huang, Shuangshuang Jin, Bruce Palmer, William Perkins, and Yousu Chen. 2021. Implicit-integration dynamics simulation with the GridPACK framework. In 2021 IEEE Power & Energy Society General Meeting (PESGM). IEEE, 1–5.Google Scholar
Cross Ref
- Zeng Bo, Ouyang Shaojie, Zhang Jianhua, Shi Hui, Wu Geng, and Zeng Ming. 2015. An analysis of previous blackouts in the world: Lessons for China’s power industry. Renewable and Sustainable Energy Reviews 42 (2015), 1151–1163.Google ScholarCross Ref
- Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. 2016. Openai gym. arXiv preprint arXiv:1606.01540 (2016).Google Scholar
- Steven L Brunton and J Nathan Kutz. 2022. Data-driven science and engineering: Machine learning, dynamical systems, and control. Cambridge University Press.Google Scholar
- Gerald W Cauley, David N Cook, Holly A Hawkins, and Critical Infrastructure Protection. 2011. NORTH AMERICAN ELECTRIC) RELIABILITY CORPORATION). (2011).Google Scholar
- Dept. Valley Forge. 2009. Exelon Transmission Planning Criteria, PJM Transm. Plan.Google Scholar
- Istemihan Genc, Ruisheng Diao, Vijay Vittal, Sharma Kolluri, and Sujit Mandal. 2010. Decision tree-based preventive and corrective control applications for dynamic security enhancement in power systems. IEEE Transactions on Power Systems 25, 3 (2010), 1611–1619.Google ScholarCross Ref
- Qiuhua Huang, Renke Huang, Weituo Hao, Jie Tan, Rui Fan, and Zhenyu Huang. 2019. Adaptive power system emergency control using deep reinforcement learning. IEEE Transactions on Smart Grid 11, 2 (2019), 1171–1182.Google ScholarCross Ref
- Renke Huang, Yujiao Chen, Tianzhixi Yin, Xinya Li, Ang Li, Jie Tan, Wenhao Yu, Yuan Liu, and Qiuhua Huang. 2022. Accelerated Derivative-Free Deep Reinforcement Learning for Large-Scale Grid Emergency Voltage Control. IEEE Transactions on Power Systems 37, 1 (2022), 14–25. https://doi.org/10.1109/TPWRS.2021.3095179Google ScholarCross Ref
- B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yogamani, and Patrick Pérez. 2021. Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions on Intelligent Transportation Systems 23, 6 (2021), 4909–4926.Google ScholarCross Ref
- Jens Kober, J Andrew Bagnell, and Jan Peters. 2013. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research 32, 11 (2013), 1238–1274.Google ScholarDigital Library
- P Kundur, GK Morison, and L Wang. 2000. Techniques for on-line transient stability assessment and control. In 2000 IEEE Power Engineering Society Winter Meeting. Conference Proceedings (Cat. No. 00CH37077), Vol. 1. IEEE, 46–51.Google ScholarCross Ref
- Zhihao Li, Guoqiang Yao, Guangchao Geng, and Quanyuan Jiang. 2016. An efficient optimal control method for open-loop transient stability emergency control. IEEE Transactions on Power Systems 32, 4 (2016), 2704–2713.Google ScholarCross Ref
- Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg, Joseph Gonzalez, Michael Jordan, and Ion Stoica. 2018. RLlib: Abstractions for distributed reinforcement learning. In International Conference on Machine Learning. PMLR, 3053–3062.Google Scholar
- Yuri V Makarov, Viktor I Reshetov, A Stroev, and I Voropai. 2005. Blackout prevention in the united states, europe, and russia. Proc. IEEE 93, 11 (2005), 1942–1955.Google ScholarCross Ref
- Horia Mania, Aurelia Guy, and Benjamin Recht. 2018. Simple random search provides a competitive approach to reinforcement learning. arXiv preprint arXiv:1803.07055 (2018).Google Scholar
- Paul Messina. 2017. The Exascale Computing Project. Computing in Science & Engineering 19, 3 (2017), 63–67. https://doi.org/10.1109/MCSE.2017.57Google ScholarCross Ref
- Sidhant Misra, Line Roald, Marc Vuffray, and Michael Chertkov. 2017. Fast and robust determination of power system emergency control actions. arXiv preprint arXiv:1707.07105 (2017).Google Scholar
- Bruce Palmer, William Perkins, Yousu Chen, Shuangshuang Jin, David Callahan, Kevin Glass, Ruisheng Diao, Mark Rice, Stephen Elbert, Mallikarjuna Vallem, 2016. GridPACKTM: A framework for developing power grid simulations on high-performance computing platforms. The International Journal of High Performance Computing Applications 30, 2 (2016), 223–240.Google ScholarDigital Library
- ATD Perera and Parameswaran Kamalaruban. 2021. Applications of reinforcement learning in energy systems. Renewable and Sustainable Energy Reviews 137 (2021), 110618.Google ScholarCross Ref
- Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Y Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, and Marcin Andrychowicz. 2017. Parameter space noise for exploration. arXiv preprint arXiv:1706.01905 (2017).Google Scholar
- Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. 2021. Stable-Baselines3: Reliable Reinforcement Learning Implementations. Journal of Machine Learning Research 22, 268 (2021), 1–8. http://jmlr.org/papers/v22/20-1364.htmlGoogle Scholar
- Vinay Ramakrishnaiah, Malachi Schram, Jamal Mohd-Yusof, Sayan Ghosh, Yunzhi Huang, Ai Kagawa, Christine Sweeney, and Shinjae Yoo. 2020. Easily eXtendable Architecture for Reinforcement Learning (EXARL). https://github.com/exalearn/ExaRL.Google Scholar
- Rick L. Stevens, Valerie E. Taylor, Jeffrey A. Nichols, Arthur B. Maccabe, Katherine A. Yelick, and David Brown. 2020. AI for Science.Google Scholar
- Richard S Sutton and Andrew G Barto. 2018. Reinforcement learning: An introduction. MIT press.Google ScholarDigital Library
- Zidong Zhang, Dongxia Zhang, and Robert C Qiu. 2019. Deep reinforcement learning for power system applications: An overview. CSEE Journal of Power and Energy Systems 6, 1 (2019), 213–225.Google Scholar
Recommendations
Reward Shaping in Episodic Reinforcement Learning
AAMAS '17: Proceedings of the 16th Conference on Autonomous Agents and MultiAgent SystemsRecent advancements in reinforcement learning confirm that reinforcement learning techniques can solve large scale problems leading to high quality autonomous decision making. It is a matter of time until we will see large scale applications of ...
Learning Robot Arm Controls Using Augmented Random Search in Simulated Environments
Multi-disciplinary Trends in Artificial IntelligenceAbstractWe investigate the learning of continuous action policy for controlling a six-axes robot arm. Traditional tabular Q-Learning can handle discrete actions well but less so for continuous actions since the tabular approach is constrained by the size ...
Using Transfer Learning to Speed-Up Reinforcement Learning: A Cased-Based Approach
LARS '10: Proceedings of the 2010 Latin American Robotics Symposium and Intelligent Robotics MeetingReinforcement Learning (RL) is a well-known technique for the solution of problems where agents need to act with success in an unknown environment, learning through trial and error. However, this technique is not efficient enough to be used in ...
Comments