Optimal Energy Management of Plug-In Hybrid Electric Vehicles Through Ensemble Reinforcement Learning With Exploration-to-Exploitation Ratio Control

Reinforcement learning (RL) has demonstrated its advantages in the intelligent control of many vehicle systems. However, controlling the exploration-to-exploitation (E2E) ratio of RL for the best performance in real-world operations is a great challenge. To obtain the optimal E2E ratio for managing the energy flow of a plug-in hybrid electric vehicle (PHEV) in real-world driving, this paper proposes an ensemble learning scheme based on two independent Q-learning agents working back-to-back competitively. At each sampling time, these agents generate two candidate control actions based on state observation and their control policies. Three decay functions, including the widely-used exponential decay and two new decay methods i.e., reciprocal function-base decay and step-based decay, are introduced to formulate one-dimensional look-up tables for E2E control. Then the PHEV control action will be selected from the candidate actions by a learning automata module(LAM) developed in this paper. The combinations of the three decay methods and three ensemble strategies with maximum-based, random-based, and weighted-based methods are investigated with the considerations of their energy efficiency, real-time performance, and robustness. Experiments are carried out on software-in-loop and hardware-in-the-loop platforms to demonstrate the energy-saving potentials. By implementing the ensemble learning scheme based on the weighted-based ensemble method in the control of the studied PHEV, up to 1.04% of energy can be saved under the predefined real-world driving cycles compared to the conventional Q-learning scheme.