Heuristic action execution for energy efficient charge-sustaining control of connected hybrid vehicles with model-free double Q-learning

Bin Shuai, Quan Zhou*, Ji Li, Yinglong He, Ziyang Li, Huw Williams, Hongming Xu, Shijin Shuai

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

3 Citations (Scopus)
313 Downloads (Pure)


This paper investigates a model-free supervisory control methodology with double Q-learning for the hybrid vehicle in charge-sustaining scenarios. It aims to improve the vehicle’s energy efficiency continuously while maintaining the battery’s state-of-charge in real-world driving. Two new heuristic action execution policies, the max-value-based policy and the random policy, are proposed for the double Q-learning method to reduce overestimation of the merit-function values for each action in power-split control of the vehicle. Experimental studies based on software-in-the-loop (offline learning) and hardware-in-the-loop (online learning) platforms are carried out to explore the potential of energy-saving in four driving cycles defined with real-world vehicle operations. The results from 35 rounds of offline undisturbed learning show that the heuristic action execution policies can improve the learning performance of conventional double Q-learning by achieving at least 1.09% higher energy efficiency. The proposed methods achieve similar results obtained by dynamic programming, but they have the capability of real-time online application. Double Q-learnings are shown more robust to turbulence during the disturbed learning: they realise at least three times improvement in energy efficiency compared to the standard Q-learning. Random execution policy achieves 1.18% higher energy efficiency than the max-value-based policy for the same driving condition. Significant tests show that deciding factor in the random execution policy has little impact on learning performance. By implementing the control strategies for online learning, the proposed model-free control method can save energy by more than 4.55% in the predefined real-world driving conditions compared to the method using standard Q-learning.
Original languageEnglish
Article number114900
Pages (from-to)1-11
Number of pages11
JournalApplied Energy
Early online date6 Apr 2020
Publication statusPublished - 1 Jun 2020

Bibliographical note

Funding Information:
The work is partially funded by the Innovate UK (Grant 102253 ) for the development of world-leading off-highway vehicular technologies. It is also sponsored by the State Key Laboratory of Automotive Safety and Energy ( Tsinghua University ) under Project No. KF2029 for the development of digital-twin enabled online reinforcement learning. The authors gratefully acknowledge the support from Textron Ground Support Equipment UK and Hyper-Drive Innovation.

Publisher Copyright:
© 2020


  • charge-sustaining control
  • double Q-learning
  • energy efficiency optimisation
  • hybrid vehicle
  • reinforcement learning
  • Double Q-learning
  • Hybrid vehicle
  • Charge-sustaining control
  • Energy efficiency optimisation
  • Reinforcement learning

ASJC Scopus subject areas

  • Mechanical Engineering
  • Energy(all)
  • Management, Monitoring, Policy and Law
  • Building and Construction


Dive into the research topics of 'Heuristic action execution for energy efficient charge-sustaining control of connected hybrid vehicles with model-free double Q-learning'. Together they form a unique fingerprint.

Cite this