Multi-step reinforcement learning for model-free predictive energy management of an electrified off-highway vehicle

Research output: Contribution to journalArticle

Standard

Multi-step reinforcement learning for model-free predictive energy management of an electrified off-highway vehicle. / Zhou, Quan; Li, Ji; Shuai, Bin; Williams, Huw; He, Yinglong; Li, Ziyang; Xu, Hongming; Yan, Fuwu.

In: Applied Energy, Vol. 255, 113755, 01.12.2019.

Research output: Contribution to journalArticle

Harvard

APA

Vancouver

Author

Bibtex

@article{4cc92fc4b5c2436499863977cb83b5a4,
title = "Multi-step reinforcement learning for model-free predictive energy management of an electrified off-highway vehicle",
abstract = "The energy management system of an electrified vehicle is one of the most important supervisory control systems which manages the use of on-board energy resources. This paper researches a {\textquoteleft}model-free{\textquoteright} predictive energy management system for a connected electrified off-highway vehicle. A new reinforcement learning algorithm with the capability of {\textquoteleft}multi-step{\textquoteright} learning is proposed to enable the all-life-long online optimisation of the energy management control policy. Three multi-step learning strategies (Sum-to-Terminal, Average-to-Neighbour Recurrent-to-Terminal) are researched for the first time. Hardware-in-the-loop tests are carried out to examine the control functionality for real application of the proposed {\textquoteleft}model-free{\textquoteright} method. The results show that the proposed method can continuously improve the vehicle's energy efficiency during the real-time hardware-in-the-loop test, which increased from the initial level of 34% to 44% after 5 h{\textquoteright} 35-step learning. Compared with a well-designed model-based predictive energy management control policy, the model-free predictive energy management method can increase the prediction horizon length by 71% (from 35 to 65 steps with 1 s interval in real-time computation) and can save energy by at least 7.8% for the same driving conditions.",
keywords = "Energy management, Hybrid electric vehicle, Markov decision problem, Model-free predictive control, Multi-step reinforcement learning",
author = "Quan Zhou and Ji Li and Bin Shuai and Huw Williams and Yinglong He and Ziyang Li and Hongming Xu and Fuwu Yan",
year = "2019",
month = dec
day = "1",
doi = "10.1016/j.apenergy.2019.113755",
language = "English",
volume = "255",
journal = "Applied Energy",
issn = "0306-2619",
publisher = "Elsevier",

}

RIS

TY - JOUR

T1 - Multi-step reinforcement learning for model-free predictive energy management of an electrified off-highway vehicle

AU - Zhou, Quan

AU - Li, Ji

AU - Shuai, Bin

AU - Williams, Huw

AU - He, Yinglong

AU - Li, Ziyang

AU - Xu, Hongming

AU - Yan, Fuwu

PY - 2019/12/1

Y1 - 2019/12/1

N2 - The energy management system of an electrified vehicle is one of the most important supervisory control systems which manages the use of on-board energy resources. This paper researches a ‘model-free’ predictive energy management system for a connected electrified off-highway vehicle. A new reinforcement learning algorithm with the capability of ‘multi-step’ learning is proposed to enable the all-life-long online optimisation of the energy management control policy. Three multi-step learning strategies (Sum-to-Terminal, Average-to-Neighbour Recurrent-to-Terminal) are researched for the first time. Hardware-in-the-loop tests are carried out to examine the control functionality for real application of the proposed ‘model-free’ method. The results show that the proposed method can continuously improve the vehicle's energy efficiency during the real-time hardware-in-the-loop test, which increased from the initial level of 34% to 44% after 5 h’ 35-step learning. Compared with a well-designed model-based predictive energy management control policy, the model-free predictive energy management method can increase the prediction horizon length by 71% (from 35 to 65 steps with 1 s interval in real-time computation) and can save energy by at least 7.8% for the same driving conditions.

AB - The energy management system of an electrified vehicle is one of the most important supervisory control systems which manages the use of on-board energy resources. This paper researches a ‘model-free’ predictive energy management system for a connected electrified off-highway vehicle. A new reinforcement learning algorithm with the capability of ‘multi-step’ learning is proposed to enable the all-life-long online optimisation of the energy management control policy. Three multi-step learning strategies (Sum-to-Terminal, Average-to-Neighbour Recurrent-to-Terminal) are researched for the first time. Hardware-in-the-loop tests are carried out to examine the control functionality for real application of the proposed ‘model-free’ method. The results show that the proposed method can continuously improve the vehicle's energy efficiency during the real-time hardware-in-the-loop test, which increased from the initial level of 34% to 44% after 5 h’ 35-step learning. Compared with a well-designed model-based predictive energy management control policy, the model-free predictive energy management method can increase the prediction horizon length by 71% (from 35 to 65 steps with 1 s interval in real-time computation) and can save energy by at least 7.8% for the same driving conditions.

KW - Energy management

KW - Hybrid electric vehicle

KW - Markov decision problem

KW - Model-free predictive control

KW - Multi-step reinforcement learning

UR - http://www.scopus.com/inward/record.url?scp=85071321446&partnerID=8YFLogxK

U2 - 10.1016/j.apenergy.2019.113755

DO - 10.1016/j.apenergy.2019.113755

M3 - Article

AN - SCOPUS:85071321446

VL - 255

JO - Applied Energy

JF - Applied Energy

SN - 0306-2619

M1 - 113755

ER -