A Comparison of Methods for Missing Data Treatment in Building Sensor Data

Mehdi Pazhoohesh; Zoya Pourmirza; Sara Walker

doi:10.1109/SEGE.2019.8859963

A Comparison of Methods for Missing Data Treatment in Building Sensor Data

Mehdi Pazhoohesh, Zoya Pourmirza, Sara Walker

Chemical Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Data collection is a fundamental component in the study of energy and buildings. Errors and inconsistencies in the data collected from test environment can negatively influence the energy consumption modelling of a building and other control and management applications. This paper addresses the gap in the current study of missing data treatment. It presents a comparative study of eight methods for imputing missing values in building sensor data. The data set used in this study, are real data collected from our test bed, which is a living lab in the Newcastle University. When the data imputation process is completed, we used Mean Absolute Error, and Root Mean Squared Error methods to evaluate the difference between the imputed values and real values. In order to achieve more accurate and robust results, this process has been repeated 1000, and the average of 1000 simulation is demonstrated in this paper. Finally, it is concluded that it is necessary to identify the percentage of missing data before selecting the proper imputation method, in order to achieve the best result.

Original language	English
Title of host publication	Proceedings of 2019 the 7th International Conference on Smart Energy Grid Engineering, SEGE 2019
Publisher	Institute of Electrical and Electronics Engineers (IEEE)
Pages	255-259
Number of pages	5
ISBN (Electronic)	9781728124407
DOIs	https://doi.org/10.1109/SEGE.2019.8859963
Publication status	Published - Aug 2019
Event	7th IEEE International Conference on Smart Energy Grid Engineering, SEGE 2019 - Oshawa, Canada Duration: 12 Aug 2019 → 14 Aug 2019

Publication series

Name	Proceedings of 2019 the 7th International Conference on Smart Energy Grid Engineering, SEGE 2019

Conference

Conference	7th IEEE International Conference on Smart Energy Grid Engineering, SEGE 2019
Country/Territory	Canada
City	Oshawa
Period	12/08/19 → 14/08/19

Bibliographical note

Funding Information:
ACKNOWLEDGEMENT The research reported in this paper was supported by Building as a Power Plant: The use of buildings to provide demand response project, funded by the Engineering and Physical Sciences Research Council under Programme Grant EP/P034241/1, and the Active Building Centre (ABC), supported by Industrial Strategy Challenge Fund under Programme Grant EP/S016627/1.

Funding Information:
The authors would like to acknowledge the support provided by Prince Mohammad bin Fahd University, and King Fahd University of Petroleum and Minerals.

Publisher Copyright:
© 2019 IEEE.

Keywords

data imputation
energy and building data
KNN
MAE
MCMC
missing value
RMSE

ASJC Scopus subject areas

Computer Networks and Communications
Energy Engineering and Power Technology
Artificial Intelligence
Electrical and Electronic Engineering

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

Access to Document

10.1109/SEGE.2019.8859963

Cite this

Pazhoohesh, M., Pourmirza, Z., & Walker, S. (2019). A Comparison of Methods for Missing Data Treatment in Building Sensor Data. In Proceedings of 2019 the 7th International Conference on Smart Energy Grid Engineering, SEGE 2019 (pp. 255-259). Article 8859963 (Proceedings of 2019 the 7th International Conference on Smart Energy Grid Engineering, SEGE 2019). Institute of Electrical and Electronics Engineers (IEEE). https://doi.org/10.1109/SEGE.2019.8859963

Pazhoohesh, Mehdi ; Pourmirza, Zoya ; Walker, Sara. / A Comparison of Methods for Missing Data Treatment in Building Sensor Data. Proceedings of 2019 the 7th International Conference on Smart Energy Grid Engineering, SEGE 2019. Institute of Electrical and Electronics Engineers (IEEE), 2019. pp. 255-259 (Proceedings of 2019 the 7th International Conference on Smart Energy Grid Engineering, SEGE 2019).

@inproceedings{2178d82c0ff24fd887b1859f3d579bdc,

title = "A Comparison of Methods for Missing Data Treatment in Building Sensor Data",

abstract = "Data collection is a fundamental component in the study of energy and buildings. Errors and inconsistencies in the data collected from test environment can negatively influence the energy consumption modelling of a building and other control and management applications. This paper addresses the gap in the current study of missing data treatment. It presents a comparative study of eight methods for imputing missing values in building sensor data. The data set used in this study, are real data collected from our test bed, which is a living lab in the Newcastle University. When the data imputation process is completed, we used Mean Absolute Error, and Root Mean Squared Error methods to evaluate the difference between the imputed values and real values. In order to achieve more accurate and robust results, this process has been repeated 1000, and the average of 1000 simulation is demonstrated in this paper. Finally, it is concluded that it is necessary to identify the percentage of missing data before selecting the proper imputation method, in order to achieve the best result.",

keywords = "data imputation, energy and building data, KNN, MAE, MCMC, missing value, RMSE",

author = "Mehdi Pazhoohesh and Zoya Pourmirza and Sara Walker",

note = "Funding Information: ACKNOWLEDGEMENT The research reported in this paper was supported by Building as a Power Plant: The use of buildings to provide demand response project, funded by the Engineering and Physical Sciences Research Council under Programme Grant EP/P034241/1, and the Active Building Centre (ABC), supported by Industrial Strategy Challenge Fund under Programme Grant EP/S016627/1. Funding Information: The authors would like to acknowledge the support provided by Prince Mohammad bin Fahd University, and King Fahd University of Petroleum and Minerals. Publisher Copyright: {\textcopyright} 2019 IEEE.; 7th IEEE International Conference on Smart Energy Grid Engineering, SEGE 2019 ; Conference date: 12-08-2019 Through 14-08-2019",

year = "2019",

month = aug,

doi = "10.1109/SEGE.2019.8859963",

language = "English",

series = "Proceedings of 2019 the 7th International Conference on Smart Energy Grid Engineering, SEGE 2019",

publisher = "Institute of Electrical and Electronics Engineers (IEEE)",

pages = "255--259",

booktitle = "Proceedings of 2019 the 7th International Conference on Smart Energy Grid Engineering, SEGE 2019",

}

Pazhoohesh, M, Pourmirza, Z & Walker, S 2019, A Comparison of Methods for Missing Data Treatment in Building Sensor Data. in Proceedings of 2019 the 7th International Conference on Smart Energy Grid Engineering, SEGE 2019., 8859963, Proceedings of 2019 the 7th International Conference on Smart Energy Grid Engineering, SEGE 2019, Institute of Electrical and Electronics Engineers (IEEE), pp. 255-259, 7th IEEE International Conference on Smart Energy Grid Engineering, SEGE 2019, Oshawa, Canada, 12/08/19. https://doi.org/10.1109/SEGE.2019.8859963

A Comparison of Methods for Missing Data Treatment in Building Sensor Data. / Pazhoohesh, Mehdi; Pourmirza, Zoya; Walker, Sara.
Proceedings of 2019 the 7th International Conference on Smart Energy Grid Engineering, SEGE 2019. Institute of Electrical and Electronics Engineers (IEEE), 2019. p. 255-259 8859963 (Proceedings of 2019 the 7th International Conference on Smart Energy Grid Engineering, SEGE 2019).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - A Comparison of Methods for Missing Data Treatment in Building Sensor Data

AU - Pazhoohesh, Mehdi

AU - Pourmirza, Zoya

AU - Walker, Sara

N1 - Funding Information: ACKNOWLEDGEMENT The research reported in this paper was supported by Building as a Power Plant: The use of buildings to provide demand response project, funded by the Engineering and Physical Sciences Research Council under Programme Grant EP/P034241/1, and the Active Building Centre (ABC), supported by Industrial Strategy Challenge Fund under Programme Grant EP/S016627/1. Funding Information: The authors would like to acknowledge the support provided by Prince Mohammad bin Fahd University, and King Fahd University of Petroleum and Minerals. Publisher Copyright: © 2019 IEEE.

PY - 2019/8

Y1 - 2019/8

N2 - Data collection is a fundamental component in the study of energy and buildings. Errors and inconsistencies in the data collected from test environment can negatively influence the energy consumption modelling of a building and other control and management applications. This paper addresses the gap in the current study of missing data treatment. It presents a comparative study of eight methods for imputing missing values in building sensor data. The data set used in this study, are real data collected from our test bed, which is a living lab in the Newcastle University. When the data imputation process is completed, we used Mean Absolute Error, and Root Mean Squared Error methods to evaluate the difference between the imputed values and real values. In order to achieve more accurate and robust results, this process has been repeated 1000, and the average of 1000 simulation is demonstrated in this paper. Finally, it is concluded that it is necessary to identify the percentage of missing data before selecting the proper imputation method, in order to achieve the best result.

AB - Data collection is a fundamental component in the study of energy and buildings. Errors and inconsistencies in the data collected from test environment can negatively influence the energy consumption modelling of a building and other control and management applications. This paper addresses the gap in the current study of missing data treatment. It presents a comparative study of eight methods for imputing missing values in building sensor data. The data set used in this study, are real data collected from our test bed, which is a living lab in the Newcastle University. When the data imputation process is completed, we used Mean Absolute Error, and Root Mean Squared Error methods to evaluate the difference between the imputed values and real values. In order to achieve more accurate and robust results, this process has been repeated 1000, and the average of 1000 simulation is demonstrated in this paper. Finally, it is concluded that it is necessary to identify the percentage of missing data before selecting the proper imputation method, in order to achieve the best result.

KW - data imputation

KW - energy and building data

KW - KNN

KW - MAE

KW - MCMC

KW - missing value

KW - RMSE

UR - http://www.scopus.com/inward/record.url?scp=85074075421&partnerID=8YFLogxK

U2 - 10.1109/SEGE.2019.8859963

DO - 10.1109/SEGE.2019.8859963

M3 - Conference contribution

AN - SCOPUS:85074075421

T3 - Proceedings of 2019 the 7th International Conference on Smart Energy Grid Engineering, SEGE 2019

SP - 255

EP - 259

BT - Proceedings of 2019 the 7th International Conference on Smart Energy Grid Engineering, SEGE 2019

PB - Institute of Electrical and Electronics Engineers (IEEE)

T2 - 7th IEEE International Conference on Smart Energy Grid Engineering, SEGE 2019

Y2 - 12 August 2019 through 14 August 2019

ER -

Pazhoohesh M, Pourmirza Z, Walker S. A Comparison of Methods for Missing Data Treatment in Building Sensor Data. In Proceedings of 2019 the 7th International Conference on Smart Energy Grid Engineering, SEGE 2019. Institute of Electrical and Electronics Engineers (IEEE). 2019. p. 255-259. 8859963. (Proceedings of 2019 the 7th International Conference on Smart Energy Grid Engineering, SEGE 2019). doi: 10.1109/SEGE.2019.8859963

A Comparison of Methods for Missing Data Treatment in Building Sensor Data

Abstract

Publication series

Conference

Bibliographical note

Keywords

ASJC Scopus subject areas

UN SDGs

Access to Document

Fingerprint

Cite this