Replicating HPC I/O workloads with proxy applications

James Dickson; Steven Wright; Satheesh Maheswaran; Andy Herdman; Mark C. Miller; Stephen Jarvis

doi:10.1109/PDSW-DISCS.2016.007

Replicating HPC I/O workloads with proxy applications

James Dickson^*
, Steven Wright
, Satheesh Maheswaran
, Andy Herdman
, Mark C. Miller
, Stephen Jarvis

^*Corresponding author for this work

Engineering and Physical Sciences

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

2 Citations (Scopus)

Abstract

Large scale simulation performance is dependent on a number of components, however the task of investigation and optimization has long favored computational and communication elements above I/O. Manually extracting the pattern of I/O behavior from a parent application is a useful way of working to address performance issues on a per-application basis, but developing workflows with some degree of automation and flexibility provides a more powerful approach to tackling current and future I/O challenges. In this paper we describe a workload replication workflow that extracts the I/O pattern of an application and recreates its behavior with a flexible proxy application. We demonstrate how simple lightweight characterization can be translated to provide an effective representation of a physics application, and show how a proxy replication can be used as a tool for investigating I/O library paradigms.

Original language	English
Title of host publication	Proceedings of PDSW-DISCS 2016
Subtitle of host publication	1st Joint International Workshop on Parallel Data Storage and Data Intensive Scalable Computing Systems - Held in conjunction with SC16: The International Conference for High Performance Computing, Networking, Storage and Analysis
Publisher	Institute of Electrical and Electronics Engineers (IEEE)
Pages	13-18
Number of pages	6
ISBN (Electronic)	9781509052165
DOIs	https://doi.org/10.1109/PDSW-DISCS.2016.007
Publication status	Published - 30 Jan 2017
Event	1st Joint International Workshop on Parallel Data Storage and Data Intensive Scalable Computing Systems, PDSW-DISCS 2016 - Salt Lake City, United States Duration: 14 Nov 2016 → …

Publication series

Name	Proceedings of PDSW-DISCS 2016: 1st Joint International Workshop on Parallel Data Storage and Data Intensive Scalable Computing Systems - Held in conjunction with SC16: The International Conference for High Performance Computing, Networking, Storage and Analysis

Conference

Conference	1st Joint International Workshop on Parallel Data Storage and Data Intensive Scalable Computing Systems, PDSW-DISCS 2016
Country/Territory	United States
City	Salt Lake City
Period	14/11/16 → …

Bibliographical note

Funding Information:
This work was supported in part by AWE under grant CDK0724 (AWE Technical Outreach Programme).

Publisher Copyright:
© 2016 IEEE.

ASJC Scopus subject areas

Hardware and Architecture
Information Systems

Access to Document

10.1109/PDSW-DISCS.2016.007

Cite this

Dickson, J., Wright, S., Maheswaran, S., Herdman, A., Miller, M. C., & Jarvis, S. (2017). Replicating HPC I/O workloads with proxy applications. In Proceedings of PDSW-DISCS 2016: 1st Joint International Workshop on Parallel Data Storage and Data Intensive Scalable Computing Systems - Held in conjunction with SC16: The International Conference for High Performance Computing, Networking, Storage and Analysis (pp. 13-18). Article 7836562 (Proceedings of PDSW-DISCS 2016: 1st Joint International Workshop on Parallel Data Storage and Data Intensive Scalable Computing Systems - Held in conjunction with SC16: The International Conference for High Performance Computing, Networking, Storage and Analysis). Institute of Electrical and Electronics Engineers (IEEE). https://doi.org/10.1109/PDSW-DISCS.2016.007

Dickson, James ; Wright, Steven ; Maheswaran, Satheesh et al. / Replicating HPC I/O workloads with proxy applications. Proceedings of PDSW-DISCS 2016: 1st Joint International Workshop on Parallel Data Storage and Data Intensive Scalable Computing Systems - Held in conjunction with SC16: The International Conference for High Performance Computing, Networking, Storage and Analysis. Institute of Electrical and Electronics Engineers (IEEE), 2017. pp. 13-18 (Proceedings of PDSW-DISCS 2016: 1st Joint International Workshop on Parallel Data Storage and Data Intensive Scalable Computing Systems - Held in conjunction with SC16: The International Conference for High Performance Computing, Networking, Storage and Analysis).

@inproceedings{c1413ca558144c33841ef680f0567f38,

title = "Replicating HPC I/O workloads with proxy applications",

abstract = "Large scale simulation performance is dependent on a number of components, however the task of investigation and optimization has long favored computational and communication elements above I/O. Manually extracting the pattern of I/O behavior from a parent application is a useful way of working to address performance issues on a per-application basis, but developing workflows with some degree of automation and flexibility provides a more powerful approach to tackling current and future I/O challenges. In this paper we describe a workload replication workflow that extracts the I/O pattern of an application and recreates its behavior with a flexible proxy application. We demonstrate how simple lightweight characterization can be translated to provide an effective representation of a physics application, and show how a proxy replication can be used as a tool for investigating I/O library paradigms.",

author = "James Dickson and Steven Wright and Satheesh Maheswaran and Andy Herdman and Miller, \{Mark C.\} and Stephen Jarvis",

note = "Funding Information: This work was supported in part by AWE under grant CDK0724 (AWE Technical Outreach Programme). Publisher Copyright: {\textcopyright} 2016 IEEE.; 1st Joint International Workshop on Parallel Data Storage and Data Intensive Scalable Computing Systems, PDSW-DISCS 2016 ; Conference date: 14-11-2016",

year = "2017",

month = jan,

day = "30",

doi = "10.1109/PDSW-DISCS.2016.007",

language = "English",

series = "Proceedings of PDSW-DISCS 2016: 1st Joint International Workshop on Parallel Data Storage and Data Intensive Scalable Computing Systems - Held in conjunction with SC16: The International Conference for High Performance Computing, Networking, Storage and Analysis",

publisher = "Institute of Electrical and Electronics Engineers (IEEE)",

pages = "13--18",

booktitle = "Proceedings of PDSW-DISCS 2016",

}

Dickson, J, Wright, S, Maheswaran, S, Herdman, A, Miller, MC & Jarvis, S 2017, Replicating HPC I/O workloads with proxy applications. in Proceedings of PDSW-DISCS 2016: 1st Joint International Workshop on Parallel Data Storage and Data Intensive Scalable Computing Systems - Held in conjunction with SC16: The International Conference for High Performance Computing, Networking, Storage and Analysis., 7836562, Proceedings of PDSW-DISCS 2016: 1st Joint International Workshop on Parallel Data Storage and Data Intensive Scalable Computing Systems - Held in conjunction with SC16: The International Conference for High Performance Computing, Networking, Storage and Analysis, Institute of Electrical and Electronics Engineers (IEEE), pp. 13-18, 1st Joint International Workshop on Parallel Data Storage and Data Intensive Scalable Computing Systems, PDSW-DISCS 2016, Salt Lake City, United States, 14/11/16. https://doi.org/10.1109/PDSW-DISCS.2016.007

Replicating HPC I/O workloads with proxy applications. / Dickson, James; Wright, Steven; Maheswaran, Satheesh et al.
Proceedings of PDSW-DISCS 2016: 1st Joint International Workshop on Parallel Data Storage and Data Intensive Scalable Computing Systems - Held in conjunction with SC16: The International Conference for High Performance Computing, Networking, Storage and Analysis. Institute of Electrical and Electronics Engineers (IEEE), 2017. p. 13-18 7836562 (Proceedings of PDSW-DISCS 2016: 1st Joint International Workshop on Parallel Data Storage and Data Intensive Scalable Computing Systems - Held in conjunction with SC16: The International Conference for High Performance Computing, Networking, Storage and Analysis).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Replicating HPC I/O workloads with proxy applications

AU - Dickson, James

AU - Wright, Steven

AU - Maheswaran, Satheesh

AU - Herdman, Andy

AU - Miller, Mark C.

AU - Jarvis, Stephen

PY - 2017/1/30

Y1 - 2017/1/30

N2 - Large scale simulation performance is dependent on a number of components, however the task of investigation and optimization has long favored computational and communication elements above I/O. Manually extracting the pattern of I/O behavior from a parent application is a useful way of working to address performance issues on a per-application basis, but developing workflows with some degree of automation and flexibility provides a more powerful approach to tackling current and future I/O challenges. In this paper we describe a workload replication workflow that extracts the I/O pattern of an application and recreates its behavior with a flexible proxy application. We demonstrate how simple lightweight characterization can be translated to provide an effective representation of a physics application, and show how a proxy replication can be used as a tool for investigating I/O library paradigms.

AB - Large scale simulation performance is dependent on a number of components, however the task of investigation and optimization has long favored computational and communication elements above I/O. Manually extracting the pattern of I/O behavior from a parent application is a useful way of working to address performance issues on a per-application basis, but developing workflows with some degree of automation and flexibility provides a more powerful approach to tackling current and future I/O challenges. In this paper we describe a workload replication workflow that extracts the I/O pattern of an application and recreates its behavior with a flexible proxy application. We demonstrate how simple lightweight characterization can be translated to provide an effective representation of a physics application, and show how a proxy replication can be used as a tool for investigating I/O library paradigms.

UR - https://www.scopus.com/pages/publications/85015324682

U2 - 10.1109/PDSW-DISCS.2016.007

DO - 10.1109/PDSW-DISCS.2016.007

M3 - Conference contribution

AN - SCOPUS:85015324682

T3 - Proceedings of PDSW-DISCS 2016: 1st Joint International Workshop on Parallel Data Storage and Data Intensive Scalable Computing Systems - Held in conjunction with SC16: The International Conference for High Performance Computing, Networking, Storage and Analysis

SP - 13

EP - 18

BT - Proceedings of PDSW-DISCS 2016

PB - Institute of Electrical and Electronics Engineers (IEEE)

T2 - 1st Joint International Workshop on Parallel Data Storage and Data Intensive Scalable Computing Systems, PDSW-DISCS 2016

Y2 - 14 November 2016

ER -

Dickson J, Wright S, Maheswaran S, Herdman A, Miller MC, Jarvis S. Replicating HPC I/O workloads with proxy applications. In Proceedings of PDSW-DISCS 2016: 1st Joint International Workshop on Parallel Data Storage and Data Intensive Scalable Computing Systems - Held in conjunction with SC16: The International Conference for High Performance Computing, Networking, Storage and Analysis. Institute of Electrical and Electronics Engineers (IEEE). 2017. p. 13-18. 7836562. (Proceedings of PDSW-DISCS 2016: 1st Joint International Workshop on Parallel Data Storage and Data Intensive Scalable Computing Systems - Held in conjunction with SC16: The International Conference for High Performance Computing, Networking, Storage and Analysis). doi: 10.1109/PDSW-DISCS.2016.007

Replicating HPC I/O workloads with proxy applications

Abstract

Publication series

Conference

Bibliographical note

ASJC Scopus subject areas

Access to Document

Fingerprint

Cite this