EMPIRE-PIC: a performance portable unstructured particle-in-cell code

Matthew T. Bettencourt; Dominic A.S. Brown; Keith Cartwright; Eric Cyr; Christian Glusa; Paul Lin; Stan Moore; Duncan McGregor; Roger  Pawlowski; Edward Phillips; Nathan Roberts; Steven A. Wright; Satheesh Maheswaran; John P. Jones; Stephen Jarvis

doi:10.4208/cicp.OA-2020-0261

EMPIRE-PIC: a performance portable unstructured particle-in-cell code

Matthew T. Bettencourt, Dominic A.S. Brown, Keith Cartwright, Eric Cyr, Christian Glusa, Paul Lin, Stan Moore, Duncan McGregor, Roger Pawlowski, Edward Phillips, Nathan Roberts, Steven A. Wright, Satheesh Maheswaran, John P. Jones, Stephen Jarvis

Engineering and Physical Sciences

Research output: Contribution to journal › Article › peer-review

1 Citation (Scopus)

226 Downloads (Pure)

Abstract

In this paper we introduce EMPIRE-PIC, a finite element method particle-in-cell (FEM-PIC) application developed at Sandia National Laboratories. The code has been developed in C++ using the Trilinos library and the Kokkos Performance Portability Framework to enable running on multiple modern compute architectures while only requiring maintenance of a single codebase. EMPIRE-PIC is capable of solving both electrostatic and electromagnetic problems in two- and three-dimensions to second-order accuracy in space and time. In this paper we validate the code against three benchmark problems – a simple electron orbit, an electrostatic Langmuir wave, and a transverse electromagnetic wave propagating through a plasma. We demonstrate the performance of EMPIRE-PIC on four different architectures: Intel Haswell CPUs, Intel’s Xeon Phi Knights Landing, ARM Thunder-X2 CPUs, and NVIDIA Tesla V100 GPUs attached to IBM POWER9 processors. This analysis demonstrates scalability of the code up to more than two thousand GPUs, and greater than one hundred thousand CPUs.

Original language	English
Pages (from-to)	1232-1268
Number of pages	37
Journal	Communications in Computational Physics
Volume	30
Issue number	4
Early online date	10 Aug 2021
DOIs	https://doi.org/10.4208/cicp.OA-2020-0261
Publication status	Published - Oct 2021

Bibliographical note

Funding Information:
This work was supported by the UK Atomic Weapons Establishment (AWE) under grant CDK0724 (AWE Technical Outreach Programme). Professor Stephen Jarvis is an AWE William Penney Fellow.

Sandia National Laboratories is a multimission laboratory managed and operated by National Technology and Engineering Solutions of Sandia, LLC., a wholly owned subsidiary of Honeywell International, Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA-0003525. This paper describes objective technical results and analysis. Any subjective views or opinions that might be expressed in the paper do not necessarily represent the views of the U.S. Department of Energy or the United States Government.

Publisher Copyright:
© 2021 Global-Science Press

Keywords

PIC
electrostatics
electromagnetics
HPC
performance portability

ASJC Scopus subject areas

Physics and Astronomy (miscellaneous)

Access to Document

10.4208/cicp.OA-2020-0261Licence: None: All rights reserved

BrownD2021Higher
First published in: Journal of Computational Physics in 30(4) published by Global Science Press, © 2021 Global-Science Press
Accepted author manuscript, 1.04 MBLicence: None: All rights reserved

Cite this

Bettencourt, M. T., Brown, D. A. S., Cartwright, K., Cyr, E., Glusa, C., Lin, P., Moore, S., McGregor, D., Pawlowski, R., Phillips, E., Roberts, N., Wright, S. A., Maheswaran, S., Jones, J. P., & Jarvis, S. (2021). EMPIRE-PIC: a performance portable unstructured particle-in-cell code. Communications in Computational Physics, 30(4), 1232-1268. Advance online publication. https://doi.org/10.4208/cicp.OA-2020-0261

@article{7319b917c5134b6a88fa0e834051f59f,

title = "EMPIRE-PIC: a performance portable unstructured particle-in-cell code",

abstract = "In this paper we introduce EMPIRE-PIC, a finite element method particle-in-cell (FEM-PIC) application developed at Sandia National Laboratories. The code has been developed in C++ using the Trilinos library and the Kokkos Performance Portability Framework to enable running on multiple modern compute architectures while only requiring maintenance of a single codebase. EMPIRE-PIC is capable of solving both electrostatic and electromagnetic problems in two- and three-dimensions to second-order accuracy in space and time. In this paper we validate the code against three benchmark problems – a simple electron orbit, an electrostatic Langmuir wave, and a transverse electromagnetic wave propagating through a plasma. We demonstrate the performance of EMPIRE-PIC on four different architectures: Intel Haswell CPUs, Intel{\textquoteright}s Xeon Phi Knights Landing, ARM Thunder-X2 CPUs, and NVIDIA Tesla V100 GPUs attached to IBM POWER9 processors. This analysis demonstrates scalability of the code up to more than two thousand GPUs, and greater than one hundred thousand CPUs.",

keywords = "PIC, electrostatics, electromagnetics, HPC, performance portability",

author = "Bettencourt, {Matthew T.} and Brown, {Dominic A.S.} and Keith Cartwright and Eric Cyr and Christian Glusa and Paul Lin and Stan Moore and Duncan McGregor and Roger Pawlowski and Edward Phillips and Nathan Roberts and Wright, {Steven A.} and Satheesh Maheswaran and Jones, {John P.} and Stephen Jarvis",

note = "Funding Information: This work was supported by the UK Atomic Weapons Establishment (AWE) under grant CDK0724 (AWE Technical Outreach Programme). Professor Stephen Jarvis is an AWE William Penney Fellow. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology and Engineering Solutions of Sandia, LLC., a wholly owned subsidiary of Honeywell International, Inc., for the U.S. Department of Energy{\textquoteright}s National Nuclear Security Administration under contract DE-NA-0003525. This paper describes objective technical results and analysis. Any subjective views or opinions that might be expressed in the paper do not necessarily represent the views of the U.S. Department of Energy or the United States Government. Publisher Copyright: {\textcopyright} 2021 Global-Science Press",

year = "2021",

month = oct,

doi = "10.4208/cicp.OA-2020-0261",

language = "English",

volume = "30",

pages = "1232--1268",

journal = "Communications in Computational Physics",

issn = "1815-2406",

publisher = "Global Science Press",

number = "4",

}

Bettencourt, MT, Brown, DAS, Cartwright, K, Cyr, E, Glusa, C, Lin, P, Moore, S, McGregor, D, Pawlowski, R, Phillips, E, Roberts, N, Wright, SA, Maheswaran, S, Jones, JP & Jarvis, S 2021, 'EMPIRE-PIC: a performance portable unstructured particle-in-cell code', Communications in Computational Physics, vol. 30, no. 4, pp. 1232-1268. https://doi.org/10.4208/cicp.OA-2020-0261

TY - JOUR

T1 - EMPIRE-PIC

T2 - a performance portable unstructured particle-in-cell code

AU - Bettencourt, Matthew T.

AU - Brown, Dominic A.S.

AU - Cartwright, Keith

AU - Cyr, Eric

AU - Glusa, Christian

AU - Lin, Paul

AU - Moore, Stan

AU - McGregor, Duncan

AU - Pawlowski, Roger

AU - Phillips, Edward

AU - Roberts, Nathan

AU - Wright, Steven A.

AU - Maheswaran, Satheesh

AU - Jones, John P.

AU - Jarvis, Stephen

N1 - Funding Information: This work was supported by the UK Atomic Weapons Establishment (AWE) under grant CDK0724 (AWE Technical Outreach Programme). Professor Stephen Jarvis is an AWE William Penney Fellow. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology and Engineering Solutions of Sandia, LLC., a wholly owned subsidiary of Honeywell International, Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA-0003525. This paper describes objective technical results and analysis. Any subjective views or opinions that might be expressed in the paper do not necessarily represent the views of the U.S. Department of Energy or the United States Government. Publisher Copyright: © 2021 Global-Science Press

PY - 2021/10

Y1 - 2021/10

N2 - In this paper we introduce EMPIRE-PIC, a finite element method particle-in-cell (FEM-PIC) application developed at Sandia National Laboratories. The code has been developed in C++ using the Trilinos library and the Kokkos Performance Portability Framework to enable running on multiple modern compute architectures while only requiring maintenance of a single codebase. EMPIRE-PIC is capable of solving both electrostatic and electromagnetic problems in two- and three-dimensions to second-order accuracy in space and time. In this paper we validate the code against three benchmark problems – a simple electron orbit, an electrostatic Langmuir wave, and a transverse electromagnetic wave propagating through a plasma. We demonstrate the performance of EMPIRE-PIC on four different architectures: Intel Haswell CPUs, Intel’s Xeon Phi Knights Landing, ARM Thunder-X2 CPUs, and NVIDIA Tesla V100 GPUs attached to IBM POWER9 processors. This analysis demonstrates scalability of the code up to more than two thousand GPUs, and greater than one hundred thousand CPUs.

AB - In this paper we introduce EMPIRE-PIC, a finite element method particle-in-cell (FEM-PIC) application developed at Sandia National Laboratories. The code has been developed in C++ using the Trilinos library and the Kokkos Performance Portability Framework to enable running on multiple modern compute architectures while only requiring maintenance of a single codebase. EMPIRE-PIC is capable of solving both electrostatic and electromagnetic problems in two- and three-dimensions to second-order accuracy in space and time. In this paper we validate the code against three benchmark problems – a simple electron orbit, an electrostatic Langmuir wave, and a transverse electromagnetic wave propagating through a plasma. We demonstrate the performance of EMPIRE-PIC on four different architectures: Intel Haswell CPUs, Intel’s Xeon Phi Knights Landing, ARM Thunder-X2 CPUs, and NVIDIA Tesla V100 GPUs attached to IBM POWER9 processors. This analysis demonstrates scalability of the code up to more than two thousand GPUs, and greater than one hundred thousand CPUs.

KW - PIC

KW - electrostatics

KW - electromagnetics

KW - HPC

KW - performance portability

UR - https://doc.global-sci.org/uploads/online_news/CiCP/76a163bb373694516654c66dc987b7a8.pdf

UR - http://www.scopus.com/inward/record.url?scp=85110747561&partnerID=8YFLogxK

U2 - 10.4208/cicp.OA-2020-0261

DO - 10.4208/cicp.OA-2020-0261

M3 - Article

AN - SCOPUS:85110747561

SN - 1815-2406

VL - 30

SP - 1232

EP - 1268

JO - Communications in Computational Physics

JF - Communications in Computational Physics

IS - 4

ER -

EMPIRE-PIC: a performance portable unstructured particle-in-cell code

Abstract

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Fingerprint

Cite this