EMPIRE-PIC: a performance portable unstructured particle-in-cell code

Matthew T. Bettencourt, Dominic A.S. Brown, Keith Cartwright, Eric Cyr, Christian Glusa, Paul Lin, Stan Moore, Duncan McGregor, Roger Pawlowski, Edward Phillips, Nathan Roberts, Steven A. Wright, Satheesh Maheswaran, John P. Jones, Stephen Jarvis

Research output: Contribution to journalArticlepeer-review

1 Citation (Scopus)
243 Downloads (Pure)


In this paper we introduce EMPIRE-PIC, a finite element method particle-in-cell (FEM-PIC) application developed at Sandia National Laboratories. The code has been developed in C++ using the Trilinos library and the Kokkos Performance Portability Framework to enable running on multiple modern compute architectures while only requiring maintenance of a single codebase. EMPIRE-PIC is capable of solving both electrostatic and electromagnetic problems in two- and three-dimensions to second-order accuracy in space and time. In this paper we validate the code against three benchmark problems – a simple electron orbit, an electrostatic Langmuir wave, and a transverse electromagnetic wave propagating through a plasma. We demonstrate the performance of EMPIRE-PIC on four different architectures: Intel Haswell CPUs, Intel’s Xeon Phi Knights Landing, ARM Thunder-X2 CPUs, and NVIDIA Tesla V100 GPUs attached to IBM POWER9 processors. This analysis demonstrates scalability of the code up to more than two thousand GPUs, and greater than one hundred thousand CPUs.
Original languageEnglish
Pages (from-to)1232-1268
Number of pages37
JournalCommunications in Computational Physics
Issue number4
Early online date10 Aug 2021
Publication statusPublished - Oct 2021

Bibliographical note

Funding Information:
This work was supported by the UK Atomic Weapons Establishment (AWE) under grant CDK0724 (AWE Technical Outreach Programme). Professor Stephen Jarvis is an AWE William Penney Fellow.

Sandia National Laboratories is a multimission laboratory managed and operated by National Technology and Engineering Solutions of Sandia, LLC., a wholly owned subsidiary of Honeywell International, Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA-0003525. This paper describes objective technical results and analysis. Any subjective views or opinions that might be expressed in the paper do not necessarily represent the views of the U.S. Department of Energy or the United States Government.

Publisher Copyright:
© 2021 Global-Science Press


  • PIC
  • electrostatics
  • electromagnetics
  • HPC
  • performance portability

ASJC Scopus subject areas

  • Physics and Astronomy (miscellaneous)


Dive into the research topics of 'EMPIRE-PIC: a performance portable unstructured particle-in-cell code'. Together they form a unique fingerprint.

Cite this