Resident block-structured adaptive mesh refinement on thousands of graphics processing units

David Beckingsale, Wayne Gaudin, Andrew Herdman, Stephen Jarvis

Research output: Chapter in Book/Report/Conference proceedingConference contribution

15 Citations (Scopus)


Block-structured adaptive mesh refinement (AMR) is a technique that can be used when solving partial differential equations to reduce the number of cells necessary to achieve the required accuracy in areas of interest. These areas (shock fronts, material interfaces, etc.) are recursively covered with finer mesh patches that are grouped into a hierarchy of refinement levels. Despite the potential for large savings in computational requirements and memory usage without a corresponding reduction in accuracy, AMR adds overhead in managing the mesh hierarchy, adding complex communication and data movement requirements to a simulation. In this paper, we describe the design and implementation of a resident GPU-based AMR library, including: the classes used to manage data on a mesh patch, the routines used for transferring data between GPUs on different nodes, and the data-parallel operators developed to coarsen and refine mesh data. We validate the performance and accuracy of our implementation using three test problems and two architectures: an 8 node cluster, and 4,196 nodes of Oak Ridge National Laboratory's Titan supercomputer. Our GPU-based AMR hydrodynamics code performs up to 4.87× faster than the CPU-based implementation, and is scalable on 4,196 K20x GPUs using a combination of MPI and CUDA.

Original languageEnglish
Title of host publicationProceedings - 2015 44th International Annual Conference on Parallel Processing, ICPP 2015
PublisherInstitute of Electrical and Electronics Engineers (IEEE)
Number of pages10
ISBN (Electronic)9781467375870
Publication statusPublished - 8 Dec 2015
Event44th International Conference on Parallel Processing, ICPP 2015 - Beijing, China
Duration: 1 Sept 20154 Sept 2015

Publication series

NameProceedings of the International Conference on Parallel Processing
ISSN (Print)0190-3918


Conference44th International Conference on Parallel Processing, ICPP 2015

Bibliographical note

Publisher Copyright:
© 2015 IEEE.


  • Adaptive mesh refinement
  • CUDA
  • Hydrodynamics
  • Mini-applications

ASJC Scopus subject areas

  • Software
  • Mathematics(all)
  • Hardware and Architecture


Dive into the research topics of 'Resident block-structured adaptive mesh refinement on thousands of graphics processing units'. Together they form a unique fingerprint.

Cite this