Scalability of an Eulerian-Lagrangian large-eddy simulation solver with hybrid MPI/OpenMP parallelisation

Eulerian-Lagrangian approaches capable of accurately reproducing complex fluid flows are becoming more and more popular due to the increasing availability and capacity of High Performance Computing facilities. However, the parallelisation of the Lagrangian part of such methods is challenging when a large number of Lagrangian markers are employed. In this study, a hybrid MPI/OpenMP parallelisation strategy is presented and implemented in a finite difference based large-eddy simulation code featuring the immersed boundary method which generally employs a large number of Lagrangian markers. A master-scattering-gathering strategy is used to deal with the handling of the Lagrangian markers and OpenMP is employed to distribute their computational load across several CPU threads. A classical domain-decomposition-based MPI approach is used to carry out the Eulerian, fixed-mesh fluid calculations. The results demonstrate that by using an effective combination of MPI and OpenMP the code can outperform a pure MPI parallelisation approach by up to 20%. Outcomes from this paper are of interest to various Eulerian-Lagrangian applications including the immersed boundary method, discrete element method or Lagrangian particle tracking.