Predictive analysis of large-scale coupled CFD simulations with the CPX mini-app

Archie Powell; Karim Choudry; Arun Prabhakar; Istvan Z. Reguly; Dario Amirante; Stephen Jarvis; Gihan R. Mudalige

doi:10.1109/HiPC53243.2021.00028

Predictive analysis of large-scale coupled CFD simulations with the CPX mini-app

Archie Powell, Karim Choudry, Arun Prabhakar, Istvan Z. Reguly, Dario Amirante, Stephen Jarvis, Gihan R. Mudalige

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

2 Citations (Scopus)

Abstract

As the complexity of multi-physics simulations increases, there is a need for efficient flow of information between components. Discrete ‘coupler’ codes can abstract away this process, improving solver interoperability. One such multi-physics problem is modelling the high pressure compressor of turbofan engines, where instances of rotor/stator CFD simulations are coupled. Configuring couplers and allocating resources correctly can be challenging for such problems due to the sliding interfaces between codes. In this research, we present CPX, a mini-coupler designed to model the performance behaviour of a production coupler framework at Rolls-Royce plc., used for coupling rotor/stator simulations. CPX, the first mini-coupler framework of its kind, is combined with a CFD mini-app to predict the run-time and scaling behaviour of large scale coupled CFD simulations. We demonstrate high qualitative and quantitative predictive accuracy with a less than 17 % mean error. A performance model is developed to predict the ‘optimum’ configuration of resources, and is tested to show the high accuracy of these predictions. The model is also used to project the ‘optimum’ configuration for a 6 Billion cell test case, a problem size representative of current leading-edge production workloads, on a 100,000 core cluster and a 400 GPU cluster. Further testing reveals that the ‘optimum’ configuration is unstable if not set up correctly, and therefore a trade-off needs to be made with a marginally less-than-optimal setup to ensure stability. The work illustrates the significant utility of CPX to carry out such rapid design space and run-time setup exploration studies to obtain the best performance from production CFD coupled simulations.

Original language	English
Title of host publication	2021 IEEE 28th International Conference on High Performance Computing, Data, and Analytics (HiPC)
Publisher	IEEE
Pages	141-151
Number of pages	11
ISBN (Electronic)	9781665410168
ISBN (Print)	9781665410175
DOIs	https://doi.org/10.1109/HiPC53243.2021.00028
Publication status	Published - 24 Jan 2022
Event	28th IEEE International Conference on High Performance Computing, Data, and Analytics, HiPC 2021 - Virtual, Bangalore, India Duration: 17 Dec 2021 → 18 Dec 2021

Publication series

Name	International Conference on High Performance Computing
ISSN (Print)	1094-7256
ISSN (Electronic)	2640-0316

Conference

Conference	28th IEEE International Conference on High Performance Computing, Data, and Analytics, HiPC 2021
Country/Territory	India
City	Virtual, Bangalore
Period	17/12/21 → 18/12/21

Bibliographical note

Funding Information:
There are several areas that can be considered further in future work. One obvious area is to improve the search algorithm, as the brute force approach currently used is inefficient. Work has already begun on implementing a tree-based method which will significantly reduce this search time. A key issue with the RR coupler is the time-consuming process of manually partitioning the interfaces. Automating the partitioning process can improve the utility of the performance model, as well as significantly reducing the setup time. Additionally, since the RR Coupler supports the use of conjugate heat transfer as the interface between CFD models, CHT test cases can be explored in CPX by removing the need to execute the interface search routines that must occur when coupling sliding planes. Furthermore, we plan to extend the functionality of CPX so that it supports the creation of proxy configurations that are representative of more complex coupling scenarios. This includes the coupling of MG-CFD with a combustion mini-app to model the interaction between the engine compressor and the combustion chamber, as well as functionality that allows for coupling between CFD and FEM solvers. We also plan to extend the performance model to determine optimum run-time configurations for these new solvers, and carry out testing CPX with the GPU versions of MG-CFD to validate the GPU run-times predicted by the model. These extensions will allow for proxy configurations that are more representative (in terms of performance) of a complete gas turbine engine simulation to be created using CPX. Finally, the prediction accuracy of the mini-coupler motivates its use in predicting the cost and performance of simulations executed with the RR Coupler on various open platforms and systems, such as cloud instances, which is not possible with the full code due to licensing restrictions. The CPX coupler is available as open-source software at [27]. ACKNOWLEDGEMENTS This research is supported by Rolls-Royce plc., and by the UK EPSRC (EP/S005072/1 – Strategic Partnership in Computational Science for Advanced Simulation and Modelling of Engineering Systems – ASiMoV). Gihan Mudalige was supported by the Royal Society Industry Fellowship Scheme (INF/R1/1800 12). We would also like to thank Chris Goddard at Rolls-Royce for their guidance for this work.

Publisher Copyright:
© 2021 IEEE.

Keywords

CFD
Coupling
Mini-App
Performance model

ASJC Scopus subject areas

Artificial Intelligence
Computer Networks and Communications
Computer Science Applications
Hardware and Architecture
Information Systems

Access to Document

10.1109/HiPC53243.2021.00028

Cite this

Powell, A., Choudry, K., Prabhakar, A., Reguly, I. Z., Amirante, D., Jarvis, S., & Mudalige, G. R. (2022). Predictive analysis of large-scale coupled CFD simulations with the CPX mini-app. In 2021 IEEE 28th International Conference on High Performance Computing, Data, and Analytics (HiPC) (pp. 141-151). Article 9680414 (International Conference on High Performance Computing). IEEE. https://doi.org/10.1109/HiPC53243.2021.00028

@inproceedings{654d23e435fe46c894d681944eeab0aa,

title = "Predictive analysis of large-scale coupled CFD simulations with the CPX mini-app",

abstract = "As the complexity of multi-physics simulations increases, there is a need for efficient flow of information between components. Discrete {\textquoteleft}coupler{\textquoteright} codes can abstract away this process, improving solver interoperability. One such multi-physics problem is modelling the high pressure compressor of turbofan engines, where instances of rotor/stator CFD simulations are coupled. Configuring couplers and allocating resources correctly can be challenging for such problems due to the sliding interfaces between codes. In this research, we present CPX, a mini-coupler designed to model the performance behaviour of a production coupler framework at Rolls-Royce plc., used for coupling rotor/stator simulations. CPX, the first mini-coupler framework of its kind, is combined with a CFD mini-app to predict the run-time and scaling behaviour of large scale coupled CFD simulations. We demonstrate high qualitative and quantitative predictive accuracy with a less than 17 % mean error. A performance model is developed to predict the {\textquoteleft}optimum{\textquoteright} configuration of resources, and is tested to show the high accuracy of these predictions. The model is also used to project the {\textquoteleft}optimum{\textquoteright} configuration for a 6 Billion cell test case, a problem size representative of current leading-edge production workloads, on a 100,000 core cluster and a 400 GPU cluster. Further testing reveals that the {\textquoteleft}optimum{\textquoteright} configuration is unstable if not set up correctly, and therefore a trade-off needs to be made with a marginally less-than-optimal setup to ensure stability. The work illustrates the significant utility of CPX to carry out such rapid design space and run-time setup exploration studies to obtain the best performance from production CFD coupled simulations.",

keywords = "CFD, Coupling, Mini-App, Performance model",

author = "Archie Powell and Karim Choudry and Arun Prabhakar and Reguly, {Istvan Z.} and Dario Amirante and Stephen Jarvis and Mudalige, {Gihan R.}",

note = "Funding Information: There are several areas that can be considered further in future work. One obvious area is to improve the search algorithm, as the brute force approach currently used is inefficient. Work has already begun on implementing a tree-based method which will significantly reduce this search time. A key issue with the RR coupler is the time-consuming process of manually partitioning the interfaces. Automating the partitioning process can improve the utility of the performance model, as well as significantly reducing the setup time. Additionally, since the RR Coupler supports the use of conjugate heat transfer as the interface between CFD models, CHT test cases can be explored in CPX by removing the need to execute the interface search routines that must occur when coupling sliding planes. Furthermore, we plan to extend the functionality of CPX so that it supports the creation of proxy configurations that are representative of more complex coupling scenarios. This includes the coupling of MG-CFD with a combustion mini-app to model the interaction between the engine compressor and the combustion chamber, as well as functionality that allows for coupling between CFD and FEM solvers. We also plan to extend the performance model to determine optimum run-time configurations for these new solvers, and carry out testing CPX with the GPU versions of MG-CFD to validate the GPU run-times predicted by the model. These extensions will allow for proxy configurations that are more representative (in terms of performance) of a complete gas turbine engine simulation to be created using CPX. Finally, the prediction accuracy of the mini-coupler motivates its use in predicting the cost and performance of simulations executed with the RR Coupler on various open platforms and systems, such as cloud instances, which is not possible with the full code due to licensing restrictions. The CPX coupler is available as open-source software at [27]. ACKNOWLEDGEMENTS This research is supported by Rolls-Royce plc., and by the UK EPSRC (EP/S005072/1 – Strategic Partnership in Computational Science for Advanced Simulation and Modelling of Engineering Systems – ASiMoV). Gihan Mudalige was supported by the Royal Society Industry Fellowship Scheme (INF/R1/1800 12). We would also like to thank Chris Goddard at Rolls-Royce for their guidance for this work. Publisher Copyright: {\textcopyright} 2021 IEEE.; 28th IEEE International Conference on High Performance Computing, Data, and Analytics, HiPC 2021 ; Conference date: 17-12-2021 Through 18-12-2021",

year = "2022",

month = jan,

day = "24",

doi = "10.1109/HiPC53243.2021.00028",

language = "English",

isbn = "9781665410175",

series = "International Conference on High Performance Computing",

publisher = "IEEE",

pages = "141--151",

booktitle = "2021 IEEE 28th International Conference on High Performance Computing, Data, and Analytics (HiPC)",

}

Powell, A, Choudry, K, Prabhakar, A, Reguly, IZ, Amirante, D, Jarvis, S & Mudalige, GR 2022, Predictive analysis of large-scale coupled CFD simulations with the CPX mini-app. in 2021 IEEE 28th International Conference on High Performance Computing, Data, and Analytics (HiPC)., 9680414, International Conference on High Performance Computing, IEEE, pp. 141-151, 28th IEEE International Conference on High Performance Computing, Data, and Analytics, HiPC 2021, Virtual, Bangalore, India, 17/12/21. https://doi.org/10.1109/HiPC53243.2021.00028

Predictive analysis of large-scale coupled CFD simulations with the CPX mini-app. / Powell, Archie; Choudry, Karim; Prabhakar, Arun et al.
2021 IEEE 28th International Conference on High Performance Computing, Data, and Analytics (HiPC). IEEE, 2022. p. 141-151 9680414 (International Conference on High Performance Computing).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Predictive analysis of large-scale coupled CFD simulations with the CPX mini-app

AU - Powell, Archie

AU - Choudry, Karim

AU - Prabhakar, Arun

AU - Reguly, Istvan Z.

AU - Amirante, Dario

AU - Jarvis, Stephen

AU - Mudalige, Gihan R.

N1 - Funding Information: There are several areas that can be considered further in future work. One obvious area is to improve the search algorithm, as the brute force approach currently used is inefficient. Work has already begun on implementing a tree-based method which will significantly reduce this search time. A key issue with the RR coupler is the time-consuming process of manually partitioning the interfaces. Automating the partitioning process can improve the utility of the performance model, as well as significantly reducing the setup time. Additionally, since the RR Coupler supports the use of conjugate heat transfer as the interface between CFD models, CHT test cases can be explored in CPX by removing the need to execute the interface search routines that must occur when coupling sliding planes. Furthermore, we plan to extend the functionality of CPX so that it supports the creation of proxy configurations that are representative of more complex coupling scenarios. This includes the coupling of MG-CFD with a combustion mini-app to model the interaction between the engine compressor and the combustion chamber, as well as functionality that allows for coupling between CFD and FEM solvers. We also plan to extend the performance model to determine optimum run-time configurations for these new solvers, and carry out testing CPX with the GPU versions of MG-CFD to validate the GPU run-times predicted by the model. These extensions will allow for proxy configurations that are more representative (in terms of performance) of a complete gas turbine engine simulation to be created using CPX. Finally, the prediction accuracy of the mini-coupler motivates its use in predicting the cost and performance of simulations executed with the RR Coupler on various open platforms and systems, such as cloud instances, which is not possible with the full code due to licensing restrictions. The CPX coupler is available as open-source software at [27]. ACKNOWLEDGEMENTS This research is supported by Rolls-Royce plc., and by the UK EPSRC (EP/S005072/1 – Strategic Partnership in Computational Science for Advanced Simulation and Modelling of Engineering Systems – ASiMoV). Gihan Mudalige was supported by the Royal Society Industry Fellowship Scheme (INF/R1/1800 12). We would also like to thank Chris Goddard at Rolls-Royce for their guidance for this work. Publisher Copyright: © 2021 IEEE.

PY - 2022/1/24

Y1 - 2022/1/24

N2 - As the complexity of multi-physics simulations increases, there is a need for efficient flow of information between components. Discrete ‘coupler’ codes can abstract away this process, improving solver interoperability. One such multi-physics problem is modelling the high pressure compressor of turbofan engines, where instances of rotor/stator CFD simulations are coupled. Configuring couplers and allocating resources correctly can be challenging for such problems due to the sliding interfaces between codes. In this research, we present CPX, a mini-coupler designed to model the performance behaviour of a production coupler framework at Rolls-Royce plc., used for coupling rotor/stator simulations. CPX, the first mini-coupler framework of its kind, is combined with a CFD mini-app to predict the run-time and scaling behaviour of large scale coupled CFD simulations. We demonstrate high qualitative and quantitative predictive accuracy with a less than 17 % mean error. A performance model is developed to predict the ‘optimum’ configuration of resources, and is tested to show the high accuracy of these predictions. The model is also used to project the ‘optimum’ configuration for a 6 Billion cell test case, a problem size representative of current leading-edge production workloads, on a 100,000 core cluster and a 400 GPU cluster. Further testing reveals that the ‘optimum’ configuration is unstable if not set up correctly, and therefore a trade-off needs to be made with a marginally less-than-optimal setup to ensure stability. The work illustrates the significant utility of CPX to carry out such rapid design space and run-time setup exploration studies to obtain the best performance from production CFD coupled simulations.

AB - As the complexity of multi-physics simulations increases, there is a need for efficient flow of information between components. Discrete ‘coupler’ codes can abstract away this process, improving solver interoperability. One such multi-physics problem is modelling the high pressure compressor of turbofan engines, where instances of rotor/stator CFD simulations are coupled. Configuring couplers and allocating resources correctly can be challenging for such problems due to the sliding interfaces between codes. In this research, we present CPX, a mini-coupler designed to model the performance behaviour of a production coupler framework at Rolls-Royce plc., used for coupling rotor/stator simulations. CPX, the first mini-coupler framework of its kind, is combined with a CFD mini-app to predict the run-time and scaling behaviour of large scale coupled CFD simulations. We demonstrate high qualitative and quantitative predictive accuracy with a less than 17 % mean error. A performance model is developed to predict the ‘optimum’ configuration of resources, and is tested to show the high accuracy of these predictions. The model is also used to project the ‘optimum’ configuration for a 6 Billion cell test case, a problem size representative of current leading-edge production workloads, on a 100,000 core cluster and a 400 GPU cluster. Further testing reveals that the ‘optimum’ configuration is unstable if not set up correctly, and therefore a trade-off needs to be made with a marginally less-than-optimal setup to ensure stability. The work illustrates the significant utility of CPX to carry out such rapid design space and run-time setup exploration studies to obtain the best performance from production CFD coupled simulations.

KW - CFD

KW - Coupling

KW - Mini-App

KW - Performance model

UR - http://www.scopus.com/inward/record.url?scp=85125677109&partnerID=8YFLogxK

U2 - 10.1109/HiPC53243.2021.00028

DO - 10.1109/HiPC53243.2021.00028

M3 - Conference contribution

AN - SCOPUS:85125677109

SN - 9781665410175

T3 - International Conference on High Performance Computing

SP - 141

EP - 151

BT - 2021 IEEE 28th International Conference on High Performance Computing, Data, and Analytics (HiPC)

PB - IEEE

T2 - 28th IEEE International Conference on High Performance Computing, Data, and Analytics, HiPC 2021

Y2 - 17 December 2021 through 18 December 2021

ER -

Powell A, Choudry K, Prabhakar A, Reguly IZ, Amirante D, Jarvis S et al. Predictive analysis of large-scale coupled CFD simulations with the CPX mini-app. In 2021 IEEE 28th International Conference on High Performance Computing, Data, and Analytics (HiPC). IEEE. 2022. p. 141-151. 9680414. (International Conference on High Performance Computing). doi: 10.1109/HiPC53243.2021.00028

Predictive analysis of large-scale coupled CFD simulations with the CPX mini-app

Abstract

Publication series

Conference

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Fingerprint

Cite this