First full-scale 3D high-fidelity simulations of blood flow in the human vasculature

A major effort is now underway to perform the first full-scale 3D high-fidelity simulations of blood flow in the human vasculature. Led by Peter Coveney within his Centre for Computational Science (CCS) at University College London (UCL), this major, large scale team endeavour, involves colleagues and collaborators from across Europe and the USA. The members of the team include UCL, Leibniz Rechenzentrum (LRZ), Jülich Supercomputing Centre (JSC), the IT’IS Foundation and the University of Tennessee at Chattanooga. LRZ, a core partner in our CoE, is providing the use of their new supercomputer, SuperMUC-NG, through a large scale, Gauss Centre for Supercomputing, award for 2019-20 with Prof Dieter Kranzlmüller and Prof Peter Coveney. JSC is involved via our sister (CoE POP) and has been working with us to optimise the performance of the HemeLB code, with Dr Brian Wylie at JSC. The (IT’IS Foundation), based in Zürich, Switzerland, is an associate partner of our CoE and has provided the vascular model data. Dr Jon McCullough is performing the scientific studies within the CCS at UCL which currently address the coupling of arterial and venous trees.

HemeLB is a massively parallel, highly scalable blood flow simulator optimised for sparse geometries such as those that are found in the human vasculature. Currently, it is a plain MPI code which scales impressively on all multicore platforms. With the advent of SuperMUC-NG, #9 in the Top500 list with almost 27 Petaflops performance and over 300k cores, HemeLB has been ported to the behemoth and, after reported very good benchmarks last November over one half of the machine, we were able to get access to the entire production partition a few days ago.

Using a scaling baseline with 18 compute nodes, the smallest possible run with the memory available on SuperMUC-NG, we find a 190x speedup (on 358x cores), which can be seen in the scaling plots of the work. The version of HemeLB, and associated patches, used for this work in addition to the dataset for the scaling runs was provided by Dr Alex Patronis, who worked with Coveney at CCS before taking up a new position at JSC in mid 2019. Essential support was provided from LRZ to enable this opportunity to undertake large runs on SuperMUC-NG and we are grateful for their assistance in this work.

The performance of HemeLB seen in this work on SuperMUC-NG, currently the most powerful machine in the EU, is a very encouraging milestone on the path to the exascale era. We are expecting the first exascale machines to be ready in the USA as early as 2021 and in the EU by 2023. With these and other machines we aim to run the first full scale virtual human simulations and expect such studies to become commonplace at and beyond the exascale.

Peter Coveney likens the endeavour to an assault on Mount Everest (or the North Face of the Eiger if you prefer): “we’ve now left base camp and are moving towards Camp 2…”