Coupled Cardiac Simulations

Integrative HPC Framework for Coupled Cardiac Simulations IFCCS

PI: Rolf Krause (Università della Svizzera italiana), Alfio Quarteroni (EPFL)

Co-PIs: Angelo Auricchio (Fondazione Cardiocentro, Ticino), Simone Deparis (EPFL), Mark Potse (Università della Svizzera italiana)

July 1, 2014 - June 30, 2017

The goal of this project is to develop and realize a community-based, flexible and validated, Integrative HPC Framework for Coupled Cardiac Simulations (IFCCS) within the PASC network Life Sciences Across Scales. IFCCSwill provide a software library oriented towards current and future hybrid architectures, which combines computational models for cardiac electrophysiology, cardiac tissue mechanics, and blood dynamics. With this aim, we will focus on the co-design of new (as well as the improvement of existing) optimal preconditioners widely used in cardiovascular simulations, like Algebraic and Geometric Multigrid Methods and Additive Schwarz (AS) methods. The design of our library will explicitely allow for hybrid parallelism (MPI/multi-threaded/accelerator). The resulting software implementing the new family of algorithms will be inserted in the general Trilinos framework and could also utilize accelerator-specific libraries like the AmgX framework designed by Nvidia. Testing will be done through the LIFEV library and PROPAG- 5, both production tools for Coupled multiphysics Cardiac Simulations. These codes are well established in the scientific community and implement state-of-the-art methods for high-performance computing. The structure of the heart is complex and of multiscale nature; to achieve a reasonable time to solution for high-fidelity simulations it is necessary to combine numerical discretization beyond the state of the art and parallel solution algorithms with the next generation supercomputers. This includes the development of advanced preconditioning and solution techniques, which have to be designed with the availaible supercomputing hardware and software in mind.

The main task of the proposed project is to design and realize new solution methods in strong co-design relation with hybrid systems and to create a widely usable software framework for coupled simulations of complete heart function. This includes multiple levels of parallelism and the creation of an integrated framework, as well as validation in a clinically relevant context. The ultimate goal of this framework is to allow for multi-physics simulation (electro-mechanics-fluid) as well as for partially coupled (electro-mechanics, mechanics-fluid) simulations in a transparent manner. In this way, depending on the cardiac disease under consideration and its possible treatment strategies, the most effective computational model will be readily at hand.

LIFEV (EPFL-CMCS) is a parallel finite-element library of numerical methods including multi-scale and multi-physics models for cardiovascular applications. LIFEV is organized in modules, for example a core module with the basic finite-element classes and physics-specific modules, e.g. structure, Navier- Stokes, and Fluid-Structure Interaction (FSI). It provides its own set of preconditioners designed for saddle-point and fluid-structure interaction problems based on the packages in Trilinos. In the AS context, one subdomain is assigned to each MPI process. We are currently developing multiple levels of parallelism, which allows, within AS, to uncouple the number of parallel processes from the number of subdomains used in the domain-decomposition schemes. In this way, it is possible to use parallel solvers (either preconditioned local GMRES or factorization methods) for subdomain problems. Recent developments identified the kernels which, on a single subdomain, can be parallelized further using either multiple threads or subgroups of MPI processes. Using multiple threads allows for a reduction of memory requirements and MPI communication at the expense of identifying race conditions and atomic operations. Further developments in this direction are necessary to make use of many threads, either on many-core architectures or on GPUs. This will include code refactoring and possibly rethinking of algorithms or numerical schemes. PROPAG-5 (USI-ICS) is a program for large-scale simulations of cardiac electrophysiology. It has been used in several clinical applications. The code is parallelized with a hybrid MPI-OpenMP scheme and uses a semi-structured mesh for highly efficient simulation of the activation spreading and ion channel dynamics in the heart.

On the software side, PROPAG-5 already has undergone significant improvement during the last years at ICS. Parallel I/O has been optimized. Additionally, the data structures for the linear algebra have been separated from the mesh data structures, thus adding more flexibility for different discretization approaches. PROPAG-5 has been redesigned for massively parallel clustered multicore architectures by combining MPI and OpenMP and it has been tested intensively as well as been used for several production-type projects at CSCS. These changes provide a perfect basis in order to switch from the currently used discretization in space, which is done by finite differences, to a finite-element discretization. As mesh management and linear algebra are already separated, only the assembling routines for the stiffness matrix have to be redesigned. This first step in the refactoring process will allow for a straightforward coupling with LIFEV. In a second step, based on the in-house experience with working on GPUs, we intend to move the computationally intensive computation of the ion channels to accelerators, thus exploiting an additional level of the available parallelism on modern machines such as Piz Daint. Here, we will develop an abstract interface to accelerators (starting from the ion channels), which will give us flexibility for future hardware changes. On the technical side, the ICS library CUTLIB exploits recent methodological and software developments for the coupling of discretization methods, which will allow for a monolithic solution approach for the resulting coupled nonlinear systems as well as for partitioned approaches. This library allows for the efficient detection of intersections between arbitrarily distributed and non-matching meshes, and works in parallel itself. Refactoring techniques will be applied in order to achieve a seamless integration of the existing codes. In order to ensure applicability on future HPC platforms, we will separate computationally costly kernels such as the integration of the ion-channel models, in order to allow for GPU-based and accelerator-based implementations. We will also focus on developing new multi-physics coupling algorithms that can couple together the three different subsystems in a stable yet efficient way. The resulting software platform will be made available as a community code with the ultimate goal to further foster the growth of a Swiss-based, but internationally open community for cardiac simulation. Particular emphasis will be put on ensuring that eventually our computational framework can be embedded seamlessly into a clinical workflow, ranging from medical images and computational geometries to computer simulations and, finally, treatment planning. To suitably exploit the results of our collaboration, clinically relevant validation cases will be set up together with medical doctors.