GeoPC

GeoPC: Infrastructure development for hybrid parallel smoothers for multigrid preconditioners

PI: Paul Tackley (ETH Zurich)

Co-PIs: Dave A. May, Olaf Schenk, William B. Sawyer, Matthew G. Knepley, Marcus Grote, Karl Rupp

January 1, 2014 - December 31, 2016

The GeoPC project is developing computational infrastructure to enable massively parallel, scalable smoothers and coarse grid solvers to be used within multigrid preconditioners. This infrastructure is intended to (i) facilitate the execution of high resolution, 3D geodynamic models of the planetary evolution; (ii) provide the Earth Science community (and others) with a suite of continually maintained and re-usable HPC components to build robust multi-level preconditioners (iii) position Swiss computational geosciences in the emerging exa-scale era.

GeoPC is a PASC co-design project involving the University of Lugano (USI), the Swiss Federal Institute of Technology (ETH), University of Chicago (UC), Vienna University of Technology (TU Wien), as well as other stakeholders, and employs two PostDocs for the three-year project duration.

Complex multi-scale interactions characterize the physics of the Earth. Quantification of these interactions is crucial to the integration of coupled geological systems which are today mostly treated in isolation. Resolving structures, processes and dynamics on a wide range of interacting spatio-temporal scales is a unifying grand challenge common to all branches of Earth science and one which must be addressed in order to achieve a comprehensive understanding of the Earth (and other planets) as a multi-physics system. The defining goal of the Domain Science Network "Solid Earth Dynamics" is to address the challenge of multi-scale Earth modeling through a coordinated inter-disciplinary effort combined with long-term application support that ensures a sustained impact on the geoscience community.

In the last forty years, numerical modeling has become a central technique to develop insight into the physics of geological processes and to furthering our understanding of the Earth as a coupled dynamical system linking processes in the core, mantle, lithosphere, topography and atmosphere. Computational geodynamics is focused on modeling the the evolution and deformation of rocks over million year time spans. The long-term evolution is governed via the equations describing the dynamics of incompressible very viscous, creeping flow (i.e. Stokes equations). The rheology of the bulk composition of a given parcel of rock is described by a visco-elasto-plastic material. From laboratory experiments, we know to first order that rock rheology is both highly temperature dependent and a function of both the velocity and pressure. When considering realistic Earth like parameters, such a rheology manifests itself into an effective viscosity (within the deviatoric stress tensor) which possess enormous variations in magnitude (O(1010)) over a range of different length scales, ranging from O(1000) km to O(1) m.

While two-dimensional numerical models have facilitated the characterization of some basic dynamics, mantle convection and plate tectonics are inherently three-dimensional, containing hot cylindrical plumes that generate hotspot volcanoes, transform plate boundaries and toroidal surface motions, and several different types of 3-D convective instability. Achieving sufficient resolution in a global 3D spherical shell domain is extremely challenging; for example using elements the dimension of the oceanic crust (7 km) would require 2.6 billion elements while a uniform global 2 km resolution (as used in some of our regional models) would require 113 billion elements.

The numerical solution of this saddle point problem possessing an elliptic operator with a highly spatially variable, heterogenous coefficient represents > 80% of the total run-time in typically geodynamic simulations. Thus in developing realistic 3D geodynamic models on both a regional and global scale with sufficient rheological complexity, of crucial importance is the preconditioner used to solve the variable viscosity Stokes problem. The strength of the Stokes preconditioner fundamentally controls the scientific throughput achievable and represents the largest bottleneck in the development of our understanding of geodynamic processes.

Within the GeoPC project, we will develop a HPC infrastructure to support the construction of robust multi-level preconditioners suitable for the variable viscosity Stokes problems arising from geodynamic applications. Specifically the developments will focus on multigrid smoothers and coarse grid solvers which utilise hybrid MPI-OpenMP-GPU/MIC parallelism. The hybrid smoothers and coarse grid solvers developed will be included directly within the Portable Extensible Toolkit for Scientific computing (PETSc) library [6, 44]. Testing of the components within complex geodynamic simulations will be facilitated by using an existing parallel 3D finite element geodynamic library which already exploits PETSc infrastructure for the linear and non-linear solvers. Such testing is intended to demonstrate that the developments solve real problems and the methods extend will beyond those of purely academic interest.

The impacts of our project are far reaching due to the generality of the HPC solver components we will develop. Throughout this project, we intend to focus testing and development around regional geodynamic applications. The scalability and parallel efficiency we expect to obtain will for the first time enable, self-consistent 3D geodynamic models which will accurately resolve flow in the mantle, visco-elasto-plastic deformation in the lithosphere and upper/lower crustal scale processes. Being able to simulate the coupled mantle-lithosphere-crustal system over 1-100 million year time scales will provide, for example, a modeling tool which can be used to explore hypotheses related to the formation of the entire European Alps. Beyond our regional scale geodynamic test suite, the inclusion of the hybrid smoothers and coarse grid solvers within PETSc constitutes a long term contribution to the Earth science community. Application areas within the geoscience community which can exploit the tools developed here for solving elliptic PDEs includes; reactive transport modeling in porous media, reservoir and groundwater flow modeling, geoelectric forward and inverse models, geothermal models and potential field methods.