ENIAC

ENIAC: Enabling the ICON model on heterogeneous architectures

PI: Ulrike Lohmann (ETH Zurich)

Co-PIs: Sylvaine Ferrachat, Oliver Fuhrer, Marco Giorgetta, Xavier Lapillonne, Christina Schnadt Poberaj, Bjorn Stevens, Günther Zängl

July 1, 2017 - June 30, 2020

Recent years have seen an immense progress in our capacity to predict weather and climate evolution using numerical models. An important driver for this development has been the rapid progress in high-performance computing (HPC). It is expected that with the further increase of high-performance computing capacity, the computational resolution of such models will continue to be refined in the next decades. This development offers exciting prospects. From a weather and climate science perspective, increasing model resolution will make it possible to base such models on a set of equations that is much closer to first principles. In particular, at horizontal resolutions of O(1 km), the models become cloud resolving and start to explicitly represent the dynamics of deep convective and thunderstorm clouds without the help of semi-empirical parameterizations. This could enable more sophisticated climate-change scenarios with better guidance for impact assessment and climate change adaptation measures. From a computer science perspective, this strategy poses major challenges. First, harvesting the computational capacity of emerging HPC architectures increasingly involves the use of heterogeneous many-core architectures consisting of both CPUs and accelerators (e.g., GPUs). The efficient exploitation of such architectures requires a paradigm shift and has only just started. Second, continuously adapting state-of-the art weather and climate models to emerging and new hardware and the associated programming models poses a formidable challenge, as these models typically entail huge code bases maintained by a large community of domain-scientists.

The application under consideration in this proposal is the Icosahedral Non-hydrostatic (ICON) atmospheric general circulation model, which simulates the time evolution of the atmosphere in a global or regional setting, and can be employed either for climate or numerical weather predictions, depending on its configuration. ICON is currently under development at the German Weather Service (DWD) and the Max Planck Institute for Meteorology (MPI-M) and is already used in production at the DWD. It was introduced into production at the DWD as the primary global model in 2015. In Switzerland, the ICON model will be introduced in climate research at ETH starting from 2017. It is currently planned that ICON will be introduced into the operational weather forecasting system of MeteoSwiss in 2022. In view of the fact that ICON will be widespread on a large spectrum of architectures in the coming years, it is crucial that it incorporates the newest techniques to ensure portable performance. The current ICON code has been developed in Fortran to run on tradiational x86 multi-core systems using on-node OpenMP threading and MPI for distributed memory parallelism, and substantial code refactoring or re-rewrites of some components may be required to address many-core architectures such as Graphical Processing Units (GPUs) or Intel Xeon Phi accelerators. Achieving near-optimal performance on different hardware architectures using the current programming models has proven to be challenging in preliminary studies.

The aim of this project is to adapt the weather and climate code ICON to emerging and future hardware architectures while addressing the issue of performance portability. The focus will be on:

• Porting and optimizing key components of the model to many-core architectures, namely NVIDIA GPUs and Intel Xeon Phi many-core architecture.
• Achieving a high degree of performance portability by:
  • Applying and further developing the existing CLAW source-to-source translation tool, which provides high-level abstractions for weather and climate models.
  • Using alternative approaches such as the GridTools domain-specific language.
• Enabling ICON for the high-priority use-cases in the weather and climate community on emerging heterogeneous high-performance computing platforms such as Piz Daint at CSCS.

The project will build on top of our previous experience porting the COSMO regional weather and climate code to GPU as well as on the outcome of the previous ETH-funded CLAW and PASC GridTools projects. The project team has established and thriving collaborations and the project is embedded in a inter-disciplinary team of domainscientists and computational and computer scientists.

The project is divided in 3 work packages:

• Work package A aims at developing a GPU-enabled and Xeon-Phi optimized version of the ICON atmosphere model for climate simulations. This work package relies on existing compiler technologies only, such as OpenACC directives for the GPU and OpenMP and auto-vectorization for the Xeon-Phi target ensuring to have a usable code at the end of the project. It extends the current OpenACC port of the dynamical core and will have a particular emphasis on the radiation scheme.
• Work package B is devoted to the consolidation and extension of the CLAW tool, which will be applied to the physical parameterizations in order to achieve performance portability. Inputs from work packages A and C will be used to refined required abstractions and optimizations.
• Work package C aims at exploring more aggressive approaches to and performance portability using the domain-specific language of the GridTools library. It considers a complete re-write of some model components, namely the radiation and parts of the dynamics.

To carry out the proposed work, we request funding for 3 Post-Doc positions and a total of 5.5 FTEs. They will be tightly integrated into existing teams of researchers and developers, which already have long-standing collaborations. The total amount of requested funding is complemented by a substantial contribution of in-kind funding. In addition, one of the 3 Post-Doc positions will be funded at 50% by the Max-Plank Institute for Meteorology (MPI-M) in Hamburg. The project will be developed in the framework of the Center for Climate Systems Modeling (C2SM), which has been established to foster and coordinate research activities at ETH, MeteoSwiss, CSCS, as well as in collaboration with MPI-M in Hamburg, Germany, and the German Weather Service (DWD) with the overarching goal to improve our current capability to predict weather and climate.

We expect this project to have a significant impact on several research groups of the PASC community, which run or will run the ICON model, but also at many other important research institutions in Europe. We further expect that the development and extension of the CLAW compiler, which is an open-source software, will have impact on other applications of the PASC community being run at CSCS and in other computing centers in Europe. It is in-line with and a major contribution towards the roadmap of the PASC Climate and Atmospheric Modeling Network.