ALBERT

Description: Water bodies influence local weather and climate, especially in lake-rich areas. The FLake (Fresh-water Lake model) parameterisation is employed in the Integrated Forecasting System (IFS) of the European Centre for Medium-Range Weather Forecasts (ECMWF) model which is used operationally to produce global weather predictions. Lake depth and lake fraction are the main driving parameters in the FLake parameterisation. The lake parameter fields for the IFS should be global and realistic, because FLake runs over all the grid boxes, and then only lake-related results are used further. In this study new datasets and methods for generating lake fraction and lake depth fields for the IFS are proposed. The data include the new version of the Global Lake Database (GLDBv3) which contains depth estimates for unstudied lakes based on a geological approach, the General Bathymetric Chart of the Oceans and the Global Surface Water Explorer dataset which contains information on the spatial and temporal variability of surface water. The first new method suggested is a two-step lake fraction calculation; the first step is at 1 km grid resolution and the second is at the resolution of other grids in the IFS system. The second new method involves the use of a novel algorithm for ocean and inland water separation. This new algorithm may be used by anyone in the environmental modelling community. To assess the impact of using these innovations, in situ measurements of lake depth, lake water surface temperature and ice formation/disappearance dates for 27 lakes collected by the Finnish Environment Institute were used. A set of offline experiments driven by atmospheric forcing from the ECMWF ERA5 Reanalysis were carried out using the IFS HTESSEL land surface model. In terms of lake depth, the new dataset shows a much lower mean absolute error, bias and error standard deviation compared to the reference set-up. In terms of lake water surface temperature, the mean absolute error is reduced by 13.4 %, the bias by 12.5 % and the error standard deviation by 20.3 %. Seasonal verification of the mixed layer depth temperature and ice formation/disappearance dates revealed a cold bias in the meteorological forcing from ERA5. Spring, summer and autumn verification scores confirm an overall reduction in the surface water temperature errors. For winter, no statistically significant change in the ice formation/disappearance date errors was detected.

Description: Weather and climate models are complex pieces of software which include many individual components, each of which is evolving under the pressure to exploit advances in computing to enhance some combination of a range of possible improvements (higher spatio/temporal resolution, increased fidelity in terms of resolved processes, more quantification of uncertainty etc). However, after many years of a relatively stable computing environment with little choice in processing architecture or programming paradigm (basically X86 processors using MPI for parallelism), the existing menu of processor choices includes significant diversity, and more is on the horizon. This computational diversity, coupled with ever increasing software complexity, leads to the very real possibility that weather and climate modelling will arrive at a chasm which will separate scientific aspiration from our ability to develop and/or rapidly adapt codes to the available hardware. In this paper we review the hardware and software trends which are leading us towards this chasm, before describing current progress in addressing some of the tools which we may be able to use to bridge the chasm. This brief introduction to current tools and plans is followed by a discussion outlining the scientific requirements for quality model codes which have satisfactory performance and portability, while simultaneously supporting productive scientific evolution. We assert that the existing method of incremental model improvements employing small steps which adjust to the changing hardware environment is likely to be inadequate for crossing the chasm between aspiration and hardware at a satisfactory pace, in part because institutions cannot have all the relevant expertise in house. Instead, we outline a methodology based on large community efforts in engineering and standardisation, one which will depend on identifying a taxonomy of key activities – perhaps based on existing efforts to develop domain specific languages, identify common patterns in weather and climate codes, and develop community approaches to commonly needed tools, libraries etc – and then collaboratively building up those key components. Such a collaborative approach will depend on institutions, projects and individuals adopting new interdependencies and ways of working.

Description: In the simulation of complex multi-scale flows arising in weather and climate modelling, one of the biggest challenges is to satisfy strict service requirements in terms of time to solution and to satisfy budgetary constraints in terms of energy to solution, without compromising the accuracy and stability of the application. These simulations require algorithms that minimise the energy footprint along with the time required to produce a solution, maintain the physically required level of accuracy, are numerically stable, and are resilient in case of hardware failure. The European Centre for Medium-Range Weather Forecasts (ECMWF) led the ESCAPE (Energy-efficient Scalable Algorithms for Weather Prediction at Exascale) project, funded by Horizon 2020 (H2020) under the FET-HPC (Future and Emerging Technologies in High Performance Computing) initiative. The goal of ESCAPE was to develop a sustainable strategy to evolve weather and climate prediction models to next-generation computing technologies. The project partners incorporate the expertise of leading European regional forecasting consortia, university research, experienced high-performance computing centres, and hardware vendors. This paper presents an overview of the ESCAPE strategy: (i) identify domain-specific key algorithmic motifs in weather prediction and climate models (which we term Weather & Climate Dwarfs), (ii) categorise them in terms of computational and communication patterns while (iii) adapting them to different hardware architectures with alternative programming models, (iv) analyse the challenges in optimising, and (v) find alternative algorithms for the same scheme. The participating weather prediction models are the following: IFS (Integrated Forecasting System); ALARO, a combination of AROME (Application de la Recherche à l'Opérationnel à Meso-Echelle) and ALADIN (Aire Limitée Adaptation Dynamique Développement International); and COSMO–EULAG, a combination of COSMO (Consortium for Small-scale Modeling) and EULAG (Eulerian and semi-Lagrangian fluid solver). For many of the weather and climate dwarfs ESCAPE provides prototype implementations on different hardware architectures (mainly Intel Skylake CPUs, NVIDIA GPUs, Intel Xeon Phi, Optalysys optical processor) with different programming models. The spectral transform dwarf represents a detailed example of the co-design cycle of an ESCAPE dwarf. The dwarf concept has proven to be extremely useful for the rapid prototyping of alternative algorithms and their interaction with hardware; e.g. the use of a domain-specific language (DSL). Manual adaptations have led to substantial accelerations of key algorithms in numerical weather prediction (NWP) but are not a general recipe for the performance portability of complex NWP models. Existing DSLs are found to require further evolution but are promising tools for achieving the latter. Measurements of energy and time to solution suggest that a future focus needs to be on exploiting the simultaneous use of all available resources in hybrid CPU–GPU arrangements.

Description: We present a nonhydrostatic finite-volume global atmospheric model formulation for numerical weather prediction with the Integrated Forecasting System (IFS) at ECMWF and compare it to the established operational spectral-transform formulation. The novel Finite-Volume Module of the IFS (henceforth IFS-FVM) integrates the fully compressible equations using semi-implicit time stepping and non-oscillatory forward-in-time (NFT) Eulerian advection, whereas the spectral-transform IFS solves the hydrostatic primitive equations (optionally the fully compressible equations) using a semi-implicit semi-Lagrangian scheme. The IFS-FVM complements the spectral-transform counterpart by means of the finite-volume discretization with a local low-volume communication footprint, fully conservative and monotone advective transport, all-scale deep-atmosphere fully compressible equations in a generalized height-based vertical coordinate, and flexible horizontal meshes. Nevertheless, both the finite-volume and spectral-transform formulations can share the same quasi-uniform horizontal grid with co-located arrangement of variables, geospherical longitude–latitude coordinates, and physics parameterizations, thereby facilitating their comparison, coexistence, and combination in the IFS. We highlight the advanced semi-implicit NFT finite-volume integration of the fully compressible equations of IFS-FVM considering comprehensive moist-precipitating dynamics with coupling to the IFS cloud parameterization by means of a generic interface. These developments – including a new horizontal–vertical split NFT MPDATA advective transport scheme, variable time stepping, effective preconditioning of the elliptic Helmholtz solver in the semi-implicit scheme, and a computationally efficient implementation of the median-dual finite-volume approach – provide a basis for the efficacy of IFS-FVM and its application in global numerical weather prediction. Here, numerical experiments focus on relevant dry and moist-precipitating baroclinic instability at various resolutions. We show that the presented semi-implicit NFT finite-volume integration scheme on co-located meshes of IFS-FVM can provide highly competitive solution quality and computational performance to the proven semi-implicit semi-Lagrangian integration scheme of the spectral-transform IFS.

Description: In the simulation of complex multi-scale flow problems, such as those arising in weather and climate modelling, one of the biggest challenges is to satisfy operational requirements in terms of time-to-solution and energy-to-solution yet without compromising the accuracy and stability of the calculation. These competing factors require the development of state-of-the-art algorithms that can optimally exploit the targeted underlying hardware and efficiently deliver the extreme computational capabilities typically required in operational forecast production. These algorithms should (i) minimise the energy footprint along with the time required to produce a solution, (ii) maintain a satisfying level of accuracy, (iii) be numerically stable and resilient, in case of hardware or software failure. The European Centre for Medium Range Weather Forecasts (ECMWF) is leading a project called ESCAPE (Energy-efficient SCalable Algorithms for weather Prediction on Exascale supercomputers) which is funded by Horizon 2020 (H2020) under initiative Future and Emerging Technologies in High Performance Computing (FET-HPC). The goal of the ESCAPE project is to develop a sustainable strategy to evolve weather and climate prediction models to next-generation computing technologies. The project partners incorporate the expertise of leading European regional forecasting consortia, university research, experienced high-performance computing centres and hardware vendors. This paper presents an overview of results obtained in the ESCAPE project in which weather prediction have been broken down into smaller building blocks called dwarfs. The participating weather prediction models are: IFS (Integrated Forecasting System), ALARO – a combination of AROME (Application de la Recherche à l'Opérationnel a Meso-Echelle) and ALADIN (Aire Limitée Adaptation Dynamique Développement International) and COSMO-EULAG – a combination of COSMO (Consortium for Small-scale Modeling) and EULAG (Eulerian/semi-Lagrangian fluid solver). The dwarfs are analysed and optimised in terms of computing performance for different hardware architectures (mainly Intel Skylake CPUs, NVIDIA GPUs, Intel Xeon Phi). The ESCAPE project includes the development of new algorithms that are specifically designed for better energy efficiency and improved portability through domain specific languages. In addition, the modularity of the algorithmic framework, naturally allows testing different existing numerical approaches, and their interplay with the emerging heterogeneous hardware landscape. Throughout the paper, we will compare different numerical techniques to solve the main building blocks that constitute weather models, in terms of energy efficiency and performance, on a variety of computing technologies.

Description: We present a nonhydrostatic finite-volume global atmospheric model formulation for numerical weather prediction with the Integrated Forecasting System (IFS) at ECMWF, and compare it to the established operational spectral-transform formulation. The novel Finite-Volume Module of IFS (henceforth IFS-FVM) integrates the fully compressible equations using semi-implicit time stepping and non-oscillatory forward-in-time (NFT) Eulerian advection, whereas the spectral-transform IFS solves the hydrostatic primitive equations (optionally the fully compressible equations) using a semi-implicit semi-Lagrangian scheme. The IFS-FVM complements the spectral-transform counterpart by means of the finite-volume discretisation with a local communication footprint, fully conservative and monotone advective transport, all-scale deep-atmosphere fully compressible equations in a generalised height-based vertical coordinate, applicable on flexible meshes. Nevertheless, both the finite-volume and spectral-transform formulations can share the same quasi-uniform horizontal grid with co-located arrangement of variables, geospherical longitude-latitude coordinates, and physical parametrisations, thereby facilitating their comparison, coexistence and combination in IFS. We highlight the advanced semi-implicit NFT finite-volume integration of the fully compressible equations of the novel IFS-FVM considering comprehensive moist-precipitating dynamics with coupling to the IFS cloud parametrisation by means of a generic interface. These developments – including a new horizontal-vertical split NFT MPDATA advective transport scheme, variable time stepping, effective preconditioning of the elliptic Helmholtz solver in the semi-implicit scheme, and a computationally efficient coding implementation – provide a basis for the efficacy of IFS-FVM and its application in global numerical weather prediction. Here, numerical experiments focus on relevant dry and moist-precipitating baroclinic instability at various resolutions. We show that the presented semi-implicit NFT finite-volume integration scheme on co-located meshes of IFS-FVM can provide highly competitive solution quality and computational performance to the proven semi-implicit semi-Lagrangian integration scheme of the spectral-transform IFS.

Description: Water bodies influence local weather and climate, especially in lake-rich areas (e.g. lake regions in Canada and northern Russia). In 2015 a parametrization to represent inland water bodies was included in the Integrated Forecasting System (IFS) used operationally at the European Centre for Medium-Range Weather Forecasts (ECMWF) to produce global weather predictions. The parameterization is based on the Fresh-water Lake model (FLake). In the IFS, FLake runs on any inland surface grid-point. It thus depends on global, realistic and complete lake depth and lake cover data as input. Operationally used lake depths make use of the Global Lake Data Base (GLDBv1) over lakes, a default value of 25 m where lake depth information is missing, and over the ocean bathymetry from ETOPO1. In this study we present the upgraded GLDBv3 dataset, which makes use of mean depth estimates based on geological origin over land, and the GEBCO ocean bathymetry. To assess the impact of using GLDBv3 instead of GLDBv1 in-situ measurements of lake water surface temperatures and information on ice formation/disappearance dates for 27 lakes collected by the Finnish Environment Institute were used for verification. The dataset includes daily temperature measurements and ice formation/disappearance dates recorded by observers. A set of surface experiments were carried out using the IFS and atmospheric forcing from the ERA5 reanalysis to test the operational and new lake depths. These simulations were done at a grid spacing of about 9 km and covered the 5-year period 2010–2014. When verified against in-situ lake depth measurements, the new lake depths in GLDBv3 have a much lower mean absolute error, bias and standard deviation error compared to the current operationally used GLDBv1 depths. Indirect verification was used to compare measured and modelled lake water surface temperatures and ice formation/disappearance dates. On average the mean absolute error of lake surface temperatures is reduced by 13.4 %, biases are reduced by 12.5 % and the standard deviation error by 20.3 % when GLDBv3 is used instead of GLDBv1. Seasonal verification of mixed layer depth temperature and ice formation/disappearance dates (using lake mixing seasons) revealed a cold bias in meteorological forcing from ERA5. However, for spring, summer and autumn verification confirms an overall reduction in the errors in surface water temperatures. For winter verification based on ice formation/disappearance dates shows no statistically significant change in ice disappearance date errors.

Description: Weather and climate models are complex pieces of software which include many individual components, each of which is evolving under pressure to exploit advances in computing to enhance some combination of a range of possible improvements (higher spatio-temporal resolution, increased fidelity in terms of resolved processes, more quantification of uncertainty, etc.). However, after many years of a relatively stable computing environment with little choice in processing architecture or programming paradigm (basically X86 processors using MPI for parallelism), the existing menu of processor choices includes significant diversity, and more is on the horizon. This computational diversity, coupled with ever increasing software complexity, leads to the very real possibility that weather and climate modelling will arrive at a chasm which will separate scientific aspiration from our ability to develop and/or rapidly adapt codes to the available hardware. In this paper we review the hardware and software trends which are leading us towards this chasm, before describing current progress in addressing some of the tools which we may be able to use to bridge the chasm. This brief introduction to current tools and plans is followed by a discussion outlining the scientific requirements for quality model codes which have satisfactory performance and portability, while simultaneously supporting productive scientific evolution. We assert that the existing method of incremental model improvements employing small steps which adjust to the changing hardware environment is likely to be inadequate for crossing the chasm between aspiration and hardware at a satisfactory pace, in part because institutions cannot have all the relevant expertise in house. Instead, we outline a methodology based on large community efforts in engineering and standardisation, which will depend on identifying a taxonomy of key activities – perhaps based on existing efforts to develop domain-specific languages, identify common patterns in weather and climate codes, and develop community approaches to commonly needed tools and libraries – and then collaboratively building up those key components. Such a collaborative approach will depend on institutions, projects, and individuals adopting new interdependencies and ways of working.

Description: The sensitivities of the Madden–Julian oscillation (MJO) forecasts to various different configurations of the parameterized physics are examined with the global model of ECMWF's Integrated Forecasting System (IFS). The motivation for the study was to simulate the MJO as a nonlinear free wave under active interactions with higher-latitude Rossby waves. To emulate free dynamics in the IFS, various momentum-dissipation terms (“friction”) as well as diabatic heating were selectively turned off over the tropics for the range of the latitudes from 20∘ S to 20∘ N. The reduction of friction sometimes improves the MJO forecasts, although without any systematic tendency. Contrary to the original motivation, emulating free dynamics with an operational forecast model turned out to be rather difficult, because forecast performance sensitively depends on the specific type of friction turned off. The result suggests the need for theoretical investigations that much more closely follow the actual formulations of model physics: a naive approach with a dichotomy of with or without friction simply fails to elucidate the rich behaviour of complex operational models. The paper further exposes the importance of physical processes other than convection for simulating the MJO in global forecast models.