ALBERT

Description: Environmental scientists aim at understanding not only single components but systems, one example is the flood system; scientists investigate the conditions, drivers and effects of flood events and the relations between them. Investigating environmental systems with a data-driven research approach requires linking a variety of data, analytical methods, and derived results. Several obstacles exist in the recent scientific work environment that hinder scientists to easily create these links. They are distributed and heterogeneous data sets, separated analytical tools, discontinuous analytical workflows, as well as isolated views to data and data products. We address these obstacles with the exception of distributed and heterogeneous data since this is part of other ongoing initiatives. Our goal is to develop a framework supporting the data-driven investigation of environmental systems. First we integrate separated analytical tools and methods by the means of a component-based software framework. Furthermore we allow for seamless and continuous analytical workflows by applying the concept of digital workflows, which also demands the aforementioned integration of separated tools and methods. Finally we provide integrated views of data and data products by interactive visual interfaces with multiple linked views. The combination of these three concepts from computer science allows us to create a digital research environment that enable scientists to create the initially mentioned links in a flexible way. We developed a generic concept for our approach, implemented a corresponding framework and finally applied both to realize a “Flood Event Explorer” prototype supporting the comprehensive investigation of a flood system. In order to implement a digital workflow our approach intends to precisely define the workflow’s requirements. We mostly do this by conducting informal interviews with the domain scientists. The defined requirements also include the needed analytical tools and methods, as well as the utilized data and data products. For technically integrating the needed tools and methods our created software framework provides a modularization approach based on a messaging system. This allows us to create custom modules or wrap existing implementations and tools. The messaging system (e.g. pulsar) then connects these individual modules. This enables us to combine multiple methods and tools into a seamless digital workflow. The described approach of course demands the proper definition of interfaces to modules and data sources. Finally our software framework provides multiple generic visual front-end components (e.g. tables, maps and charts) to create interactive linked views supporting the visual analysis of the workflow’s data.

Description: SEVA is a scalable exploration tool that supports users to conduct change detection based on optical Sentinel-2 satellite observations. It supports the following essential steps of change detection: a) exploration and selection of optical satellite images to recognize proper data for the current application scenario, b) automated extraction of changes from the optical satellite images, c) analysis of errors and d) assessment and interpretation of the extracted changes.

Description: Surrogate playground is an automated machine learning approach written for rapidly screening a large number of different models to serve as surrogates for a slow running simulator. This code was written for a reactive transport application where a fluid flow model (hydrodynamics) is coupled to a geochemistry simulator (reactions in time and space) to simulate scenarios such as underground storage of CO2 or hydrogen storage for excess energy from wind farms. The challenge for such applications is that the geochemistry simulator is typically slow compared to fluid dynamics and constitutes the main bottleneck for producing highly detailed simulations of such application scenarios. This approach attempts to find machine learning models that can replace the slow running simulator when trained on input-output data from the geochemistry simulator. The code may be of more general interest as this prototype can be used to screen many different machine learning models for any regression problem in general. To illustrate this it also includes a demonstration example using the Boston housing standard data-set.

Description: Gms-index-mediator is a standalone index for spatio-temporal data acting as a mediator between an application and a database. Even modern databases need several minutes to execute a spatio-temporal query to huge tables containing several million entries. Our index-mediator speeds the execution of such queries up by several magnitues, resulting in response times around 100ms. This version is tailored towards the GeoMultiSens database, but can be adapted to work with custom table layouts with reasonable effort.

Description: PyRQA is a tool to conduct recurrence quantification analysis (RQA) and to create recurrence plots in a massively parallel manner using the OpenCL framework. It is designed to process very long time series consisting of hundreds of thousands of data points efficiently.

Description: The success of scientific projects increasingly depends on using data analysis tools and data in distributed IT infrastructures. Scientists need to use appropriate data analysis tools and data, extract patterns from data using appropriate computational resources, and interpret the extracted patterns. Data analysis tools and data reside on different machines because the volume of the data often demands specific resources for their storage and processing, and data analysis tools usually require specific computational resources and run-time environments. The data analytics software framework DASF, which we develop in Digital Earth (Bouwer et al. (2022)), provides a framework for scientists to conduct data analysis in distributed environments.

Description: The Digital Earth Flood Event Explorer supports geoscientists and experts to analyse flood events along the process cascade event generation, evolution and impact across atmospheric, terrestrial, and marine disciplines. It applies the concept of scientific workflows and the component-based Data Analytics Software Framework (DASF, Eggert and Dransch, 2021) to an exemplary showcase. It aims at answering the following geoscientific questions: - How does precipitation change over the course of the 21st century under different climate scenarios over a certain region? - What are the main hydro-meteorological controls of a specific flood event? - What are useful indicators to assess socio-economic flood impacts? - How do flood events impact the marine environment? - What are the best monitoring sites for upcoming flood events? The Flood Event Explorer developed scientific workflows for each geoscientific question providing enhanced analysis methods from statistics, machine learning, and visual data exploration that are implemented in different languages and software environments, and that access data form a variety of distributed databases. The collaborating scientists are from different Helmholtz research centers and belong to different scientific fields such as hydrology, climate-, marine-, and environmental science, and computer- and data science. It is funded by the Initiative and Networking Fund of the Helmholtz Association through the Digital Earth project (https://www.digitalearth-hgf.de/).

Description: The success of scientific projects increasingly depends on using data analysis tools and data in distributed IT infrastructures. Scientists need to use appropriate data analysis tools and data, extract patterns from data using appropriate computational resources, and interpret the extracted patterns. Data analysis tools and data reside on different machines because the volume of the data often demands specific resources for their storage and processing, and data analysis tools usually require specific computational resources and run-time environments. The data analytics software framework DASF, developed at the GFZ German Research Centre for Geosciences (https://www.gfz-potsdam.de) and funded by the Initiative and Networking Fund of the Helmholtz Association through the Digital Earth project (https://www.digitalearth-hgf.de/), provides a framework for scientists to conduct data analysis in distributed environments. The data analytics software framework DASF supports scientists to conduct data analysis in distributed IT infrastructures by sharing data analysis tools and data. For this purpose, DASF defines a remote procedure call (RPC) messaging protocol that uses a central message broker instance. Scientists can augment their tools and data with this protocol to share them with others. DASF supports many programming languages and platforms since the implementation of the protocol uses WebSockets. It provides two ready-to-use language bindings for the messaging protocol, one for Python and one for the Typescript programming language. In order to share a python method or class, users add an annotation in front of it. In addition, users need to specify the connection parameters of the message broker. The central message broker approach allows the method and the client calling the method to actively establish a connection, which enables using methods deployed behind firewalls. DASF uses Apache Pulsar (https://pulsar.apache.org/) as its underlying message broker. The Typescript bindings are primarily used in conjunction with web frontend components, which are also included in the DASF-Web library. They are designed to attach directly to the data returned by the exposed RPC methods. This supports the development of highly exploratory data analysis tools. DASF also provides a progress reporting API that enables users to monitor long-running remote procedure calls. One application using the framework is the Digital Earth Flood Event Explorer (https://git.geomar.de/digital-earth/flood-event-explorer). The Digital Earth Flood Event Explorer integrates several exploratory data analysis tools and remote procedures deployed at various Helmholtz centers across Germany.