ALBERT

Description: The O2A (Observation to Archive) is a data-flow framework for heterogeneous sources, including multiple institutions and scales of Earth observation. In the O2A, once data transmission is set up, processes are executed to automatically ingest (i.e. collect and harmonize) and quality control data in near real-time. We consider a web-based sensor description application to support transmission and harmonization of observational time-series data. We also consider a product-oriented quality control, where a standardized and scalable approach should integrate the diversity of sensors connected to the framework. A review of literature and observation networks of marine and terrestrial environments is under construction to allow us, for example, to characterize quality tests in use for generic and specific applications. In addition, we use a standardized quality flag scheme to support both user and technical levels of information. In our outlook, a quality score should pair the quality flag to indicate the overall plausibility of each individual data value or to measure the flagging uncertainty. In this work, we present concepts under development and give insights into the data ingest and quality control currently operating within the O2A framework.

Description: The understanding of extreme events strongly depends on knowledge gained from data. Data integration of mul-tiple sources, scales and earth compartments is the fo-cus of the project Digital Earth, which also join efforts on the quality control of data. Automatic quality control is embedded in the ingest component of the O2A, the ob-servation-to-archive data flow framework of the Alfred-Wegener-Institute. In that framework, the O2A-Sensor provides observation properties to the O2A-Ingest, which delivers quality-flagged data to the O2A-dash-board. The automatic quality control currently follows a procedural approach, where modules are included to implement formulations found in the literature and other operational observatory networks. A set of plausibility tests including range, spike and gradient tests are cur-rently operational. The automatic quality control scans the ingesting data in near-real-time (NRT) format, builds a table of devices, and search - either by absolute or derivative values - for correctness and validity of obser-vations. The availability of observation properties, for in-stance tests parameters like physical or operation ranges, triggers the automatic quality control, which in turn iterates through the table of devices to set the qual-ity flag for each sample and observation. To date, the quality flags in use are sequential and qualitative, i.e. it describes a level of quality in the data. A new flagging system is under development to include a descriptive characteristic that will comprise technical and user inter-pretation. Within Digital Earth, data on flood and drought events along the Elbe River and methane emissions in the North Sea are to be reviewed using automatic qual-ity control. Fast and scalable automatic quality control will disentangle uncertainty raised by quality issues and thus improve our understanding of extreme events in those cases.

Description: During the largest polar expedition in history starting in September 2019, the German research icebreaker Polarstern spends a whole year drifting with the ice through the Arctic Ocean. The MOSAiC expedition takes the closest look ever at the Arctic even throughout the polar winter to gain fundamental insights and most unique on-site data for a better understanding of global climate change. Hundreds of researchers from 20 countries are involved. Scientists will use the in situ gathered data instantaneously in near-real time modus as well as long afterwards all around the globe taking climate research to a completely new level. Hence, proper data management, sampling strategies beforehand, and monitoring actual data flow as well as processing, analysis and sharing of data during and long after the MOSAiC expedition are the most essential tools for scientific gain and progress. To prepare for that challenge we adapted and integrated the research data management framework O2A “Data flow from Observations to Archives” to the needs of the MOSAiC expedition on board Polarstern as well as on land for data storage and access at the Alfred Wegener Institute Computing and Data Center in Bremerhaven, Germany. Our O2A-framework assembles a modular research infrastructure comprising a collection of tools and services. These components allow researchers to register all necessary sensor metadata beforehand linked to automatized data ingestion and to ensure and monitor data flow as well as to process, analyze, and publish data to turn the most valuable and uniquely gained arctic data into scientific outcomes. The framework further allows for the integration of data obtained with discrete sampling devices into the data flow. These requirements have led us to adapt the generic and cost-effective framework O2A to enable, control, and access the flow of sensor observations to archives in a cloud-like infrastructure on board Polarstern and later on to land based repositories for international availability. Major roadblocks of the MOSAiC-O2A data flow framework are (i) the increasing number and complexity of research platforms, devices, and sensors, (ii) the heterogeneous interdisciplinary driven requirements towards, e. g., satellite data, sensor monitoring, in situ sample collection, quality assessment and control, processing, analysis and visualization, and (iii) the demand for near real time analyses on board as well as on land with limited satellite bandwidth. The key modules of O2A's digital research infrastructure established by AWI are implementing the FAIR principles: SENSORWeb, to register sensor applications and sampling devices and capture controlled meta data before and alongside any measurements in the field Data ingest, allowing researchers to feed data into storage systems and processing pipelines in a prepared and documented way, at best in controlled near real-time data streams Dashboards allowing researchers to find and access data and share and collaborate among partners Workspace enabling researchers to access and use data with research software utilizing a cloud-based virtualized infrastructure that allows researchers to analyze massive amounts of data on the spot Archiving and publishing data via repositories and Digital Object Identifiers (DOI)

Description: Modern digital scientific workflows - often implying Big Data challenges - require data infrastructures and innovative data science methods across disciplines and technologies. Diverse activities within and outside HGF deal with these challenges, on all levels. The series of Data Science Symposia fosters knowledge exchange and collaboration in the Earth and Environment research community. We invited contributions to the overarching topics of data management, data science and data infrastructures. The series of Data Science Symposia is a joint initiative by the three Helmholtz Centers HZG, AWI and GEOMAR Organization: Hela Mehrtens and Daniela Henkel (GEOMAR)

Description: During the MOSAiC expedition, the German research icebreaker Polarstern spends a full year drifting through the Arctic Ocean. Scientists from 20 countries participate in the largest polar expedition in history exploring the Arctic climate system. The experiment covers a large suite of in-situ and remote sensing observations of physical, ecological and biogeochemical parameters to describe the processes coupling the atmosphere, sea ice, and ocean. In addition to forefront instrumentation and observational techniques, proper data management is essential for large and complex projects and field programs. Key elements are agreements on consistent sampling strategies, the possibility to monitor the data flow, to facilitate near real-time processing, and analysis and sharing of data during and long after the expedition. Furthermore, data publication and documentation are crucial for such a collaborative effort and will build the legacy of the project and finally take climate science to the next level. We adapted our modular research data management framework O2A “Data flow from Observations to Archives” to meet the expedition requirements and ensure central data archival for generations to come. Researchers register all necessary sensor metadata beforehand. Essential metadata of scientific actions in the field are ingested immediately with the FloeNavi, a novel system enabling navigation on a drifting ice floe. O2A provides tools to automatize data ingestion, monitor the data flow and process, analyze and publish data. Integration of ship- and land-based components and a shared storage ensure seamless continuation of collaboration during and after the expedition laying the fundamentals for numerous data publications.