ALBERT

Description: DASHBOARD.awi.de is a component of our data flow framework designed to enable a semi-automated flow of sensor observations to archives (acronym O2A). The dramatic increase in the number and type of platforms and respective sensors operated by Alfred Wegener Institute along with complex project-driven requirements in terms of satellite communication, sensor monitoring, quality control and validation, processing pipelines, visualization, and archival under FAIR principles, led us to build a generic and cost-effective data flow framework. Most important, all components and services which make up this framework are extensible and exchangeble, were built using open access technologies (e.g. elastic search) and vocabularies (SeaVox NERC 2.0 vocabulary) and are compliant with various interoperability standards recommended by the international community. In this poster we illustrate the DASHBOARD.awi.de component which is a web-based monitoring environment for supporting scientists in the graphing, mapping and simple analysis of time series. With a set of fit-for-purpose widgets including data download, scientists are able to identify gaps and outliers in the streamed data. Morover, we are in the process of building alerting solutions for individual parameters using the parameter properties available from SENSOR.awi.de (e.g. min/max parameter range). The streaming services attached to this component are using the near real-time data transfered from remote field sites to local databases and storage systems. For this sake, we are participating in the RDA "Array Database Assessment" Working Group and "Big Data" Interest Group. The graphing solutions built within DASHBOARD.awi.de can be easily re-used in other context (e.g. project web pages) . Our solutions support the OGC standard Observations and Measurements (O&M), JSON and CSV as exchange data format. To date, ~ 150 individual parameters are being transfered in near real-time from remote sites to our onshore storage systems and monitored by scientists. In the week of the 11th RDA Plenary Berlin, we will be able to interact with live data from Polarstern crossing the Drake Passage, on its way to Antarctica. Scientific Discipline/Research Area: Data flow, standardized vocabulary, Visualisation, Streaming. Relevance/Link to RDA: The streaming services attached to this component are using the near real-time data transfered from remote field sites to local databases and storage systems. For this sake, we are participating in the RDA "Array DB Assesment" Working Group and "Big Data" Interest Group.

Description: The Multidisciplinary drifting Observatory for the Study of Arctic Climate (MOSAiC) is a collaborative, international project to address pressing scientific questions in the central Arctic. The project’s success, and its ultimate impact on science and society, relies upon professional coordination and data sharing across the participants. A transparent Data Policy is essential to achieve MOSAiC science objectives, to facilitate collaboration, and to enable broad use and impact of the MOSAiC data legacy.

Description: To support ship navigation, station planning, and scientific data evaluation, a new near real time sea-ice information system was installed onboard FS Polarstern during summer 2016.

Description: DATA.awi.de is a component of our data flow framework designed to enable a semi-automated flow of sensor observations to archives (acronym O2A). The dramatic increase in the number and type of platforms and respective sensors operated by Alfred Wegener Institute along with complex project-driven requirements in terms of satellite communication, sensor monitoring, quality control and validation, processing pipelines, visualization, and archival under FAIR principles, led us to build a generic and cost-effective data flow framework. Most important, all components and services which make up this framework are extensible and exchangeble, were built using open access technologies (e.g. elastic search) and vocabularies (SeaVox NERC 2.0 vocabulary) and are compliant with various interoperability standards recommended by the international community. In this poster we illustrate the DATA.awi.de component which is a one-stop-shop framework for enabling discovery and dissemination of heterogeneous scientific information. Because the metadata and data generated and captured by the other O2A components are machine-readable and interoperable, we were able to build harvesting and indexing solutions which enable scientists and other stakeholders to discover content ranging from platforms/sensors, tracklines, field reports, near real-time data to quality-controlled data, map products and peer-reviewed publications. Scientific Disciplin/Research Area: Findability, Interoperability, Integration, Re-use. Relevance/Link to RDA: In the context of our harvesting approach, we are interacting with the RDA "Brokering Framework" Working Group and "Brokering" Interest Group.

Description: Over the last two decades, the Alfred Wegener Institute (AWI) has been continuously committing to develop and sustain an e-Infrastructure for coherent discovery, visualization, dissemination and archival of scientific information in polar and marine regions. Most of the data originates from research activities being carried out in a wide range of AWI-operated research platforms: vessels, land-based stations, ocean-based stations and aircrafts. Archival and publishing in PANGAEA repository along with DOI assignment to individual datasets is a typical end-of-line step for most data owners. Within AWI, a workflow for data acquisition from vessel-mounted devices along with ingestion procedures for the raw data into the institutional archives has been well-established for many years. However, the increasing number of ocean-based stations and respective sensors along with heterogeneous project-driven requirements towards satellite communication, sensor monitoring, QA/QC control and validation, processing algorithms, visualization and dissemination has recently lead us to build a more generic and cost-effective framework. This framework, hereafter named O2A, has as main strength its seamless flow of sensor observation to archives and the fact that it complies with internationally used OGC standards and thus assuring interoperability in international context (e.g. SOS/SWE, WPS, WMS WFS,..). O2A is comprised of several extensible and exchangeable modules (e.g. controlled vocabularies and gazetteers, file type and structure validation, aggregation solutions, processing algorithms, etc) as well as various interoperability services. At the first data tier level, not only each sensor is being described following SensorML data model standards but the data is being fed to an SOS interface offering streaming solutions along with support to O&M encoding. Project administrators or data specialists are now able to monitor the individual sensors displayed in a map by simply clicking on the station and viewing the near real-time data for the selected station and sensor. In addition, the monitoring dashboards we built provide assistance to data scientists and administrators in terms of early detection of malfunction of sensors (e.g., email/SMS notification), filtering of data values for certain range (e.g. temperature values above a certain range) and data aggregation (e.g. calculation of daily averages).

Description: The increasing number and complexity of research platforms and respective devices and sensors along with heterogeneous project-driven requirements towards satellite communication, sensor monitoring, quality assessment and control, processing, analysis and visualization has recently lead us to build a generic and cost-effective framework (O2A) to enable the flow of sensor observations to archives. O2A is comprised of several extensible and exchangeable components as well as various interoperability services and is meant to offer practical solutions towards supporting the typical scientific workflow ranging from data acquisition activities until the very last data publication activities. The web-based sensor monitoring component built within O2A offers a dashboard-oriented approach for displaying near real-time and delayed-mode sensor output parameters including simultaneous map and diagram viewing. This module allows project administrators and data specialists to monitor individual sensors in near real-time as well as to view the data values within a wished temporal range and/or geographical coverage. Additional examples of O2A components are the AWI-specific SensorML profile and raw data ingest solutions, the various data workspace areas and dispatcher middleware, the GIS infrastructure, and the ticket and data curation system as central hub supporting the final data publication activity. Finally, in the context of the large-scale multi-disciplinary project "Frontiers of Arctic Monitoring" Project (FRAM), we illustrate how the proposed O2A framework will assist scientists in developing enhanced data products and facilitate data re-use in the future.

Description: The data archival process starts with the acquisition of information on scientific projects. This applies specially to time-consuming and cost intensive acquisition of data in remote polar regions. Within the ingestion framework for raw data at AWI, the data acquisition and processing workflow has been defined as a generic RDIF project. Once a RDIF projects has been defined, a seamless data transfer from scientifc acquisition systems to data archives follows atomatically. In the framework of the data flow, the datasets undergo a "file structure" validation process in which not only the associated metadata is created but also a digital object identifier (DOI) is assigned. The assigned identifier beomes an integral part of the metadata being used in all following processing steps and visualization procedures.

Description: Numerous research institutes are active in climate research by modeling and analyzing climate scenarios within their fields of expertise. Climate change, however, acts across many compartments of the earh system and thus requires an integrated approach across earth science disciplines. The integration of climate-relevant variables across earth compartments has been improved over the last decades allowing validated predictions for global or large scale climate scenarios today. However, ongoing research and climate reality show that global scale climate changes increasingly also trigger local climate changes and enhance extreme weather events. Although their occurrence is (still) not predictable, extreme weathers may affect the region of occurrence tremendously and can come with a great potential of causing disasters including effects on geomorphology, vegetation, rivers, as well as infrastructure (e. g., cities, roads, railroads) which can be damaged by extreme weather events. Both MOSES and Digital Earth focus on the Elbe river system use case that addresses floods and draughts within climate change. Performing test campaigns in the entire Elbe system, the eight Helmholtz Centres of the Sector Earth and Environment of BMBF are currently clustering their expertise within the framework of MOSES and Digital Earth to establish a Modular and Mobile Sensor Network. MOSES will measure short, medium, and long term effects of floods and draughts in the river Elbe, and Digital Earth will process and model the data in real-time. Both projects combine their outstanding expertise including the necessary data integration and processing steps across scientific disciplines and will provide the required IT-expertise and infrastructure to ensure the seamless data flow from the sensor network to real-time analysis and visualization. This includes the automatization of access to data from very diverse sources, related validation and transformation processes, as well as the installation of further processing and/or analyzing routines. It comprises of a mixture of powerful hardware, automatization scripts, and applications to administer all of it. Here we will focus on the Sensor Network and the IT infrastructure part (Digital Earth WP1, WP2) showing O2A, AWI's data flow framework that covers all infrastructure units from observation, via analysis and visualization to archive (SENOR.AWI.DE, DASHOARD.AWI.DE, MAPS.AWI.DE, DATA.AWI.DE, PANGAEA.DE)

Description: The O2A (Observation to Archive) is a data-flow framework for heterogeneous sources, including multiple institutions and scales of Earth observation. In the O2A, once data transmission is set up, processes are executed to automatically ingest (i.e. collect and harmonize) and quality control data in near real-time. We consider a web-based sensor description application to support transmission and harmonization of observational time-series data. We also consider a product-oriented quality control, where a standardized and scalable approach should integrate the diversity of sensors connected to the framework. A review of literature and observation networks of marine and terrestrial environments is under construction to allow us, for example, to characterize quality tests in use for generic and specific applications. In addition, we use a standardized quality flag scheme to support both user and technical levels of information. In our outlook, a quality score should pair the quality flag to indicate the overall plausibility of each individual data value or to measure the flagging uncertainty. In this work, we present concepts under development and give insights into the data ingest and quality control currently operating within the O2A framework.