ALBERT

Description: High-quality data is necessary for modern machine learning. However, the acquisition of such data is difficult due to noisy and ambiguous annotations of humans. The aggregation of such annotations to determine the label of an image leads to a lower data quality. We propose a data-centric image classification benchmark with ten real-world datasets and multiple annotations per image to allow researchers to investigate and quantify the impact of such data quality issues. With the benchmark we can study the impact of annotation costs and (semi-)supervised methods on the data quality for image classification by applying a novel methodology to a range of different algorithms and diverse datasets. Our benchmark uses a two-phase approach via a data label improvement method in the first phase and a fixed evaluation model in the second phase. Thereby, we give a measure for the relation between the input labeling effort and the performance of (semi-)supervised algorithms to enable a deeper insight into how labels should be created for effective model training. Across thousands of experiments, we show that one annotation is not enough and that the inclusion of multiple annotations allows for a better approximation of the real underlying class distribution. We identify that hard labels can not capture the ambiguity of the data and this might lead to the common issue of overconfident models. Based on the presented datasets, benchmarked methods, and analysis, we create multiple research opportunities for the future directed at the improvement of label noise estimation approaches, data annotation schemes, realistic (semi-)supervised learning, or more reliable image collection.

Description: Spectacular advances have been made in the field of machine vision over the past decade. While this discipline is traditionally driven by geometric models, neural networks have proven to be superior in some applications and have significantly expanded the limits of what is possible. At the same time, conventional graphic models describe the relationship between images and the associated scene with textures and light in a physically realistic manner and are an important part of photogrammetry. Differential renderers combine these approaches by enabling gradient-based optimization in fixed structures of a graphics pipeline and thus adapt the learning process of neural networks. This fusion of formalized knowledge and machine learning motivates the idea of a modular differentiable renderer in which physical and statistical models can be recombined depending on the use case. We therefore present Gemini Connector: an initiative for the modular development and combination of differentiable physical models and neural networks. We examine opportunities and problems and motivate the idea with the extension of a differentiable rendering pipeline to include models of underwater optics for the analysis of deep sea images. Finally, we discuss use cases, especially within the Cross-Domain Fusion initiative.

Description: In engineering, machines are typically built after a careful conception and design process: All components of a system, their roles and the interaction between them is well understood, and often even digital models of the system exist before the actual hardware is built. This enables simulations and even feedback loops between the real-world system and a digital model, leading to a digital twin that allows better testing, prediction and understanding of complex effects. On the contrary, in Earth sciences, and particularly in ocean sciences, models exist only for certain aspects of the real world, of certain processes and of some interactions and dependencies between different “components” of the ocean. These individual models cover large temporal (seconds to millions of years) and spatial (millimetres to thousands of kilometres) scales, a variety of field data underpin them, and their results are represented in many different ways. A key to enabling digital twins in the oceans is fusion at different levels, in particular, fusion of data sources and modalities, fusion over different scales and fusion of differing representations. We outline these challenges and exemplify different envisioned digital twins employed in the oceans involving remote sensing, underwater photogrammetry and computer vision, focusing on optical aspects of the digital twinning process. In particular, we look at the holistic sensing scenarios of optical properties in coastal waters as well as seafloor dynamics at volcanic slopes and discuss road blockers for digital twins as well as potential solutions to increase and widen the use of digital twins.