ALBERT

Description: The question of how to derive and present uncertainty information in climate data records (CDRs) has received sustained attention within the European Space Agency Climate Change Initiative (CCI), a programme to generate CDRs addressing a range of essential climate variables (ECVs) from satellite data. Here, we review the nature, mathematics, practicalities, and communication of uncertainty information in CDRs from Earth observations. This review paper argues that CDRs derived from satellite-based Earth observation (EO) should include rigorous uncertainty information to support the application of the data in contexts such as policy, climate modelling, and numerical weather prediction reanalysis. Uncertainty, error, and quality are distinct concepts, and the case is made that CDR products should follow international metrological norms for presenting quantified uncertainty. As a baseline for good practice, total standard uncertainty should be quantified per datum in a CDR, meaning that uncertainty estimates should clearly discriminate more and less certain data. In this case, flags for data quality should not duplicate uncertainty information, but instead describe complementary information (such as the confidence in the uncertainty estimate provided or indicators of conditions violating the retrieval assumptions). The paper discusses the many sources of error in CDRs, noting that different errors may be correlated across a wide range of timescales and space scales. Error effects that contribute negligibly to the total uncertainty in a single-satellite measurement can be the dominant sources of uncertainty in a CDR on the large space scales and long timescales that are highly relevant for some climate applications. For this reason, identifying and characterizing the relevant sources of uncertainty for CDRs is particularly challenging. The characterization of uncertainty caused by a given error effect involves assessing the magnitude of the effect, the shape of the error distribution, and the propagation of the uncertainty to the geophysical variable in the CDR accounting for its error correlation properties. Uncertainty estimates can and should be validated as part of CDR validation when possible. These principles are quite general, but the approach to providing uncertainty information appropriate to different ECVs is varied, as confirmed by a brief review across different ECVs in the CCI. User requirements for uncertainty information can conflict with each other, and a variety of solutions and compromises are possible. The concept of an ensemble CDR as a simple means of communicating rigorous uncertainty information to users is discussed. Our review concludes by providing eight concrete recommendations for good practice in providing and communicating uncertainty in EO-based climate data records.

Description: Accurate and long-term rainfall estimates are the main inputs for several applications, spanning from crop modeling to climate analysis. In this study, we present a new rainfall data set (SM2RAIN-CCI) obtained from the inversion of the satellite soil moisture (SM) observations derived from the ESA Climate Change Initiative (CCI) via SM2RAIN (Brocca et al., 2014). Daily rainfall estimates are generated for an 18-year long period (1998–2015), with a spatial sampling of 0.25° on a global scale and are based on the integration of the ACTIVE and the PASSIVE ESA CCI SM data sets. The quality of the SM2RAIN-CCI rainfall data set is evaluated by comparing it with two stateof-art rainfall satellite products, i.e. the Tropical Measurement Mission Multi-satellite Precipitation Analysis 3B42 real-time product (TMPA 3B42RT) and the Climate Prediction Center Morphing Technique (CMORPH), and one modelled data set (ERA-Interim). The assessment is carried out on a global scale at 1° of spatial sampling and 5-day of temporal sampling by comparing these products with the gauge-based Global Precipitation Climatology Centre Full Data Daily (GPCC-FDD) product. SM2RAIN-CCI shows relatively good results in terms of correlation coefficient (median value 〉 0.56), Root Mean Square Difference (RMSD, median value

Description: Climate data records (CDRs) derived from Earth observation (EO) should include rigorous uncertainty information, to support application of the data in policy, climate modelling and numerical weather prediction reanalysis. Uncertainty, error and quality are distinct concepts, and CDR products should follow international norms for presenting quantified uncertainty. Ideally, uncertainty should be quantified per datum in a CDR, and the uncertainty estimates should be able to discriminate more and less certain data with confidence. In this case, flags for data quality should not duplicate uncertainty information, but instead describe complementary information (such as the confidence held in the uncertainty estimate provided, or indicators of conditions violating retrieval assumptions). Errors have many sources and some are correlated across a wide range of time and space scales. Error effects that contribute negligibly to the total uncertainty in a single satellite measurement can be the dominant sources of uncertainty in a CDR on large space and long time scales that are highly relevant for some climate applications. For this reason, identifying and characterizing the relevant sources of uncertainty for CDRs is particularly challenging. Characterisation of uncertainty caused by a given error effect involves assessing the magnitude of the effect, the shape of the error distribution, and the propagation of the uncertainty to the geophysical variable in the CDR accounting for its error correlation properties. Uncertainty estimates can and should be validated as part of CDR validation, where possible. These principles are quite general, but the form of uncertainty information appropriate to different essential climate variables (ECVs) is highly variable, as confirmed by a quick review of the different approaches to uncertainty taken across different ECVs in the European Space Agency’s Climate Change Initiative. User requirements for uncertainty information can conflict with each other, and again a variety of solutions and compromises are possible. The concept of an ensemble CDR as a simple means of communicating rigorous uncertainty information to users is discussed. Our review concludes by providing eight recommendations for good practice in providing and communicating uncertainty in EO-based climate data records.

Description: The soil moisture dataset that is generated via the Climate Change Initiative (CCI) of the European Space Agency (ESA) (ESA CCI SM) is a popular research product. It is composed of observations from 10 different satellites and aims to exploit the individual strengths of active (radar) and passive (radiometer) sensors, thereby providing surface soil moisture estimates at a spatial resolution of 0.25°. However, the annual updating cycle limits the use of the ESA CCI SM dataset for operational applications. Therefore, this study proposes an adaptation of the ESA CCI product for daily global updates via satellite-derived near-real-time (NRT) soil moisture observations. In order to extend the ESA CCI SM dataset from 1978 to present we use NRT observations from the Advanced Scatterometer on-board the two MetOp satellites and the Advanced Microwave Scanning Radiometer 2 on-board GCOM-W. Since these NRT observations do not incorporate the latest algorithmic updates, parameter databases and intercalibration efforts, by nature they offer a lower quality than reprocessed offline datasets. In addition to adaptations of the ESA CCI SM processing chain for NRT datasets, the quality of the NRT datasets is a main source of uncertainty. Our findings indicate that, despite issues in arid regions, the new CCI NRT dataset shows a good correlation with ESA CCI SM. The average global correlation coefficient between CCI NRT and ESA CCI SM (Pearson's R) is 0.80. An initial validation with 40 in situ observations in France, Spain, Senegal and Kenya yields an average R of 0.58 and 0.49 for ESA CCI SM and CCI NRT, respectively. In summary, the CCI NRT product is nearly as accurate as the existing ESA CCI SM product and, therefore, of significant value for operational applications such as drought and flood forecasting, agricultural index insurance or weather forecasting.

Description: Long-term gridded precipitation products are crucial for several applications in hydrology, agriculture and climate sciences. Currently available precipitation products suffer from space and time inconsistency due to the non-uniform density of ground networks and the difficulties in merging multiple satellite sensors. The recent “bottom-up” approach that exploits satellite soil moisture observations for estimating rainfall through the SM2RAIN (Soil Moisture to Rain) algorithm is suited to build a consistent rainfall data record as a single polar orbiting satellite sensor is used. Here we exploit the Advanced SCATterometer (ASCAT) on board three Meteorological Operational (MetOp) satellites, launched in 2006, 2012, and 2018, as part of the European Organisation for the Exploitation of Meteorological Satellites (EUMETSAT) Polar System programme. The continuity of the scatterometer sensor is ensured until the mid-2040s through the MetOp Second Generation Programme. Therefore, by applying the SM2RAIN algorithm to ASCAT soil moisture observations, a long-term rainfall data record will be obtained, starting in 2007 and lasting until the mid-2040s. The paper describes the recent improvements in data pre-processing, SM2RAIN algorithm formulation, and data post-processing for obtaining the SM2RAIN–ASCAT quasi-global (only over land) daily rainfall data record at a 12.5 km spatial sampling from 2007 to 2018. The quality of the SM2RAIN–ASCAT data record is assessed on a regional scale through comparison with high-quality ground networks in Europe, the United States, India, and Australia. Moreover, an assessment on a global scale is provided by using the triple-collocation (TC) technique allowing us also to compare these data with the latest, fifth-generation European Centre for Medium-Range Weather Forecasts (ECMWF) reanalysis (ERA5), the Early Run version of the Integrated Multi-Satellite Retrievals for Global Precipitation Measurement (IMERG), and the gauge-based Global Precipitation Climatology Centre (GPCC) products. Results show that the SM2RAIN–ASCAT rainfall data record performs relatively well at both a regional and global scale, mainly in terms of root mean square error (RMSE) when compared to other products. Specifically, the SM2RAIN–ASCAT data record provides performance better than IMERG and GPCC in data-scarce regions of the world, such as Africa and South America. In these areas, we expect larger benefits in using SM2RAIN–ASCAT for hydrological and agricultural applications. The limitations of the SM2RAIN–ASCAT data record consist of the underestimation of peak rainfall events and the presence of spurious rainfall events due to high-frequency soil moisture fluctuations that might be corrected in the future with more advanced bias correction techniques. The SM2RAIN–ASCAT data record is freely available at https://doi.org/10.5281/zenodo.3405563 (Brocca et al., 2019) (recently extended to the end of August 2019).

Description: The European Space Agency's Climate Change Initiative for Soil Moisture (ESA CCI SM) merging algorithm generates consistent quality-controlled long-term (1978–2018) climate data records for soil moisture, which serves thousands of scientists and data users worldwide. It harmonises and merges soil moisture retrievals from multiple satellites into (i) an active-microwave-based-only product, (ii) a passive-microwave-based-only product and (iii) a combined active–passive product, which are sampled to daily global images on a 0.25∘ regular grid. Since its first release in 2012 the algorithm has undergone substantial improvements which have so far not been thoroughly reported in the scientific literature. This paper fills this gap by reviewing and discussing the science behind the three major ESA CCI SM merging algorithms, versions 2 (https://doi.org/10.5285/3729b3fbbb434930bf65d82f9b00111c; Wagner et al., 2018), 3 (https://doi.org/10.5285/b810601740bd4848b0d7965e6d83d26c; Dorigo et al., 2018) and 4 (https://doi.org/10.5285/dce27a397eaf47e797050c220972ca0e; Dorigo et al., 2019), and provides an outlook on the expected improvements planned for the next algorithm, version 5.

Description: In just the past 5 years, the field of Earth observation has progressed beyond the offerings of conventional space-agency-based platforms to include a plethora of sensing opportunities afforded by CubeSats, unmanned aerial vehicles (UAVs), and smartphone technologies that are being embraced by both for-profit companies and individual researchers. Over the previous decades, space agency efforts have brought forth well-known and immensely useful satellites such as the Landsat series and the Gravity Research and Climate Experiment (GRACE) system, with costs typically of the order of 1 billion dollars per satellite and with concept-to-launch timelines of the order of 2 decades (for new missions). More recently, the proliferation of smartphones has helped to miniaturize sensors and energy requirements, facilitating advances in the use of CubeSats that can be launched by the dozens, while providing ultra-high (3–5 m) resolution sensing of the Earth on a daily basis. Start-up companies that did not exist a decade ago now operate more satellites in orbit than any space agency, and at costs that are a mere fraction of traditional satellite missions. With these advances come new space-borne measurements, such as real-time high-definition video for tracking air pollution, storm-cell development, flood propagation, precipitation monitoring, or even for constructing digital surfaces using structure-from-motion techniques. Closer to the surface, measurements from small unmanned drones and tethered balloons have mapped snow depths, floods, and estimated evaporation at sub-metre resolutions, pushing back on spatio-temporal constraints and delivering new process insights. At ground level, precipitation has been measured using signal attenuation between antennae mounted on cell phone towers, while the proliferation of mobile devices has enabled citizen scientists to catalogue photos of environmental conditions, estimate daily average temperatures from battery state, and sense other hydrologically important variables such as channel depths using commercially available wireless devices. Global internet access is being pursued via high-altitude balloons, solar planes, and hundreds of planned satellite launches, providing a means to exploit the internet of things as an entirely new measurement domain. Such global access will enable real-time collection of data from billions of smartphones or from remote research platforms. This future will produce petabytes of data that can only be accessed via cloud storage and will require new analytical approaches to interpret. The extent to which today's hydrologic models can usefully ingest such massive data volumes is unclear. Nor is it clear whether this deluge of data will be usefully exploited, either because the measurements are superfluous, inconsistent, not accurate enough, or simply because we lack the capacity to process and analyse them. What is apparent is that the tools and techniques afforded by this array of novel and game-changing sensing platforms present our community with a unique opportunity to develop new insights that advance fundamental aspects of the hydrological sciences. To accomplish this will require more than just an application of the technology: in some cases, it will demand a radical rethink on how we utilize and exploit these new observing systems.

Description: The European Space Agency's Climate Change Initiative for Soil Moisture (ESA CCI SM) merging algorithm generates consistent quality-controlled long-term (1978–2018) Climate Data Records for soil moisture which serves thousands of scientists and data users worldwide. It harmonises and merges soil moisture retrievals from multiple satellites into (i) an active-microwave-based only, (ii) a passive-microwave-based only, and a combined active-passive product, which are sampled to daily global images on a 0.25 degree regular grid. Since its first release in 2012 the algorithm has undergone substantial improvements which have so far not been thoroughly reported in the scientific literature. This paper fills this gap by reviewing and discussing the science behind the three major ESA CCI SM merging algorithm versions 2 (https://doi.org/10.5285/3729b3fbbb434930bf65d82f9b00111c; Wagner et al., 2018), 3 (https://doi.org/10.5285/b810601740bd4848b0d7965e6d83d26c; Dorigo et al., 2018a), and 4 (https://doi.org/10.5285/3a8a94c3fa464d68b6d70df291afd457; Dorigo et al., 2018b) and provides an outlook to the expected improvements planned for the next algorithm version 5.

Description: In just the past five years, the field of Earth observation has evolved from the relatively staid approaches of government space agencies into a plethora of sensing opportunities afforded by CubeSats, Unmanned Aerial Vehicles (UAVs), and smartphone technologies that have been embraced by both for-profit companies and individual researchers. Over the previous decades, space agency efforts have brought forth well-known and immensely useful satellites such as the Landsat series and the Gravity Research and Climate Experiment (GRACE) system, with costs typically on the order of one billion dollars per satellite and with concept-to-launch timelines on the order of two decades (for new missions). More recently, the proliferation of smartphones has helped to miniaturise sensors and energy requirements, facilitating advances in the use of CubeSats that can be launched by the dozens, while providing 3–5 m resolution sensing of the Earth on a daily basis. Start-up companies that did not exist five years ago now operate more satellites in orbit than any space agency and at costs that are a mere fraction of an agency mission. With these advances come new space-borne measurements, such as high-definition video for understanding real-time cloud formation, storm development, flood propagation, precipitation tracking, or for constructing digital surfaces using structure-from-motion techniques. Closer to the surface, measurements from small unmanned drones and tethered balloons have mapped snow depths, floods, and estimated evaporation at sub-meter resolution, pushing back on spatiotemporal constraints and delivering new process insights. At ground level, precipitation has been measured using signal attenuation between antennae mounted on cell phone towers, while the proliferation of mobile devices has enabled citizenscience to record photos of environmental conditions, estimate daily average temperatures from battery state, and enable the measurement of other hydrologically important variables such as channel depths using commercially available wireless devices. Global internet access is being pursued via high altitude balloons, solar planes, and hundreds of planned satellite launches, providing a means to exploit the Internet of Things as a new measurement domain. Such global access will enable real-time collection of data from billions of smartphones or from remote research platforms. This future will produce petabytes of data that can only be accessed via cloud storage and will require new analytical approaches to interpret. The extent to which today's hydrologic models can usefully ingest such massive data volumes is not clear. Nor is it clear whether this deluge of data will be usefully exploited, either because the measurements are superfluous, inconsistent, not accurate enough, or simply because we lack the capacity to process and analyse them. What is apparent is that the tools and techniques afforded by this array of novel and game-changing sensing platforms presents our community with a unique opportunity to develop new insights that advance fundamental aspects of the hydrological sciences. To accomplish this will require more than just an application of the technology: in some cases, it will demand a radical rethink on how we utilise and exploit these new observation platforms to enhance our understanding of the Earth system.

Description: Accurate and long-term rainfall estimates are the main inputs for several applications, from crop modeling to climate analysis. In this study, we present a new rainfall data set (SM2RAIN-CCI) obtained from the inversion of the satellite soil moisture (SM) observations derived from the ESA Climate Change Initiative (CCI) via SM2RAIN (Brocca et al., 2014). Daily rainfall estimates are generated for an 18-year long period (1998–2015), with a spatial sampling of 0.25° on a global scale, and are based on the integration of the ACTIVE and the PASSIVE ESA CCI SM data sets.The quality of the SM2RAIN-CCI rainfall data set is evaluated by comparing it with two state-of-the-art rainfall satellite products, i.e. the Tropical Measurement Mission Multi-satellite Precipitation Analysis 3B42 real-time product (TMPA 3B42RT) and the Climate Prediction Center Morphing Technique (CMORPH), and one modeled data set (ERA-Interim). A quality check is carried out on a global scale at 1° of spatial sampling and 5 days of temporal sampling by comparing these products with the gauge-based Global Precipitation Climatology Centre Full Data Daily (GPCC-FDD) product. SM2RAIN-CCI shows relatively good results in terms of correlation coefficient (median value  〉  0.56), root mean square difference (RMSD, median value