Identification
The document SRON-S5P-LEV2-MA-002 outlines the technical specifications essential for the effective utilization of S5p/TROPOMI Level 2 products, focusing specifically on Carbon Monoxide data This user manual serves as a comprehensive guide for users to understand and apply the information accurately.
Purpose and objective
The Sentinel-5 Precursor (S5p) mission is a low Earth orbit satellite system designed to monitor air quality, climate, and the ozone layer, contributing to the Global Monitoring of the Environment and Security (GMES/COPERNICUS) program It includes a satellite bus, the TROPOspheric Monitoring Instrument (TROPOMI) payload, and a supporting ground system For detailed insights into the mission's objectives, refer to the journal paper in [RD1] and for a comprehensive overview, see [RD2] Additional information about S5p and TROPOMI can be accessed through various dedicated websites, such as [ER1, ER2].
The TROPOMI instrument generates various geophysical (L2) products from collected data, with the raw data processing (L0 – L1b) and L2 data processing detailed in separate algorithm theoretical basis documents (ATBD) This Product User Manual (PUM) outlines the technical characteristics of the S5p/TROPOMI Level 2 geophysical data products, essential for their effective and accurate utilization.
The PUM outlines the standardized format of data files and metadata utilized across all Level 2 products, with a dedicated section focusing on the Carbon Monoxide product.
Document overview
This article provides an overview of S5p L2 products, detailing the necessary information for data acquisition and inspection, as well as support options It includes a comprehensive description of the Carbon Monoxide data product, featuring examples and usage guidelines The structure, format, and metadata of L2 data are explored, followed by an in-depth analysis of Carbon Monoxide data, including units and quality assurance parameters The concluding chapter offers insights into generic metadata, while the Appendix outlines measurement flags, processing quality flags, and surface classifications.
Applicable documents
[AD1] Tailoring of the Earth Observation File Format Standard for the Sentinel 5 precursor Ground Segment. source:ESA/ESTEC;ref:S5P-TN-ESA-GS-106;issue:2.2;date:2015-02-20.
Standard documents
There are no standard documents
Electronic references
[ER1] Tropomi official website URLhttp://www.tropomi.eu.
[ER2] S5P official website URL https://sentinel.esa.int/web/sentinel/missions/ sentinel-5p.
[ER3] Robert B Schmunk; Panoply netCDF, HDF and GRIB Data Viewer URLhttp://www.giss.nasa. gov/tools/panoply/.
[ER4] Infrastructure for Spatial Information in the European Community (INSPIRE) Directive 2007/2/EC URL http://inspire.jrc.ec.europa.eu/.
[ER5] Brian Eaton, Jonathan Gregory, Bob Drach et al.; NetCDF Climate and Forecast (CF) Metadata Conventions Lawrence Livermore National Laboratory (2014) Version 1.7 draft; URL http:// cfconventions.org.
[ER6] ESIP;Attribute Conventions for Dataset Discovery (ACDD) 1st edition (2013) URLhttp://wiki. esipfed.org/index.php/Attribute_Convention_for_Data_Discovery_(ACDD).
[ER7] NetCDF Users Guide (2011) URLhttp://www.unidata.ucar.edu/software/netcdf/docs/ netcdf.html.
[ER8] USGS; Global Land Cover Characteristics Data Base Version 2.0 (2012) Website last visited on March
6, 2017; URLhttps://lta.cr.usgs.gov/glcc/globdoc2_0.
The ECS SDP Toolkit (2012) provides access to DEM and land-sea mask data, which can be downloaded from the NASA EDHS FTP site at ftp://edhs1.gsfc.nasa.gov/edhs/sdptk/DEMdata For more information and resources, visit the official toolkit download page at http://newsroom.gsfc.nasa.gov/sdptoolkit/TKDownload.html.
[ER10] UDUNITS 2 Manual (2011) URLhttp://www.unidata.ucar.edu/software/udunits/.
[ER11] Cooperative Ocean/Atmosphere Research Data Service; Conventions for the standardization of
NetCDF files (1995) URL http://ferret.wrc.noaa.gov/noaa_coop/coop_cdf_profile.html.
3 Terms, definitions and abbreviated terms
Terms, definitions, and abbreviated terms that are specific for this document can be found below.
Terms and definitions
ATBD Algorithm Theoretical Basis Document
Acronyms and Abbreviations
ATBD Algorithm Theoretical Basis Document
DLR Deutsches Zentrum für Luft- und Raumfahrt
KNMI Koninklijk Nederlands Meteorologisch Instituut
IODD Input Output Data Definition
OCRA Optical Cloud Recognition Algorithm
ROCINN Retrieval of Cloud Information using Neural Networks
UPAS Universal Processor for UV/VIS Atmospheric Spectrometers
4 Overview of the Sentinel 5 precursor/TROPOMI Level 2 Products
The Sentinel 5 Precursor (S5P) mission, operational from 2017 to 2023, focuses on delivering vital information regarding air quality and climate as part of the European COPERNICUS program for Earth Observation Utilizing the TROPOMI instrument, the mission conducts daily global assessments of essential atmospheric components, including ozone, nitrogen dioxide, sulfur dioxide, carbon monoxide, methane, and formaldehyde, alongside cloud and aerosol characteristics A comprehensive list of standard S5P/TROPOMI Level 2 products is available, with additional products like the UV index currently in development for future release.
Table 1: Standard S5P L2 products with name, identifier, and responsible institutes.
Product ATBD PUM Identifier Institution
Cloud [RD3] [RD4] L2 CLOUD_ DLR
NPP-VIIRS Clouds [RD5] [RD5] L2 NP_BDx RAL
HCHO [RD6] [RD7] L2 HCHO BIRA/DLR
SO 2 [RD8] [RD9] L2 SO2 _ BIRA/DLR
O 3 Total Column [RD10] [RD11] L2 O3 BIRA/DLR
O 3 Tropospheric Column [RD12] [RD13] L2 O3_TCL IUP/DLR
Aerosol layer height [RD14] [RD15] L2 AER_LH KNMI
Ultra violet aerosol index [RD16] [RD17] L2 AER_AI KNMI
O 3 Full Profile [RD18] [RD19] L2 O3 PR KNMI
O 3 Troposheric Profile [RD18] [RD19] L2 O3_TPR KNMI
Tropospheric NO 2 [RD20] [RD21] L2 NO2 _ KNMI
CO [RD22] This document L2 CO SRON/KNMI
CH 4 [RD23] [RD24] L2 CH4 _ SRON/KNMI
File name convention
The table outlines an identifier that serves as a substring of the real name for S5p products, with complete filename conventions detailed in [RD25, chapter 4] Additionally, there may be intermediate L2 products available within the PDGS framework that are not listed in the table Each product mentioned has an associated Product User Manual (PUM), and both product documentation, such as Algorithm Theoretical Basis Documents (ATBDs) and PUMs, will be updated with new processor releases User documentation can be accessed via the TROPOMI website [ER1], while further information about the S5p mission is available on the official ESA website for the Sentinel 5 Precursor mission [ER2].
In the current PUM the Carbon Monoxide product is described and an example of the full real name is as following:
S5P_NRTI_L2 CO _20140101T000000_20140102T000000_00099_01_000200_20141010T173511.nc The components of this file name are given in table 2
The S5P product file name consists of various components that are separated by underscores, with the file extension indicated by a period at the end Character indexing begins at 0, and the end-index follows Python's convention, marking the first character that is not included in the specified block.
4 8 4 Processing stream, one of “NRTI” (near real-time), “OFFL” (offline) or “RPRO”
9 19 10 Product identifier, as listed in table 1
20 35 15 Start of granule in UTC as “YYYYMMDDTHHMMSS” The “T” is a fixed character.
36 51 15 End of the granulein UTC as “YYYYMMDDTHHMMSS” The “T” is a fixed character.
61 67 6 Processor version number as “MMmmpp”, with “MM” the major version number, “mm” the minor version number, and “pp” the patch level.
68 83 15 The time of processing for this granule in UTC as “YYYYMMDDTHHMMSS” The “T” is a fixed character.
84 86 2 The file name extension All Sentinel 5 precursor files are netCDF-4 files and use the extension “nc”
5 Data Distribution and Product Support
In this chapter, information on TROPOMI Carbon Monoxide product data distribution and support are given.
Information to supply with a support request
We have meticulously prepared the processors, processing system, and data distribution system for the Sentinel 5 Precursor mission's level 2 products Despite our careful work, you may encounter issues when reading the level 2 files or have questions regarding the product user manual or the ATBD For assistance, please reach out to the Earth Observation Help Desk at ESA by emailing EOSupport@copernicus.esa.int, and be sure to specify that your inquiry is related to the Sentinel 5 Precursor (S5p) / TROPOMI mission.
When seeking technical support, it's essential to include specific details about the file you are attempting to read A straightforward method to achieve this is by generating a "dump" of the file's header using the "ncdump" tool from the netCDF-4 library Providing just the header information is sufficient for assistance.
Using the command “ncdump -h FILE.nc > FILE.cdl” allows you to extract all metadata from the specified NetCDF file, enabling you to understand the file's production details Remember to substitute FILE.nc with the actual name of your file in the command.
When encountering issues with header generation, please provide the original file name of the granule, the exact error message, and the software version, including netCDF-4 and HDF-5 versions, to facilitate a prompt response Additionally, including a checksum to verify file integrity can expedite our troubleshooting process, ensuring a more efficient resolution to the problem.
6 General Reader and Visualisation Tools
Panoply is a versatile cross-platform application designed for visualizing and reading geo-gridded data from various datasets, including netCDF, HDF, and GRIB formats, as well as Sentinel 5 precursor Level 2 datafiles With Panoply 4, users can efficiently plot and analyze complex data arrays.
Slice and plot geo-gridded latitude-longitude, latitude-vertical, longitude-vertical, or time-latitude arrays from larger multidimensional variables.
Slice and plot "generic" 2D arrays from larger multidimensional variables.
Slice 1D arrays from larger multidimensional variables and create line plots.
Combine two geo-gridded arrays in one plot by differencing, summing or averaging.
Plot lon-lat data on a global or regional map using any of over 100 map projections or make a zonal average line plot.
Overlay continent outlines or masks on lon-lat map plots.
Use any of numerous color tables for the scale colorbar, or apply your own custom ACT, CPT, or RGB color table.
Save plots to disk GIF, JPEG, PNG or TIFF bitmap images or as PDF or PostScript graphics files. Export lon-lat map plots in KMZ format.
Export animations as AVI or MOV video or as a collection of invididual frame images.
Launched on October 13, 2017, the Copernicus Sentinel 5 Precursor (S5P) is the inaugural European Sentinel satellite focused on atmospheric composition monitoring Its mission, running from 2017 to 2023, aims to provide global insights into air quality, climate, and the ozone layer The initial six months were dedicated to special observations for satellite commissioning and ground processing systems, with the operational phase commencing in April 2018.
The S5P mission features the TROPOspheric Monitoring Instrument (TROPOMI), developed by The Netherlands in collaboration with the European Space Agency (ESA) TROPOMI is a nadir-viewing shortwave spectrometer designed to measure various wavelengths, including the UV-visible range (270 – 500 nm), near-infrared (710 – 770 nm), and shortwave infrared (2314 – 2382 nm).
The instrument employs passive remote sensing techniques to measure solar radiation reflected and emitted by the Earth at the top of the atmosphere Operating in a push-broom configuration, it captures light across a wide swath simultaneously This light is then dispersed onto two-dimensional imaging detectors, where the swath position is mapped in one direction and the spectral information in another, ensuring comprehensive data collection.
The TROPOMI instrument captures a 2600 km wide strip of the Earth's surface on a two-dimensional detector every second, while the satellite travels approximately 7 km This scanning process enables the detection of various ground pixels in the across-track direction and different wavelengths After each measurement, a new scan begins, allowing continuous monitoring of the Earth's atmosphere as the satellite moves.
~2600 km wavelengths across track (swath)
TROPOMI offers enhanced spatial resolution, measuring 3.5 km² at nadir for bands 2-6 (UVN), 7 km² for bands 7 and 8 (SWIR), and 21-28 km² for band 1 (deep UV) It captures approximately 20 million spectra daily, marking a significant advancement over its predecessors, including OMI, SCIAMACHY, and GOME-2 This high resolution, paired with a wide swath, enables TROPOMI to achieve daily global coverage For detailed geophysical operational data products, refer to section 4.
The S5P satellite will operate in a loose formation with the U.S Suomi NPP satellite, aiming to leverage the high spatial resolution cloud observation capabilities of the VIIRS instrument With a temporal separation of less than 5 minutes and both satellites crossing the equator around 13:30 local solar time, this formation facilitates the creation of synergistic data products and enhances scientific research opportunities.
The spectral range is divided among four distinct detectors, each operating on different geographic grids To effectively combine products from these various detectors, careful re-mapping is necessary to address the spatial discrepancies.
More details on the TROPOMI instrument and the operational concepts can be found in the Level 0 to 1BATBD [RD26, parts I – III].
Figure 3: CO total column mixing ratios of TROPOMI averaged from November 13th to 19th, 2017, from [RD27]
8 S5p/TROPOMI L2 Carbon Monoxide Product Description
Carbon monoxide (CO) is a crucial trace gas that influences tropospheric chemistry and serves as a significant pollutant in urban environments Major sources of CO include fossil fuel combustion, biomass burning, and the atmospheric oxidation of methane (CH4) and other hydrocarbons In Northern mid-latitudes, fossil fuel combustion is the primary CO source, while in tropical regions, biomass burning and isoprene oxidation are more significant The long atmospheric lifetime of methane results in a nearly uniform global distribution of CO The primary sink for CO is its reaction with hydroxyl radicals (OH).
TROPOMI measures global carbon monoxide (CO) levels by utilizing both clear and cloudy sky Earth radiance observations In the shortwave infrared (SWIR) range of 2.3 µm, TROPOMI’s clear sky data offers insights into CO total columns with a focus on the tropospheric boundary layer In contrast, the sensitivity of CO column measurements in cloudy conditions varies based on the light path.
Data Product Examples
Following the successful launch of TROPOMI on October 13, 2017, as part of ESA’s Sentinel-5 Precursor satellite, the first calibrated SWIR radiance data was received on November 9, 2017 The data quality was sufficient to accurately process the CO total column product from the offline data stream Initial global CO observations from TROPOMI were recorded from November 13 to 17 and on the 19th, with no data available on November 18 For our analysis, we focused on observations with a solar zenith angle (SZA) of less than 80 degrees and excluded the two westernmost pixels due to unresolved performance issues Additionally, we included only clear-sky and cloudy observations with cloud heights below 5000 meters, demonstrating the effectiveness of TROPOMI in capturing atmospheric data.
TROPOMI has successfully identified carbon monoxide (CO) enhancements from significant sources such as wildfires in regions like Brazil, Africa, Madagascar, and Australia, as well as from anthropogenic air pollution in India and China The mission's high spatial resolution and excellent signal-to-noise ratio enable the detection of CO increases from weaker regional sources as well Notably, TROPOMI has demonstrated its ability to monitor air pollution daily over urban and industrial areas, which is one of its primary objectives, as illustrated by enhanced CO levels detected near industrial zones.
Figure 4: Total column mixing ratio for individual TROPOMI ground pixels for Italy on 25th December 2017, from [RD28]
Venice, along with Turin, Milan, and Rome, faces significant air pollution challenges The daily global monitoring capabilities of TROPOMI provide valuable insights into the temporal evolution of air quality at the city level This data not only enhances the understanding of the impact of emission regulations but also necessitates accurate assessments of the absolute uncertainty associated with the TROPOMI CO product.
Product Geophysical Validation
Borsdorff et al (2018) conducted a preliminary data validation of the TROPOMI CO data product by comparing it with the near-real-time CO analysis from the ECMWF Integrated Forecasting System (IFS), which incorporates IASI and MOPITT observations They collocated TROPOMI CO retrievals with the 6-hourly CAMS CO fields, interpolating the CAMS data to match the time and location of each TROPOMI measurement By integrating the CAMS profile with the column averaging kernel, they obtained the corresponding column density, considering vertical retrieval sensitivity This methodology enabled a direct comparison between CAMS and TROPOMI data, demonstrating a strong agreement between the two datasets.
The histogram illustrates a minor mean difference of +3.2% between TROPOMI and CAMS CO data, accompanied by a standard deviation of 5.5% The two datasets exhibit a strong correlation, evidenced by a Pearson correlation coefficient of 0.97 Notably, there is a significant agreement between the datasets over oceanic regions, where data is derived solely from cloudy observations in the shortwave infrared spectral range.
Table 3: Ground-based FTIR stations used for validation The latitude and longitude values are given in degrees, the surface elevation in km.
Name Latitude Longitude Altitude Type
The TCCON range at Lauder shows a value of 45.04 169.68 0.37, indicating that the ocean surface appears very dark, particularly under glint observation geometry Consequently, cloud-free measurements often fail to capture enough light for significant data retrieval However, the strong correlation observed reinforces confidence in the TROPOMI CO data product, even under cloudy conditions.
The validation of the TROPOMI CO data product requires independent ground reference observations for both clear-sky and cloudy conditions Borsdorff et al (2018) conducted initial validation using CO measurements from nine TCCON-operated FTS stations across various elevations in the northern and southern hemispheres TROPOMI observations were compared to TCCON data by selecting retrievals from the same day within a 50 km radius of each station, adjusting the CO column based on the station's altitude Daily averages of dry air mixing ratios (XCO) were calculated from the adapted TROPOMI data and TCCON measurements Data gaps in the TROPOMI time series were attributed to high cloud conditions and instrument characterization during commissioning Biases for each TCCON station under different sky conditions, along with standard deviations and coincident daily means, were analyzed to assess the data quality.
Borsdorff et al demonstrated a strong correlation between TROPOMI CO measurements and TCCON data, with a small mean bias of 6.0 ppb for clear skies, 6.2 ppb for cloudy skies, and 5.8 ppb overall The station-to-station deviations were 3.9 ppb for clear skies, 2.4 ppb for cloudy skies, and 2.9 ppb for combined conditions Additionally, the mean standard deviation of the bias was consistent, reinforcing the reliability of the data retrieval for both clear and cloudy scenes This consistency highlights the effectiveness of the SICOR algorithm in ensuring comprehensive data coverage for the TROPOMI CO product.
History of product changes
This manual outlines the L2 Carbon Monoxide product (version 1.0.1), utilizing L1B data from processor version 1.0.0.19194 During the commissioning phase, only minor adjustments were made to the CO retrieval software The latest software update enhances retrieval convergence for cloudy-sky observations by incorporating cloud height estimates from a non-scattering CH4 retrieval, without compromising the data quality of the TROPOMI CO product.
Using the S5p/TROPOMI L2 Carbon Monoxide
The operational S5p level-2 processor employs the SICOR physics-based retrieval algorithm determining
The algorithm utilizes a profile scaling method to determine CO total column abundances from Earth radiance measurements in the 2.3 µm band This approach simultaneously calculates the CO total column along with various effective parameters.
Figure 5 illustrates the differences in total column mixing ratios of CO (TROPOMI - CAMS), averaged over the same timeframe as depicted in Figure 3 This data, sourced from [RD27], is derived from TROPOMI measurements, including factors such as the height and optical depth of the scattering layer, as well as the Lambertian surface albedo For practical applications, it is essential to concentrate on the CO-related output parameters.
1 The retrieved total column of CO.
2 The corresponding column averaging kernel.
Figure 9 illustrates the CO retrieval product over land for a cloudy atmosphere and a surface albedo
The analysis reveals the discrepancies between the true and retrieved CO columns, highlighting minimal differences (approximately 1%) for cloud fractions ranging from 0 to 1 These variations stem from the atmospheric scattering model used in the retrieval process Notably, the presence of clouds does not diminish the retrieved CO column as the sensitivity of measurements regarding CO levels above the cloud effectively informs the total CO concentration.
The accurate estimation of the CO column relies on selecting the correct relative profile, as demonstrated in Figure 9 If the relative profile does not match the true vertical distribution of CO, the column averaging kernel becomes essential for proper interpretation of the retrieval results Additionally, increased cloud coverage enhances the signal-to-noise ratio (SNR), thereby reducing retrieval noise in the CO column This relationship between retrieval sensitivity and cloud coverage is depicted in the column averaging kernels in Figure 9, where line colors represent varying cloud fractions Notably, when the cloud fraction exceeds zero, the column averaging kernel increases above the retrieved cloud height, while retrieval sensitivity decreases below the cloud, resulting in a nearly offsetting effect on the retrieved CO column.
Figure 6: Histogram of the differences shown in Fig 5, from [RD27]
Daily mean dry air column mixing ratios (XCO) were measured by TROPOMI (in pink) and TCCON stations (in blue) for Ascension Island and Reunion, utilizing a co-location radius of 50 km The standard deviation of individual retrievals for each day is represented as error bars.
Figure 8: Mean bias (TROPOMI - TCCON) between co-located daily mean XCO values of TROPOMI and
The TCCON data includes the global mean bias (b¯), which represents the average of all station biases, and the standard deviation of the bias (σ¯) that measures variability across stations Additionally, the average of all standard deviations (std¯) and the mean number of coincident daily mean pairs (n¯) are calculated TROPOMI retrievals are categorized based on sky conditions: clear-sky (yellow), cloudy-sky (blue), and a combination of both (pink), as referenced in [RD28].
The CO data product illustrated in Figure 9 demonstrates simulated SWIR measurements for a scene partially obscured by a water cloud at altitudes between 2 and 3 km, characterized by an optical depth of τcld0 and a surface albedo of A s = 0.05 The left panel shows the difference (∆CO) between the actual CO column and the retrieved CO column, plotted against cloud fraction (f cld) using the true relative CO profile for scaling during inversion The middle panel provides a 1σ retrieval noise estimate based on varying cloud fractions, while the right panel presents the column averaging kernel as a function of altitude across different cloud fractions.
9 General structure of S5P/TROPOMI Level 2 files
This section provides an overview of the fundamental structure of Sentinel 5 Precursor Level 2 files, with detailed insights available in subsections 9.1–9.3 and sections 11–13 A comprehensive description of the variables in the Carbon Monoxide files can be found in section 10 Figure 10 illustrates the generic structure of a TROPOMI Level 2 file, which consists of the file itself as the outermost layer The file is organized into two main groups: "PRODUCT" and "METADATA," each containing sub-groups to facilitate data accessibility The functions of these groups are discussed in detail below.
The product variables address key questions regarding what, when, where, and how well the data is captured This group encompasses essential data fields, including the precision of main parameters, latitude, longitude, and variables that establish observation time Additionally, it outlines necessary dimensions for data, such as time reference, measurement counts in granules, and spectra in measurements Depending on the product, it may also include pressure-level or state-vector dimensions The "qa_value" parameter aggregates processing flags into a single continuous value, representing quality as a percentage—where 100% indicates optimal performance and 0% signifies a processing failure, with intermediate values reflecting varying degrees of quality.
In the ‘PRODUCT’ group a sub-group ‘SUPPORT_DATA’ can be found:
SUPPORT_DATA Additional data that is not directly needed for using and understanding the main data product is stored in sub-groups of this group.
The data in this group is further split up into the following sub groups:
GEOLOCATIONS Additional geolocation and geometry related fields, including the pixel boundar- ies (pixel corners), viewing- and solar zenith angles, azimuth angles, and spacecraft location.
DETAILED_RESULTS Additional output, including state-vector elements that are not the main parameter(s), output describing the quality of the retrieval result, such as a χ 2 value, and detailed processing flags.
The article discusses the additional input data utilized in deriving output results, which includes meteorological data, surface albedo values, and surface altitude It is important to note that while input profile information is not stored within this data, it can be downloaded from other sources.
Metadata encompasses a collection of essential items, including those specified in the header file and mandated by various standards such as INSPIRE, ISO 19115, ISO 19115-2, ISO 19157, and OGC 10-157r3 These standards are designed to enhance the discoverability of datasets.
Metadata will be organized as attributes, with specific standards grouped using sub-groups within the Metadata category Certain attributes, such as those defined by the CF metadata conventions, the Attribute Convention for Dataset Discovery, the NetCDF-4 user guide, and the ESA CCI project, must be included at the global level Adhering to these conventions ensures interoperability, leading to the inclusion of specified global attributes in the output files at the root level.
ALGORITHM_SETTINGS An attribute is added to this group for each key in the configuration file The exact contents differ for each processor.
GRANULE_DESCRIPTION Parameters describing the granule, such as an outline of the geolocations covered in the granule, the time coverage, and processing facility.
QA_STATISTICS Quality assurance statistics This group contains two types of data:
The total pixel count is categorized into several criteria, including the number of input pixels, successfully processed pixels, and those that failed due to specific reasons Additionally, event counting encompasses the number of warnings raised, which include alerts for the South Atlantic Anomaly, sun glint, and solar eclipses.
Histograms of key parameters in the dataset provide a clear visual representation of changes over time, making them a crucial tool for monitoring quality in scientific data Their additive nature enhances the ability to track variations effectively, contributing significantly to data analysis and quality assurance.
ESA_METADATA The metadata items that are required in the ESA header.
ISO_METADATA The ISO metadata items, organized in subgroups.
1 More detailed processing flags indicating precisely why the 100 % value isn’t reached, are available elsewhere in the product.
Root level First level group Second level group Third level group Variable Attributes
METADATA main precision qa_value latitude longitude delta_time scanline ground_pixel time …
DETAILED_RESULTS processing_quality_flags …
ISO_METADATA Attributes and sub-groups
Figure 10: Graphical description of the generic structure of a Level 2 file The elements labelled as a dimension are coordinate variables See section 9 for a full description.
EOP_METADATA The EOP metadata items, organized in subgroups.
The Level 1B processing of metadata, as outlined in the TROPOMI L01b data processor specification [RD44], serves as the foundation for the Level 2 metadata, specifically regarding key elements.
Dimensions and dimension ordering
NetCDF-4 files utilize named and shared dimensions to establish explicit connections between variables and dimensions Key dimensions include "time," which has a length of 1 for S5P, ensuring compatibility with Level 1B and future Sentinel 4 and Level 3 outputs; "scanline," indicating flight direction; and "ground_pixel," which is perpendicular to the flight path For vertical grids in profiles, the "level" dimension represents the interfaces between layers, adhering to CF metadata conventions, while the "layer" dimension encompasses the bulk between levels, with thickness defined for layers and altitude for levels, providing a memory-efficient solution despite partial CF compliance.
Other dimensions can be added as needed, but these names shall be the default for these roles.
The climate and forecast metadata conventions recommend a specific order for dimensions in a vari- able [ER5, section 2.4] Spatiotemporal dimensions should appear in the relative order: “date or time” (T),
“height or depth” (Z), “latitude” (Y), and “longitude” (X) Note that the ordering of the dimensions in CDL, our documentation and C/C++ is row-major: the last dimension is stored contiguously in memory 2
Using straight latitude and longitude is suitable for model parameters; however, the S5P/TROPOMI Level 1B/Level 2 observation grid is irregular Due to its polar orbit, the across track dimension, referred to as 'ground_pixel,' aligns closely with longitude and is designated as the X-dimension, while the along track dimension, known as 'scanline,' corresponds directly with latitude.
The CF conventions suggest incorporating additional dimensions prior to the (T;Z;Y;X) axes, advocating for the use of contiguous (T;Z;Y;X) hyperslabs to effectively distribute data across other dimensions.
We advise against following the standard recommendation and instead suggest keeping frequently accessed units together in memory while adhering to the order of (T;Y;X) It's important to note that we deviate from the CF conventions for profiles, as these are typically accessed as complete profiles rather than in horizontal slices.
Tropospheric NO 2 column This variable contains a single value per ground pixel, and the dimensions are (time, scanline, ground_pixel).
The O 3 profile variable consists of a column for each ground pixel, with a clearly defined vertical axis Its dimensions are organized as (time, scanline, ground_pixel, level), ensuring a structured representation of the data.
CF conventions in this case as ozone profiles are more likely accessed as complete profiles rather than horizontal slices.
The state_vector_length variable that accompanies the state_vector_length dimension is a string array, giving the names of the state vector elements.
2 Fortran uses column-major order, effectively reversing the dimensions in the code compared to the documentation.
Time information
Time information is organized in two stages, with the primary time dimension representing a reference time set to UTC midnight before the start of the orbit, defined by spacecraft midnight This reference time is quantified in seconds since January 1, 2010, UTC midnight, and alternative representations are detailed in Table 4 Individual measurements within the granule are offset in milliseconds from this reference time, indicated by the variable delta_time This dual reference system aligns with CF conventions, as the flight direction connects latitude and time within the orbit, resulting in closely related Y and T dimensions.
By dividing the data into a time dimension of length 1 and a scan line dimension, we achieve independent Y and T dimensions The actual observation time for each individual observation can be reconstructed using an offset and a time delta.
The 'time_utc' variable serves users by storing observation times as a string array, with each entry formatted as an ISO date string.
Table 4: Reference times available in a S5P L2 file Types: (A) global attribute, (D) dimensional variable, (V) variable All reference times ignore leap seconds.
The article discusses various time reference formats, including the ISO date/time string, which is labeled as (A) It also mentions the calculation of days since January 1, 1950, UTC midnight, utilized in various weather and climate models such as ECMWF and TM5 Additionally, it covers the Julian date of the reference time, which is relevant in the field of astronomy.
The IDL reference time system utilizes two key variables: "time_reference_seconds_since_1970," which represents the total seconds elapsed since January 1, 1970, at midnight UTC, commonly referred to as the Unix epoch, and "time," which indicates the number of seconds since the year 2010 This framework ensures compatibility with various time functions across different systems.
01-01, UTC midnight. time_utc (V) Array of ISO date/time strings [RD45], one for each obser- vation, i.e one for each element in the scanline dimension
Geolocation, pixel corners and angles
The level 2 files contain latitude, longitude, pixel corner coordinates, angles, and satellite position data sourced from the level 1B input data For detailed definitions, refer to chapters 26 and 27 of the relevant documentation It is important to note that the latitude and longitude values are not adjusted for local surface altitude; instead, they represent the intersection of the line of sight with the WGS84 ellipsoid.
The geo-coordinates of the pixel corners are shown in Figure 11 Note that this choice follows the CF metadata standard [ER5, section 7.1].
The azimuth angles, specifically the solar azimuth angle (ϕ0) and the viewing azimuth angle (ϕ), represent the angles of the sun and the instrument at the intersection of the line of sight with the WGS84 ellipsoid, measured in degrees east from local north This definition aligns with the azimuth angles used in the OMI and GOME-2 instruments; however, caution is needed when comparing these angles to a radiative transfer model Typically, a radiative transfer model utilizes ϕ and ϕ0, which differ by 180 degrees, as it follows the path of light.
Figure 11: Pixel corner coordinates The sequencef0;1;2;3grefers to the elements in thecornerdimension.
10 Description of the CO product
Description of the main output file for the CO Column product from the TROPOMI instrument on the Sentinel 5-precursor mission.
These are the file-level attributes.
In the absence of ECMWF dynamic auxiliary data, a fallback solution will be implemented, and the Level 2 output file will be marked with the "Status_MET_2D" global attribute.
In the absence of TM5 dynamic auxiliary data, a fallback solution will be implemented, and the Level 2 output file will be marked with the global attribute “Status_CTM_CO” or “Status_CTMCH4.”
Group attributes attached to CO
Conventions ‘CF-1.7’ (static) NC_STRING
The dataset adheres to climate and forecast metadata conventions, although some features, such as hierarchical data organization through groups, are not included in version 1.6 of the CF metadata conventions In these instances, we aim to maintain the essence of the conventions This attribute is derived from the NUG standard and is represented by the dynamic NC_STRING ‘%(institute)s’.
The ProcessingCenter attribute identifies the institute responsible for the original data, combining values from organizations such as BIRA, DLR, ESA, FMI, IUP, KNMI, MPIC, and SRON This value reflects both the ATBD institute and the institute that developed the processor, adhering to the NUG standard The source of this data is the Sentinel 5 precursor, TROPOMI, which utilizes space-borne remote sensing at Level 2.
The production method of original data encompasses several key elements: it includes the instrument used, a generic description of the data retrieval process, the product level, a brief product name, and the processor version This attribute is derived from the CF standard, ensuring consistency and reliability in data representation.
The system maintains an audit trail for changes made to the original data by ensuring that well-behaved generic netCDF filters automatically add their names and the parameters used during invocation to the global history attribute of the input netCDF file Each entry in this attribute starts with a timestamp that records the date and time of the program's execution, adhering to the standards set by NUG and CF.
Miscellaneous information about the data or methods used to produce it.
In cases where processing occurs in a degraded mode, it is essential to document this in the relevant attribute Degraded processing may arise from various factors, such as utilizing static backup data instead of dynamic inputs or relying on irradiance products that are several days old For a machine-readable description, refer to the "processing_status" attribute, which is based on the CF standard.
The Climate Change Initiative (CCI) project by the European Space Agency (ESA) introduces a unique tracking ID, which is a version 4 UUID This identifier enables the referencing and linking of files to processing descriptions, input data, and documentation, ensuring consistency with CMIP5 standards The attribute is derived from the CCI standard and is dynamically represented as ‘%(logical_filename)s’ (NC_STRING).
The “id” and “naming_authority” attributes are designed to ensure a globally unique identification for each dataset, with the “id” value aiming to distinctly identify the dataset The naming authority further refines the “id,” and together, they create a permanent global uniqueness The “id” attribute is derived from the CCI standard and utilizes the logical file name Additionally, the time_reference is formatted as an ISO 8601 string, represented as 'YYYY-MM-DDT00:00:00Z', which corresponds to the UTC time at midnight before the granule starts, along with a time reference in days since 1950.
The reference time expressed as the number of days since 1950-01-01 This is the reference time unit used by both TM5 and ECMWF. time_reference_julian_day 0.0 (dynamic) NC_DOUBLE
The reference time expressed as a Julian day number. time_reference_seconds_- since_1970
The reference time in Unix systems is defined as the number of seconds since January 1, 1970, at 00:00:00 UTC The start of the data granule is indicated by the variable `time_coverage_start`, formatted as an ISO 8601 string (YYYY-MM-DDTHH:MM:SS.mmmmmmZ), while the end is marked by `time_coverage_end`, also in ISO 8601 format Additionally, the duration of the time coverage is specified by the variable `time_coverage_duration` For further details, refer to the discussion on the `time_delta` variable.
Duration of the data granule as an ISO 8601 [RD45] duration string (“PT%(duration_seconds)sS”) This attribute originates from the CCI standard. time_coverage_resolution NC_STRING
The interval between measurements in the data granule is specified as an ISO 8601 duration string (“PT%(interval_- seconds)fS”), typically set at 1080 ms for most products during nominal operation However, the “L2 O3 PR” product utilizes a longer interval of 3240 ms due to coaddition This measurement attribute is derived from the CCI standard, specifically for orbit 0 (dynamic) NC_INT.
The absolute orbit number, starting at 1 – first ascending node crossing after spacecraft separation For pre-launch testing this value should be set to “ 1”. references ‘%(references)s’ (static) NC_STRING
References that describe the data or methods used to produce it This attribute originates from the CF standard. processor_version ‘%(version)s’ (dynamic) NC_STRING
The version of the data processor, as string of the form “major.minor.patch”. keywords_vocabulary ‘AGU index terms, http://publications.agu.org/author- resource-center/index-terms/’ (static)
The article outlines the guidelines for the keywords attribute, utilizing index terms published by the American Geophysical Union (AGU) It specifies that keywords, represented dynamically as ‘%(keywords_agu)s’, should be derived from the "keywords_vocabulary" that accurately describes the file's content, as provided by the ATBD authors Additionally, it references the standard name vocabulary established by the NetCDF Climate and Forecast Metadata Conventions.
Standard Name Table (v29, 08 July 2015), http:// cfconventions.org/standard-names.html’ (static)
The standard_name attributes are defined in a table format, with the naming_authority specified as ‘%(naming_authority)s’ (dynamic) NC_STRING, indicating the source of the id attribute based on the CCI standard Additionally, the cdm_data_type is designated as ‘Swath’ (static) NC_STRING.
Group “PRODUCT” in “CO ”
Group “SUPPORT_DATA” in “PRODUCT”
10.1.1.1 Group “GEOLOCATIONS” in “SUPPORT_DATA”
Variables in CO /PRODUCT/SUPPORT_DATA/GEOLOCATIONS satellite_latitudein CO /PRODUCT/SUPPORT_DATA/GEOLOCATIONS
Description: Latitude of the geodetic sub satellite point on the WGS84 reference ellipsoid.
Attributes: Name Value Type long_name ‘sub satellite latitude’ (static) NC_STRING units ‘degrees_north’ (static) NC_STRING comment ‘Latitude of the geodetic sub satellite point on the
NC_STRING valid_min -90.0 (static) NC_FLOAT valid_max 90.0 (static) NC_FLOAT satellite_longitudein CO /PRODUCT/SUPPORT_DATA/GEOLOCATIONS
Description: Longitude of the geodetic sub satellite point on the WGS84 reference ellipsoid.
Attributes: Name Value Type long_name ‘satellite_longitude’ (static) NC_STRING units ‘degrees_east’ (static) NC_STRING comment ‘Longitude of the geodetic sub satellite point on the
NC_STRING valid_min -180.0 (static) NC_FLOAT valid_max 180.0 (static) NC_FLOAT satellite_altitudein CO /PRODUCT/SUPPORT_DATA/GEOLOCATIONS
Description: The altitude of the satellite with respect to the geodetic sub satellite point on the WGS84 reference ellipsoid.
The satellite altitude is measured in meters and refers to the height of the satellite above the geodetic sub-satellite point on the WGS84 reference ellipsoid.
NC_STRING valid_min 700000.0 (static) NC_FLOAT valid_max 900000.0 (static) NC_FLOAT satellite_orbit_phasein CO /PRODUCT/SUPPORT_DATA/GEOLOCATIONS
Description: Relative offset[0:0;:::;1:0]of the measurement in the orbit.
Attributes: Name Value Type long_name ‘fractional satellite orbit phase’ (static) NC_STRING units ‘1’ (static) NC_STRING comment ‘Relative offset [0.0, , 1.0] of the measurement in the orbit’ (static)
NC_STRING valid_min -0.02 (static) NC_FLOAT valid_max 1.02 (static) NC_FLOAT solar_zenith_anglein CO /PRODUCT/SUPPORT_DATA/GEOLOCATIONS
The solar zenith angle (ϑ0) is measured from the vertical at ground pixel locations on the reference ellipsoid, with the European Space Agency (ESA) defining the day side as when ϑ0 is less than 92 degrees Pixels are processed when the zenith angle falls within the range of ϑ0 to ϑ0 max, typically between 80 and 88 degrees, depending on the specific algorithm used The maximum value for ϑ0 can be located in the algorithm's metadata settings.
Dimensions: time, scanline, ground_pixel.
The solar zenith angle, defined as the angle measured away from the vertical at the ground pixel location on the reference ellipsoid, is represented in degrees Its valid range is between 0.0 and 180.0 degrees The associated geospatial coordinates, latitude and longitude, are located in a different group, and the climate and forecast metadata conventions do not specify how to link these coordinates.
NC_STRING solar_azimuth_anglein CO /PRODUCT/SUPPORT_DATA/GEOLOCATIONS
The solar azimuth angle at a specific ground pixel on the reference ellipsoid is defined as the angle measured clockwise from North, with North at 0 degrees, East at 90 degrees, South at 180 degrees, and West at 270 degrees This definition is consistent with the terminology used in both OMI and GOME-2 level 1B files.
See the note on theviewing_azimuth_angleon the calculation of the relative azimuht angle as used in radiative transfer calculations.
Dimensions: time, scanline, ground_pixel.
The solar azimuth angle, defined as 'solar azimuth angle' with the standard name 'solar_azimuth_angle', is measured in degrees and has a valid range from -180.0 to 180.0 This angle is specified for the ground pixel location on the reference ellipsoid, with measurements taken clockwise from the North (East = 90°, South = 180°, West = 270°) However, the climate and forecast metadata conventions do not clarify how to specify related geospatial coordinates, as latitude and longitude are categorized in a different group.
NC_STRING viewing_zenith_anglein CO /PRODUCT/SUPPORT_DATA/GEOLOCATIONS
Description: Zenith angle of the satelliteϑ at the ground pixel location on the reference ellipsoid Angle is measured away from the vertical.
Dimensions: time, scanline, ground_pixel.
The viewing zenith angle, defined as 'viewing_zenith_angle', is measured in degrees and has a valid range from 0.0 to 180.0 This attribute is associated with the satellite's angle at the ground pixel location on the reference ellipsoid, indicating how far the angle is from the vertical While the latitude and longitude coordinates are categorized in a separate group, the climate and forecast metadata conventions do not specify how to link these geospatial coordinates.
NC_STRING viewing_azimuth_anglein CO /PRODUCT/SUPPORT_DATA/GEOLOCATIONS
The satellite azimuth angle at a ground pixel location on the reference ellipsoid is defined as the angle measured clockwise from true North, with North at 0°, East at 90°, South at 180°, and West at 270° This definition is consistent with the standards used in both OMI and GOME-2 level 1B files.
To calculate the azimuth difference ϕ ϕ0 it is not sufficient to just subtract solar_- azimuth_anglefromviewing_azimuth_angle The angle needed for radiative trans- fer calculations is(180 (ϕ ϕ 0 ))mod 360
Dimensions: time, scanline, ground_pixel.
The 'viewing azimuth angle' is defined as a static attribute with a long name and standard name both labeled as 'viewing_azimuth_angle' It is measured in degrees, with a valid range from -180.0 to 180.0 degrees The associated geospatial coordinates are indicated as '/PRODUCT/longitude' and '/PRODUCT/latitude', though the climate and forecast metadata conventions do not specify how to relate these coordinates This angle represents the satellite azimuth angle at the ground pixel location on the reference ellipsoid, measured clockwise from North, where East is 90 degrees, South is 180 degrees, and West is 270 degrees.
NC_STRING latitude_boundsin CO /PRODUCT/SUPPORT_DATA/GEOLOCATIONS
The article describes the latitude of the corners of ground pixels in the data, with latitude and longitude coordinates calculated for both the center and corners of these pixels based on the WGS84 ellipsoid.
The order of the pixel corners follows the CF-metadata conventions [ER5, section 7.1], i.e. the ordering is counter-clockwise when viewed from above A graphical representation is given in figure 11.
Dimensions: time, scanline, ground_pixel, corner.
Source: Processor. longitude_boundsin CO /PRODUCT/SUPPORT_DATA/GEOLOCATIONS
The article describes the longitude of the pixel corners associated with ground pixels in the dataset It explains that the latitude and longitude coordinates for both the center of the ground pixels and their corners are determined based on the WGS84 ellipsoid model.
The order of the pixel corners follows the CF-metadata conventions [ER5, section 7.1], i.e. the ordering is counter-clockwise when viewed from above A graphical representation is given in figure 11.
Dimensions: time, scanline, ground_pixel, corner.
Source: Processor. geolocation_flagsin CO /PRODUCT/SUPPORT_DATA/GEOLOCATIONS
This article discusses additional flags that characterize ground pixels, highlighting factors such as the impact of solar eclipses, potential sun glint, and the pixel's position in the orbit—whether it is descending or on the night side It also notes the significance of the pixel crossing the dateline for plotting purposes and addresses the possibility of geolocation errors.
Dimensions: time, scanline, ground_pixel.
The article discusses the static parameters used in geospatial data representation, specifically focusing on the FillValue set to 255 for NC_UBYTE coordinates, which include '/PRODUCT/longitude' and '/PRODUCT/latitude' It also outlines the flag_masks, which consist of values 0, 1, 2, 4, 8, 16, and 128, and defines the flag_meanings as 'no_error', 'solar_eclipse', 'sun_glint_possible', 'descending night', 'geo_boundary_crossing', and 'geolocation_error'.
The ground pixel quality flag is defined with static values, including flag values of 0, 1, 2, 4, 8, 16, and 128 It has a maximum value of 254 and a minimum value of 0, with units specified as '1'.
10.1.1.2 Group “DETAILED_RESULTS” in “SUPPORT_DATA”
Variables in CO /PRODUCT/SUPPORT_DATA/DETAILED_RESULTS processing_quality_flagsin CO /PRODUCT/SUPPORT_DATA/DETAILED_RESULTS
The processing quality flag signifies errors encountered during the processing of a specific pixel, resulting in a fill value in the output, as well as warnings that may impact the quality of the retrieval result For a comprehensive understanding, please refer to the detailed description in Appendix A.
Dimensions: time, scanline, ground_pixel.
Attributes: Name Value Type long_name ‘Processing quality flags’ (static) NC_STRING comment ‘Flags indicating conditions that affect quality of the retrieval.’ (static)
The NC_STRING flag_meanings encompass a variety of error and warning indicators relevant to data processing in atmospheric science Key errors include 'success', 'radiance_missing', and 'irradiance_missing', which signal issues with input data integrity Additional errors such as 'memory_error', 'initialization_error', and 'configuration_error' highlight potential system or setup failures Warnings related to atmospheric conditions include 'cloud_warning', 'sun_glint_warning', and 'altitude_consistency_warning', which suggest caution in interpreting results Furthermore, specific filters like 'cloud_fraction_fresco_filter' and 'ocean_filter' are employed to refine data analysis Understanding these flags is crucial for ensuring accurate atmospheric measurements and interpretations.