The Single Calculus Chain (SCC) is the standard EARLINET tool to perform automatic and quality checked analysis of raw lidar data. It is composed by the following modules:
- HiRELPP (High Resolution ELPP)
- CloudScreen (SCC cloud screen module)
- ELPP (EARLINET Lidar Pre-Processor)
- ELDA (EARLINET Lidar Data Analizer)
- ELDEC (EARLINET Lidar DEpolarization Calibrator)
- ELIC (EARLINET LIdar Calibrator)
- ELQUICK (EARLINET Lidar QUICJKlook)
The HiRELPP module implements the corrections to be applied to the raw lidar signals before they can be used to derive higher level products. All the operations implemented in HiRELPP are designed to preserve both the vertical and time resolution as high as possible. Some instrumental effects (like for example, dead-time correction, trigger-delay correction, overlap correction, atmospheric and electronic background subtraction, low- and high-range automatic signal glueing) are corrected following the recommendations provided by the EARLINET quality assurance program.
The dead-time corresponds to a maximum count rate. The dead- time causes a non-linearity between the actual intensity at the photo-multiplier photocathode and the counted events, which can be described theoretically by means of photon statistics. Actual detector can be modelled as the paralyzable and the non-paralyzable model. Once information about the model to use for describing the counting system and the dead-time value is determined, based on standard operating procedures defined by the ACTRIS Center for Aerosol remote Sensing), these are provided to HiRELPP and the acquired counts are corrected for the dead-time effect.
In general, the data acquisition unit of a lidar system gets a trigger from the laser to start the signal recording. Due to the electronic circuits in the laser and in the data acquisition unit, there is always a delay between the outgoing laser pulse and the time at which the acquisition system actually starts to record the lidar profile. If this trigger delay is not properly taken into account, a systematic error is made in associating each lidar range bin with the corresponding atmospheric range. Once the correct measurement of the real trigger delay is done for each detection channel following the procedure indicated CARS, such information is inserted in the SCC configuration and HiRELPP correct acquired lidar signals for the trigger-delay.
Atmospheric and electronic background subtraction¶
The lidar signal has a constant background made of an atmospheric component and an electric component. This background can be determined either in the far range of the lidar signal, far enough that the expected contribution from atmospheric backscatter is negligible, or in the pre-trigger range before the laser pulse, where the signal must be free of electronic distortions. Each one of this option can be defined into HiRELPP. Additionally, it is possible to subtract so-called dark signals, which are measured, for example, with a fully obscured telescope so that no light from the atmosphere reaches the detectors and only eventual electronic distortions are left. This allows HiRELPP to remove potential distortions affecting analog lidar signals.
Low- and high-range automatic signal glueing¶
Lidar signals can cover a quite large dynamic range, because the intensity of the light backscattered from the aerosol-laden boundary layer in the near range (e.g. at 0.5 km altitude) is several orders of magnitudes higher than the intensity of the light backscattered from the rather clean troposphere (e.g. at 10 km altitude). As it is demanding to cover this large dynamic range with one data acquisition channel with linear response, several approaches are used to overcome this problem. One option is to split the signal output from a single photomultiplier into two signals and to record one signal using analog detection mode and the other with the photon-counting method. Another option is to split the lidar signal optically using a beam splitter and to detect the split components with two detectors and subsequent data acquisitions. A third option is to use two (or more) telescopes with separate detection electronics. Both SCC preprocessors (HiRELPLP and ELPP) glues the signals for the first 2 options, while gluing is implemented directly at optical property level in the third case (ELDA). Before gluing, the near-range and the far-range signals need to be screened for low-level clouds, corrected for instrumental effects like dead time, trigger delay, etc., and the backgrounds have to be subtracted as explained above. HiRELPP and ELPP contains a fully automatic algorithm for the gluing of analog and photon-counting signals as well as for the gluing of two photon-counting signals. The algorithm is implemented through three main steps: the procedure starts with the determination of a first guess of the gluing region, after that, the algorithm optimizes the gluing region performing statistical tests (implemented only in ELPP) and finally, the signals are glued in the optimal gluing region.
The typical HiRELPP products are netCDF pre-processed files containing pre-processed (un-calibrated) range corrected time series at instrumental vertical and time resolution. If the lidar instrument has polarization capabilities the volume linear depolarization ratio is provided as well.
Lidar data contaminated by clouds has to be skipped because the retrieval algorithm implemented in the SCC are optimized for aerosol and may produce unreliable results when applied to clouds. The aim of the CloudScreen module is to detect clouds by ingesting as input un-calibrated high resolution pre-processed range corrected signal timeseries (HiRELPP products). The output of CloudScreen module is a netCDF file containing a 2-dimensional grid (x axis: time y axis: altitude) with the same resolution as the corresponding HiRELPP product, in which each pixel is flagged as cloud free or cloud contaminated. This information is then transferred to other SCC modules for the automatic removal of the cloud contribution within the aerosol optical property products.
The ELPP module implements all the needed corrections and transformations to be applied to the raw data before they can be used to derive the optical products at low temporal/spatial resolution. As HiRELPP, ELPP implements correction of some instrumental effects (like for example, dead-time correction, trigger-delay correction, overlap correction, atmospheric and electronic background subtraction, low- and high-range automatic signal glueing) following the recommendations provided by the EARLINET quality assurance program. Additionally, to HiRELPP, time integration or vertical smoothing is performed by ELPP to meet the required condition on the products statistical error (defined in the SCC database for each data product type). ELPP makes also advantage of the CloudScreen output products so that signals affected by low clouds are automatically removed already at level of lidar pre-processor. Besides these corrections, ELPP is also responsible to generate the molecular signal needed to calculate the aerosol optical products. In both aerosol backscatter (Klett, 1981; Fernald, 1984; Di Girolamo et al., 1999; Ansmann et al., 1992a; Ferrare et al., 1998) and extinction (Ansmann et al., 1990, 1992b) retrievals the molecular contribution to the atmospheric extinction and transmissivity are required as input, which are calculated by ELPP at the emission and detection wavelengths in terms of vertical profiles at the same vertical resolution as the pre-processed lidar signals. The molecular number density profile is calculated by ELPP from vertical profiles of temperature T(z) and pressure P(z) using the ideal gas law and assuming as 1 the value of the air compressibility factor. Temperature and pressure profiles are either calculated from standard atmosphere model or taken from the measurements of a close-by radiosounding that can be provided to the SCC as a separate input file or provided by model data profiles. Once the molecular number density is obtained, the calculation of the molecular optical parameters, i.e., the backscatter and extinction coefficients, is done following the procedure reported in Bucholtz (1995) and Miles et al. (2001). More details about implemented algorithms in ELPP are reported in D’Amico et al., (2016). The typical ELPP products consist of netCDF pre-processed files containing low resolution pre-processed (un-calibrated) range corrected time series.
ELDA applies the algorithms for the retrieval of aerosol optical parameters to the low resolution pre-processed signals, produced by ELPP module. The module provides aerosol optical products in a flexible way choosing from a set of possible pre-defined analysis procedures.
ELDA implements: - retrieval of aerosol extinction profile - retrieval of Raman aerosol backscatter profile - retrieval of elastic aerosol backscatter profile - particle/volume depolarization ratio profile
An automatic vertical-smoothing and time-averaging technique selects the optimal smoothing level as a function of altitude on the base of different thresholds on product uncertainties fixed in the SCC database for each product. Currently, ELDA delivers only optical products at a single wavelength (so for a multi-wavelength lidar, ELDA generates several independent optical products each referring to a single wavelength). Full description of implemented algorithms is reported in Mattis et al., (2016). For all products and retrieval algorithms, the user can choose whether the statistical uncertainties shall be calculated with the Monte Carlo method or by means of error propagation. The only exception are retrievals with the Klett-Fernald a lgorithm for which the estimation of uncertainties is implemented only with Monte Carlo method. Currently, the separated handling of statistical errors of the lidar signals, of systematic errors of the lidar signals, and of uncertainties of the retrieval algorithms is under research within the EARLINET community. ELDA allows for the automated vertical smoothing and temporal averaging of the derived products. The user has the option to adjust the degree of smoothing and averaging of each individual product by setting several parameters. In general, those parameters and constraints can be defined for two different altitude regions, below and above 2 km altitude. Two threshold values for the maximum allowable relative statistical error of the product below and above 2 km altitude (meaning high expected aerosol load and low aerosol load, respectively) can be defined. Beside these user-defined constraints, there are fixed limitations concerning the maximum allowable smoothing and averaging: it is not allowed to apply a smoothing that would result in effective vertical resolutions larger than 500m and 2km below and above 2km altitude, respectively. All methods of calculating profiles of particle backscatter coefficients include a certain calibration procedure. Usually a particle-free region in the free troposphere where the aerosol backscatter is assumed as null is used for calibration. A calibration window of user-defined width is shifted through the altitude region, where particle-free conditions typically occur (user-defined calibration interval). For each window position, the average and standard deviation of the signal or signal ratio is calculated. It is assumed that the window position where the signal or signal ratio has its minimum is closest to the assumed particle-free conditions. The average value within this calibration window and its standard deviation are used to estimate the calibration factor and its statistical uncertainty. If the user knows from ancillary data, e.g., from sun-photometer observations or from climatological data of the stratospheric particle load, that there is no particle-free altitude layer, it is possible to provide backscatter ratios different from 1 as calibration value. ELDA implements the derivative calculation into the aerosol extinction algorithm as derivative of the pre-processed signals by weighted or non-weighted linear fit method. Finally, concerning the assumptions needed in terms of Angstrom exponent (extinction calculation) and /or lidar ratio (elastic backscatter retrieval), it is possible to define in the SCC configuration the values to be used. In particular it is possible to include a lidar ratio (Angstrom) profile in order to improve the overall quality of the product. These values can be provided to the SCC together with the raw signals and are passed by ELPP to ELDA.
The ELIC module calibrates both high- and low-resolution pre-processed products (HiRELPP end ELPP products respectively) using the same calibration constant computed by ELDA during the retrieval of low-resolution optical aerosol properties (elastic/Raman bacskcatter calibration). As already mentioned, both HiRELPP and ELPP deliver pre-processed range corrected signal timeseries. Pre-processed range corrected signals are not considered robust lidar products because even if they are proportional to the concentration of atmospheric backscatterers, they depend on specific lidar instrumental characteristics as well. In the retrieval of aerosol optical products (like for example aerosol backscatter), the range corrected signals are used as input and special calibration techniques are used to remove the instrumental dependence. The more is the signal to noise ratio the better is the result of these calibration techniques. Usually, a way to increase the signal to noise ratio is to degrade the time and/or space resolution of the input signals. In general, it is more demanding to get a reliable calibration when working with high resolution lidar data. This is the reason why in the SCC workflow, the calibration is done by ELDA which deals with un-calibrated low-resolution range corrected signals. Anyway, if we assume that the instrumental conditions are stable in the time interval in which the measurements take place, it is possible to use the calibration constants retrieved by ELDA calibrating low resolution signals also to calibrate the high resolution timeseries measured in the same time window. This is the main goal of the ELIC module which runs right after ELDA, gets the calibration constants retrieved by ELDA for all lidar channels and calibrates the corresponding high- and low-resolution range corrected signal timeseries. The ELIC products are netCDF files containing fully calibrated quantities like total attenuated backscatter and volume depolarization ratio.
All the participating stations operate lidar equipped with at least 2 channels detecting independent polarization states of backscattered light and, as consequence, can deliver atmospheric volume/particle depolarization ratio profiles. Anyway, to calculate the volume/particle depolarization ratio from the ratio of these polarization channels an accurate calibration is needed. ELDEC module provide this calibration parameter following the quality assurance procedures defined within ACTRIS CARS (Centre for Aerosol Remote Sensing). In particular, the depolarization calibration is made by submitting to the SCC special raw depolarization calibration datasets.
The ELQUICK module generates standardized lidar quicklook for the whole ACTRIS/EARLINET network. Lidar quicklooks (png images) are useful representation of the high resolution timeseries of total attenuated backscatter and/or volume depolarization profiles contained in the ELIC products which can be considered as the two-dimensional pixel grid. The number of vertical pixels of this grid is the number of points of the total attenuated backscatter (or volume depolarization ratio) vertical profile while the number of horizontal pixels is to the number of total attenuated backscatter (or volume depolarization ratio) profiles included in the time series. The color corresponding to each individual pixel is, instead, connected to the value of the total attenuated backscatter (or volume depolarization ratio) at a given altitude and time. In this way, by observing such quicklook images it is easy to visualize aerosol layers and their evolution in both time and space.
All the SCC products are files in Network Common Data Form (NetCDF) which is a well known self-describing, machine-independent data format that support the creation, access, and sharing of array-oriented scientific data. For more information about NetCDF format: http://www.unidata.ucar.edu/software/netcdf/.
The NetCDF is a binary format that allows the definition of multi-dimensional variables of several types (integers, double, character, etc). For each variable it is possible to define one or more attributes where to specify variable properties like units, long name, description, etc.
It is possible to define global attributes which are not related to a specific variable but to the whole file.
A NetCDF file is composed by four different section:
- this section contains all the dimensions used in the definition of all the variables included in the NetCDF file
- this section contains all the variables stored in the NetCDF file. Each variable is defined as a multi-dimensional array of a specific type and with all the dimensions defined in the dimensions section
- global attributes
- this section lists all the attributes referring to the whole file. As the variable the attributes (global or the one attached to a specific variable) can be of different type
- in this section the data contained in each variable defined in variable section is stored. Attribute values (both global or related to a specific variable) are not reported in data section but directly in variable or global attribute sections.
- HiRELPP Products
- CloudScreen Products
- ELPP Products
- ELDEC Products
- ELDA Products
- ELIC Products
- SCC Bitmask/flag description