Overview

The “Data processing” section of the SCC is used to upload measurements and ancillary files, monitor the processing progress, and view the output results of the SCC.

The data processing documentation consist of three subsections:

• The procedures of submitting new data for processing are described in the Uploading measurements section.
• The procedures of submitting ancillary files (i.e. sounding, overlap, and lidar ratio files) that are used in the processing are described in the Ancillary files section.
• The procedure to find and explore existing measurements and ancillary files are detailed in the Browse uploaded measurements section.

Quick start

Note

The following discussion assumed that you have already set-up the system settings in the “Admin” section (see Station administration for details).

Upload a measurement

In order to start processing lidar data, you will need to upload them on the SCC server. The files should be in netcdf format, following the specific format described in The SCC input netCDF file format. There is no restriction imposed in the filename, but we suggest to use the format <measurement_ID>.nc. You can upload using the “Quick upload” link in the menu.

When you submit the processed measurements in the quick upload form, you will need to specify the system configuration that was used to perform the measurement. The channels that are defined in the file should be the ones defined in the selected system. The uploading of ancillary files or the selection of categories for the measurement are optional.

When the file is uploaded, it is checked if it conforms to the needed netCDF format. The check guarantees that the mandatory variables are present in the file and that they have the correct format. Note that the check is not exhaustive, in the sense it does not guarantee that the file is able to be correctly processed; the aim of this check is to detect as early as possible common problems in the file. If some errors are found, the file will be deleted from the server.

Ancillary files

If ancillary files are needed for the processing two things should be done:

1. The ancillary file names should be defined in the submitted measurement file. In contrast to the measurement files, the file name of the ancillary files should follow a specified format, described in The SCC input netCDF file format. In brief, a sounding file should have a name rs_YYYYMMDDccNN.nc, a overlap file a name ov_YYYYMMDDccNN.nc, and a lidar ratio file a name lr_YYYYMMDDccNN.nc, where YYYY is a year, MM is the month, DD is the day, cc is the EARLINET station call sign and NN is a number.
2. The ancillary files should be uploaded on the server. The uploading can be done in the same form as before, or independently in the “Upload ancillary” form. If an ancillary file required by a measurement is not present in the database, this file will appear as “Missing” and the processing of the measurement will not start and will be marked as “pending”. When the missing file is uploaded, its status will change to “OK” and the measurement will be processed.

Monitor processing progress

When uploading finishes successfully, you will be transfered to the measurement status page. There you can monitor the progress of the processing through the status symbols. The three symbols correspond to the phase of processing:

Uploading

It indicates if everything needed for processing is provided; practically this indicates if the measurement file and all needed ancillary files have been uploaded.

Preprocessing

This indicates the progress of the preprocessor module.

Optical processing

This indicates if progress of the optical processing module (ELDA).

These three indicators currently support four states:

Gray (Not started)

A gray indicator means that the process did not start yet.

Orange (In progress)

An orange indicator means that the process is currently performed.

Green (Success)

A green indicator means that the process finished successfully.

Red (Fail)

A red indicator means that the process failed.

While the processing is in progress the page will automatically refresh every 10 seconds.

View output

When the processing of your measurement finishes successfully, you can browse and download the results.

To browse the results, you must select the “Output” tab at the bottom part of the page. There you will find links to view graphs of the resulting optical products.

To download the results, you can use the links on the “File actions” submenu. Depending on the state of the processing you can downlaod preprocessed singals, optical products, and optical product plots.