Lidar class#

Overview#

The entire processing pipeline is driven by the Lidar class. It encapsulates:

Configuration (XML parameters, date to process),
Data containers (raw LIDAR, auxiliary atmospherics, cross-sections, radiosonde, etc.),
Processing methods (I/O, background, desaturation, geometrical factor, filtering, DIAL ozone, aerosol retrieval),
Outputs stored with NetCDF conventions (using xarray.Dataset).

Constructor#

from datetime import date
from lidar_librairy import Lidar, ParametersFromXml

# Minimal usage with a parameters XML path
lidar = Lidar(parameter_file="Configuration_files/parameters.xml", dt="2023-03-28")

# or with a ParametersFromXml instance
params = ParametersFromXml("Configuration_files/parameters.xml")
lidar = Lidar(parameter_file=params, dt=date(2023, 3, 28))

Parameters#

parameter_file (str | ParametersFromXml | None) Path to the XML file containing processing parameters, or a pre-built ParametersFromXml instance. If None, default parameters may be used by downstream code (depending on your implementation).
dt (str | datetime.date | None) Optional date to process. If provided as a string, it must be ISO formatted (YYYY-MM-DD). When set, the value is also written back into the XML at the path Read/date_to_process/date with attributes type="date" and format_date="iso".

Storing the date in XML#

If the date is provided, it is saved in the XML under:

<Dial>
  <Read>
    <date_to_process>
      <date type="date" format_date="iso">2023-03-28</date>
    </date_to_process>
  </Read>
</Dial>

Note

ISO format is YYYY-MM-DD (e.g., 2023-03-28).

Attributes#

Below are the most relevant attributes exposed by the class (internal/private names included for clarity):

_param (ParametersFromXml) XML parameter manager (read/write access to parameters and metadata).
_parameter_file (str | None) Path to the parameter file (may be None if constructed from a ParametersFromXml instance).
output (xarray.Dataset) Main container for processed outputs (profiles, uncertainties, flags, metadata).
_raw_lidar (implementation-defined) Raw LIDAR data container (as read from input files).
_lidar (implementation-defined) Working LIDAR data container (pre-processed / filtered).
_atmosphere (implementation-defined) Atmospheric auxiliary data (pressure, temperature, density, etc.).
_atmospheric_component_atm (implementation-defined) Additional atmospheric components (e.g., for plotting/diagnostics).
_cross_section_ozone (implementation-defined) Ozone absorption cross-section data.
_cross_section (dict) Dictionary of other cross-sections.
_radiosonde (implementation-defined) Radiosonde profiles if available.
_climato (dict) Climatological datasets (e.g., ozone climatology).
_geometrical_factor (implementation-defined) Overlap/geometry factor used to correct near-field response.
_ureg (unit registry) Unit handling object (pint or equivalent).
_simulated_lidar_raw_signal (implementation-defined) Theoretical/simulated raw LIDAR signal for validation and figures.

Quick example#

Creating an instance from the XML parameter file and an ISO date:

#####################
# 0. Parameter File #
#####################
lidar = Lidar(parameter_file="Configuration_files/parameters.xml", dt="2023-03-28")

Typical workflow (high level)#

Import inputs Read raw LIDAR files, auxiliary atmospheric fields, ozone cross-sections, and climatology.
Simulation (optional) Generate a theoretical LIDAR signal for comparison/validation.
Pre-processing Background removal, detector desaturation, geometrical factor correction.
Merging and filtering Channel gluing (high/low range), altitude-adaptive filtering for derivative stability.
Retrievals Aerosol backscatter (Klett), ozone via DIAL (Rayleigh/Raman where applicable).
Export NetCDF/CSV outputs with uncertainties, effective vertical resolution, and QC flags.

Design notes#

Tip

Separate instrument-specific parameters (pre-processing) from product-specific parameters (retrieval), so the same retrieval settings can be reused across instruments.

Warning

Derivative-based steps (e.g., DIAL) are sensitive to noise. Always report the effective vertical resolution and validate filter settings against known structures (e.g., tropopause).

API reference (autodoc)#

If you use Sphinx autodoc, expose the full API directly from the source code:

class SignalCreation.Lidar.Lidar(parameter_file=None, dt=None)#

Bases: object

Main library for processing ozone lidar data for ACTRIS.

Parameters:

parameter_file (str or None) – Path to the XML file containing processing parameters. If None, default parameters are used.
dt (str | date | None)

- `_param`: Instance of ParametersFromXml class for handling parameters from the XML file.

- `_parameter_file`: Path to the parameter file.

- `output`: xarray Dataset for storing processed data.

- `_raw_lidar`: Placeholder for raw lidar data.

- `_atmosphere`: Placeholder for atmospheric data.

- `_cross_section_ozone`: Placeholder for ozone cross-section data.

- `_radiosonde`: Placeholder for radiosonde data.

- `_lidar`: Placeholder for lidar data.

- `_atmospheric_component_atm`: Placeholder for atmospheric components.

- `_climato`: Dictionary for storing climatological data.

- `_cross_section`: Dictionary for storing cross-section data.

- `_geometrical_factor`: Placeholder for geometrical factor data.

- `_ureg`: Instance of the unit registry for handling units.

- `_simulated_lidar_raw_signal`: Placeholder for simulated lidar raw signal data.

Constructor

__init__(parameter_file=None, dt=None)#

Initialize the OzoneLidarProcessor.

Parameters:

parameter_file (str or None) – Path to the XML file containing processing parameters. If None, default parameters are used.
dt (str | date | None)

Reading and inputs

import_lidar_data(lidar_files=None, channel_to_remove=None, lidar_format=None, force_photocounting_units=None, xml_tag='Read/Lidar_files/')#

Import raw lidar data from file(s) and insert into _lidar attribute of the class.

Parameters:

lidar_files (str or None) – list of file to read (default: None)
channel_to_remove (int or list or None) – channels to remove (default: None)
lidar_format (str or None) – format of input data (default: None)
force_photocounting_units (str or None) – force the unit for photocounting (if MHz)
xml_tag (str) – tag into the parameter file, if needed (default: “Read/Lidar_files/”)

import_atmospheric_component(main_parameters_tag='Read/Atmosphere_file')#

Read temperature, pressure and density of the components of the atmosphere :param main_parameters_tag: tag of parameters in the parameter file (default: “Read/Atmosphere_file”) :type main_parameters_tag: str

Parameters:: main_parameters_tag (str)

import_other_atmospheric_component(atm_cpt='all')#

Import into the dataset other atmospheric components (like ozone)

Parameters:: atm_cpt (str or list[str]) – composant to import, or list of composant

Pre-processing

background(list_data_name=None, error_suffix=None, prefix_output_name=None, raw_lidar_data_prefix=None)#

Parameters:

list_data_name (None | list)
error_suffix (None | str)
prefix_output_name (None | str)
raw_lidar_data_prefix (str | None)

saturation(list_data_name=None, error_suffix=None, prefix_output_name=None, list_signal_type=None)#

Apply saturation to lidar signa

Parameters:

list_data_name (None | list) – list raw data name to process
error_suffix (None | str) – suffix of the uncertainty name (default: _Unc)
prefix_output_name (None | str) – prefix of the output name
list_signal_type (list | ndarray | None) – list signal type to process, if no list of data name (default: np.arange(100, 120))

geometrical_factor(list_data_name=None, error_suffix=None, prefix_output_name=None)#

Parameters:

list_data_name (None | list)
error_suffix (None | str)
prefix_output_name (str | None)

apply_preprocess(raw_lidar_data_prefix=None, background_prefix=None, saturation_prefix=None, geometrical_prefix=None, preprocess_prefix=None, error_suffix=None, list_signal_type=None)#

Apply pre-process for lidar data : background, saturation and geometrical factor

Parameters:

raw_lidar_data_prefix (str | None) – prefix of raw lidar data name to use
background_prefix (str | None) – prefix of background data name to use
saturation_prefix (str | None) – prefix of saturation data name to use
geometrical_prefix (str | None) – prefix of geometrical factor data name to use
preprocess_prefix (str | None) – prefix of the output preprocess data name
error_suffix (None | str) – suffix of the error
list_signal_type (list | ndarray | None) – list of data type to use if necessary (default: np.arange(100, 120))

Retrievals

aerosols_process()#

filtering(list_data_name=None, error_suffix=None, input_prefix=None, prefix_output=None, filter_output=None, resolution_output_origin=None, resolution_output_ndacc_df=None, resolution_output_ndacc_ir=None, resolution_methodology=None, list_signal_type=None, name_param_tag='Filtering', use_pr2=False, use_log=False)#

Apply filtering to lidar data

Parameters:

list_data_name (list | None) – list of raw data name to use
error_suffix (None | str) – suffix of the uncertainty (default: _Unc)
input_prefix (str | None) – prefix of input data name
prefix_output (str | None) – prefix of output data
filter_output (str | None) – prefix of the filtering data
resolution_output_origin (str | None) – name of the origin resolution output name, originator method
resolution_output_ndacc_df (str | None) – name of the NDACC resolution output name, DF method
resolution_output_ndacc_ir (str | None) – name of the NDACC resolution output name, IR method
resolution_methodology (str | list | None) – name of the methodology of calculation of resolution
list_signal_type (list | ndarray | None) – list of data type to use if necessary (default: np.arange(100, 120))
name_param_tag (str) – name of the tag into the parameter file
use_pr2 (bool) – apply the filter into the PR2 signal (default: False)
use_log (bool) – apply the filter into the signal or PR2 signal is use_pr2 is True (default: False)

glue(list_signal_type_1=None, list_signal_type_2=None, name_param_tag='MergingOzone', method=None, shift=None, alt_min=None, alt_max=None, error_suffix=None, input_prefix=None, prefix_output=None, math_operation='addition')#

Merge (or “glue”) two LIDAR signals over a defined altitude range.

This function takes two lists of signal types (typically corresponding to two different LIDAR channels) and merges them into a single signal over a specified altitude window, using a chosen interpolation method and an optional vertical shift. The merged signal and its associated uncertainty are inserted into the LIDAR dataset, along with resolution arrays.

Parameters:

list_signal_type_1 (list[str], optional) – List of signal types from channel 1. If None, will be read from XML configuration using name_param_tag/channel_1.
list_signal_type_2 (list[str], optional) – List of signal types from channel 2. If None, will be read from XML configuration using name_param_tag/channel_2.
name_param_tag (str, default="MergingOzone") – Name of the configuration tag used to read/write parameters from/to the XML configuration.
method (str or list[str], optional) – Interpolation method(s) used for merging (e.g., ‘linear’, ‘log’). If None, defaults to ‘linear’.
shift (int or list[int], optional) – Vertical shift(s) applied to align the two signals before merging. Defaults to 0 if not provided.
alt_min (pint.Quantity or str or list, optional) – Minimum altitude(s) for the merging range. If None, defaults to 23000 meters.
alt_max (pint.Quantity or str or list, optional) – Maximum altitude(s) for the merging range. If None, defaults to 24000 meters.
error_suffix (str, optional) – Suffix used to identify uncertainty variables in the dataset. Defaults to “_Unc” if not provided.
input_prefix (str, optional) – Prefix used to locate input signal variables in the dataset. If None, will be inferred based on name_param_tag.
prefix_output (str, optional) – Prefix for naming output merged variables. Defaults to the same as input_prefix.
math_operation (str)

Raises:

RuntimeError – If signal variables are missing from the dataset, or if the signal type lists have mismatched lengths.
Side Effects –
------------ –
- Parameters may be read from or written to the internal XML configuration object (self._param). –
- New merged signals, uncertainty arrays, and resolution arrays are added to the LIDAR dataset (self._lidar). –

Notes

This function handles both single and multiple signal pairs.
Unit consistency (via pint) is enforced for altitude-related parameters.
Merging is done through an external merge() function which applies the actual algorithm.

ozone_process(list_data_name=None, error_suffix=None, input_prefix=None, prefix_output=None, binomial_filter_number_of_point=None, rayleigh_cross_section_method=None, target_molecule=None, ozone_cross_section_name=None, atm_cross_section_name=None, density_name=None, aerosol_extinction_name=None, aerosol_diffusion_name=None, atm_diffusion_name=None, angstrom_coef_alpha=None, angstrom_coef_beta=None, replace_backscatter_ratio_less_than_1=None, apply_aerosol_correction=None)#

Parameters:

list_data_name (None | list)
error_suffix (None | str)
input_prefix (str | None)
prefix_output (str | None)
binomial_filter_number_of_point (str | int | None)
rayleigh_cross_section_method (str | None)
target_molecule (str | None)
ozone_cross_section_name (str | None)
atm_cross_section_name (str | None)
density_name (str | None)
aerosol_extinction_name (str | None)
aerosol_diffusion_name (str | None)
atm_diffusion_name (str | None)
angstrom_coef_alpha (ndarray | float | int | None)
angstrom_coef_beta (ndarray | float | int | None)
replace_backscatter_ratio_less_than_1 (bool | None)
apply_aerosol_correction (bool)

add_data(new_data, replace=None)#

Parameters:

new_data (Dataset | DataArray)
replace (bool | None)

aerosols_process()#

Apply pre-process for lidar data : background, saturation and geometrical factor

Parameters:

raw_lidar_data_prefix (str | None) – prefix of raw lidar data name to use
background_prefix (str | None) – prefix of background data name to use
saturation_prefix (str | None) – prefix of saturation data name to use
geometrical_prefix (str | None) – prefix of geometrical factor data name to use
preprocess_prefix (str | None) – prefix of the output preprocess data name
error_suffix (None | str) – suffix of the error
list_signal_type (list | ndarray | None) – list of data type to use if necessary (default: np.arange(100, 120))

property atmosphere#

background(list_data_name=None, error_suffix=None, prefix_output_name=None, raw_lidar_data_prefix=None)#

Parameters:

list_data_name (None | list)
error_suffix (None | str)
prefix_output_name (None | str)
raw_lidar_data_prefix (str | None)

static convert_MHz_to_nb_photon(track_MHz, label_nb_shot='nb_shot_laser', label_resolution='bin_width')#

Return a copy of the track converted from MHz to photon counting

Parameters:

track_MHz – Signal channel in MHz
nb_shot – Number of shot (given in lidar file)
resolution – Signal resolution (given in lidar file)
label_nb_shot (str)
label_resolution (str)

Returns:

Copy of signal converted to photon counting

static convert_nb_photon_to_MHz(track_nb_photon, label_nb_shot='nb_shot_laser', label_resolution='bin_width')#

Return a copy of the track converted from photon counting to MHz

Parameters:

track_nb_photon (DataArray) – Photon counting signal channel (no unity)
nb_shot – Number of shot (given in lidar file)
resolution – Signal resolution (given in lidar file)
label_nb_shot (str)
label_resolution (str)

Returns:

Copy of Signal converted to MHz

create_simulated_lidar()#

drop_variables_from_dataset(vars_to_drop=None, regex=None)#

Drop variables from an xarray.Dataset using a list of names or a regex pattern.

Parameters:

ds – Input xarray.Dataset
vars_to_drop (str | list | None) – List of variable names to drop (optional)
regex (str | None) – Regex pattern to match variable names to drop (optional)

Returns:

A new xarray.Dataset with selected variables removed

export_ascii(tag_filename_ascii='ExportData/AerisFile/filename', tag_format_ascii='ExportData/AerisFile/format')#

Parameters:

tag_filename_ascii (str)
tag_format_ascii (str)

export_data(default_write_netcdf=True, default_write_aeris=False, default_write_ascii=False, default_write_figures=False)#

Export lidar data in different formats.

Parameters:

default_write_netcdf (bool) – Flag of creation of NetCDF file (default: True)
default_write_aeris (bool) – Flag of creation of Ascii file (default: True)
default_write_ascii (bool) – Flag of creation of Ascii file (default: True)
default_write_figures (bool) – Flag of creation of figure files (default: True)

export_data_figures()#

export_netcdf(output_filename_nc=None, output_filename_xml=None)#

Export data to netcdf file

Parameters:

output_filename_nc (str or None) – filename of the NetCDF file (default: None)
output_filename_xml (str | None)

Apply filtering to lidar data

Parameters:

list_data_name (list | None) – list of raw data name to use
error_suffix (None | str) – suffix of the uncertainty (default: _Unc)
input_prefix (str | None) – prefix of input data name
prefix_output (str | None) – prefix of output data
filter_output (str | None) – prefix of the filtering data
resolution_output_origin (str | None) – name of the origin resolution output name, originator method
resolution_output_ndacc_df (str | None) – name of the NDACC resolution output name, DF method
resolution_output_ndacc_ir (str | None) – name of the NDACC resolution output name, IR method
resolution_methodology (str | list | None) – name of the methodology of calculation of resolution
list_signal_type (list | ndarray | None) – list of data type to use if necessary (default: np.arange(100, 120))
name_param_tag (str) – name of the tag into the parameter file
use_pr2 (bool) – apply the filter into the PR2 signal (default: False)
use_log (bool) – apply the filter into the signal or PR2 signal is use_pr2 is True (default: False)

geometrical_factor(list_data_name=None, error_suffix=None, prefix_output_name=None)#

Parameters:

list_data_name (None | list)
error_suffix (None | str)
prefix_output_name (str | None)

Merge (or “glue”) two LIDAR signals over a defined altitude range.

Parameters:

list_signal_type_1 (list[str], optional) – List of signal types from channel 1. If None, will be read from XML configuration using name_param_tag/channel_1.
list_signal_type_2 (list[str], optional) – List of signal types from channel 2. If None, will be read from XML configuration using name_param_tag/channel_2.
name_param_tag (str, default="MergingOzone") – Name of the configuration tag used to read/write parameters from/to the XML configuration.
method (str or list[str], optional) – Interpolation method(s) used for merging (e.g., ‘linear’, ‘log’). If None, defaults to ‘linear’.
shift (int or list[int], optional) – Vertical shift(s) applied to align the two signals before merging. Defaults to 0 if not provided.
alt_min (pint.Quantity or str or list, optional) – Minimum altitude(s) for the merging range. If None, defaults to 23000 meters.
alt_max (pint.Quantity or str or list, optional) – Maximum altitude(s) for the merging range. If None, defaults to 24000 meters.
error_suffix (str, optional) – Suffix used to identify uncertainty variables in the dataset. Defaults to “_Unc” if not provided.
input_prefix (str, optional) – Prefix used to locate input signal variables in the dataset. If None, will be inferred based on name_param_tag.
prefix_output (str, optional) – Prefix for naming output merged variables. Defaults to the same as input_prefix.
math_operation (str)

Raises:

RuntimeError – If signal variables are missing from the dataset, or if the signal type lists have mismatched lengths.
Side Effects –
------------ –
- Parameters may be read from or written to the internal XML configuration object (self._param). –
- New merged signals, uncertainty arrays, and resolution arrays are added to the LIDAR dataset (self._lidar). –

Notes

This function handles both single and multiple signal pairs.
Unit consistency (via pint) is enforced for altitude-related parameters.
Merging is done through an external merge() function which applies the actual algorithm.

import_atmospheric_component(main_parameters_tag='Read/Atmosphere_file')#

Parameters:: main_parameters_tag (str)

import_lidar_data(lidar_files=None, channel_to_remove=None, lidar_format=None, force_photocounting_units=None, xml_tag='Read/Lidar_files/')#

Import raw lidar data from file(s) and insert into _lidar attribute of the class.

Parameters:

lidar_files (str or None) – list of file to read (default: None)
channel_to_remove (int or list or None) – channels to remove (default: None)
lidar_format (str or None) – format of input data (default: None)
force_photocounting_units (str or None) – force the unit for photocounting (if MHz)
xml_tag (str) – tag into the parameter file, if needed (default: “Read/Lidar_files/”)

import_other_atmospheric_component(atm_cpt='all')#

Import into the dataset other atmospheric components (like ozone)

Parameters:: atm_cpt (str or list[str]) – composant to import, or list of composant

import_parameters(parameter_file)#

Read parameter file and put the data into the geoscene class.

Parameters:: parameter_file (str) – String, pathname of the parameter file

insert_new_parameters(data, variables_to_import=None, variables_to_import_new_name=None, tag_param_variable_to_import='import_reprocess_data/insert_new_parameters/variables_to_import', tag_param_variables_to_import_new_name='import_reprocess_data/insert_new_parameters/variables_to_import_new_name', case_same_name=None)#

Import data from a reprocess

Parameters:

data – data of the reprocessing (GeoScene)
variables_to_import (list[str] | str | None) – list of variable to add to the lidar dataset
variables_to_import_new_name (list[str] | str | None) – list of new name of variables to add
tag_param_variable_to_import (str) – name of the parameter tag for variables to import
case_same_name (str | None) – choice to do in the case of name already exists (change_name, overwrite, raise_error)
tag_param_variables_to_import_new_name (str)

property lidar: Dataset | None#

property lidar_set#

lidar_track(track_name)#

new_parameter_file(parameter_tag_import_reprocess_data=None, parameter_tag_new_parameter=None, parameter_tag_new_parameter_tag=None, parameter_tag_new_parameter_value=None)#

Change the parameter attribute of the class and return it

return: a ParametersFromXml object

Parameters:

parameter_tag_import_reprocess_data (str | None)
parameter_tag_new_parameter (str | None)
parameter_tag_new_parameter_tag (str | None)
parameter_tag_new_parameter_value (str | None)

Return type:

ParametersFromXml | None

property ozone_cross_section#

Parameters:

list_data_name (None | list)
error_suffix (None | str)
input_prefix (str | None)
prefix_output (str | None)
binomial_filter_number_of_point (str | int | None)
rayleigh_cross_section_method (str | None)
target_molecule (str | None)
ozone_cross_section_name (str | None)
atm_cross_section_name (str | None)
density_name (str | None)
aerosol_extinction_name (str | None)
aerosol_diffusion_name (str | None)
atm_diffusion_name (str | None)
angstrom_coef_alpha (ndarray | float | int | None)
angstrom_coef_beta (ndarray | float | int | None)
replace_backscatter_ratio_less_than_1 (bool | None)
apply_aerosol_correction (bool)

partial_o3_column(signal_type=None)#

Calculation of partial ozone column, in troposphere, stratosphere, and total column

Parameters:: signal_type (str | int | float | None) – signal type to use for the ozone concentration. If None, search the highest value (default: None)
Returns:

property radiosonde#

rm_channels(index_channels_to_remove=None, index_channels_to_keep=None)#

Remove channels from the dataset

Parameters:

index_channels_to_remove (int or list) – index or list of index to remove
index_channels_to_keep (list | int | None)

Returns:

saturation(list_data_name=None, error_suffix=None, prefix_output_name=None, list_signal_type=None)#

Apply saturation to lidar signa

Parameters:

list_data_name (None | list) – list raw data name to process
error_suffix (None | str) – suffix of the uncertainty name (default: _Unc)
prefix_output_name (None | str) – prefix of the output name
list_signal_type (list | ndarray | None) – list signal type to process, if no list of data name (default: np.arange(100, 120))

select_data_over_altitude(alt_min=None, alt_max=None)#

Parameters:

alt_min (float | int | Quantity | None)
alt_max (float | int | Quantity | None)

temperature(list_data_name=None, error_suffix=None, input_prefix=None, prefix_output=None)#

Parameters:

list_data_name (None | list)
error_suffix (None | str)
input_prefix (str | None)
prefix_output (str | None)

validity_domain()#

Lidar class#

Overview#

Constructor#

Parameters#

Storing the date in XML#

Attributes#

Quick example#

Typical workflow (high level)#

Design notes#

API reference (autodoc)#

See also#