LIDAR pre-processing#

This page documents SignalCreation.Lidar.Lidar.apply_preprocess(), which performs the main LIDAR signal pre-processing chain combining background subtraction, saturation correction, and geometrical factor normalization.

The resulting preprocessed signal is stored in the internal dataset (self._lidar) under a configurable prefix, together with its associated uncertainty.

Overview#

#######################
# LIDAR Preprocessing #
#######################
lidar.apply_preprocess()

What it does:

  1. Discovers all raw LIDAR channels (prefix defined by raw_lidar_data_prefix) whose Signal_Type lies in the default interval [100, 119] (uncertainty channels excluded).

  2. For each channel, retrieves: - Raw signal and its uncertainty, - Background correction, - Saturation coefficient, - Geometrical factor, all matched by Signal_Type.

  3. Computes the preprocessed signal as:

    \[P_{preproc} = (P_{raw} - B) * S / G\]

    where \(B\) = background, \(S\) = saturation coefficient, \(G\) = geometrical factor.

  4. Propagates a simplified uncertainty (quadratic combination of raw and background uncertainties).

  5. Adds the results (value + uncertainty) to the dataset under the prefix preprocess_prefix.

Method signature#

def apply_preprocess(
    self,
    raw_lidar_data_prefix: str | None = None,
    background_prefix: str | None = None,
    saturation_prefix: str | None = None,
    geometrical_prefix: str | None = None,
    preprocess_prefix: str | None = None,
    error_suffix: None | str = None,
    list_signal_type: list | np.ndarray | None = None,
) -> None:
    """Apply background, saturation, and geometrical corrections to raw LIDAR data."""

Parameters#

  • raw_lidar_data_prefix (str | None) Prefix of the raw signal variables. Defaults to nv.get_var_name('raw_lidar_data').

  • background_prefix (str | None) Prefix of the background correction variable. Defaults to nv.get_var_name('background').

  • saturation_prefix (str | None) Prefix of the saturation correction coefficient variable. Defaults to nv.get_var_name('saturation').

  • geometrical_prefix (str | None) Prefix of the geometrical factor variable. Defaults to nv.get_var_name('geometric_factor').

  • preprocess_prefix (str | None) Prefix for the output preprocessed data. Defaults to nv.get_var_name('preprocess').

  • error_suffix (str | None, default: ``”_Unc”``) Uncertainty suffix (currently validated but not explicitly used).

  • list_signal_type (list | np.ndarray | None, default: ``np.arange(100, 120)``) Restricts which Signal_Type values are processed.

Internal workflow#

  1. Channel selection Uses self._get_data_name() to find all raw LIDAR signals matching the given prefix and signal-type range.

  2. Variable matching For each channel, retrieves associated variables (background, saturation, geometrical factor) having the same Signal_Type through self._get_data_name().

  3. Preprocessing formula Combines retrieved arrays as:

    \[P_{preproc} = (P_{raw} - B) * S / G\]

    where each term is an xarray.DataArray aligned on the same altitude and time grid.

  4. Uncertainty propagation The uncertainty of the preprocessed signal is computed using standard error-propagation rules, assuming independent input variables. For

    \[P_{\text{preproc}} = \frac{(P_{\text{raw}} - B)\,S}{G},\]

    the total variance is

    \[\sigma_{P_{\text{preproc}}}^{2} = \left(\frac{S}{G}\right)^{2}\sigma_{P}^{2} + \left(\frac{S}{G}\right)^{2}\sigma_{B}^{2} + \left(\frac{P_{\text{raw}}-B}{G}\right)^{2}\sigma_{S}^{2} + \left(\frac{P_{\text{preproc}}}{G}\right)^{2}\sigma_{G}^{2}.\]

    where:

    • \(\sigma_{P}\) — uncertainty of the raw signal

    • \(\sigma_{B}\) — uncertainty of the background

    • \(\sigma_{S}\) — uncertainty of the saturation coefficient

    • \(\sigma_{G}\) — uncertainty of the geometrical factor

  5. Output insertion Each processed channel adds two variables in self._lidar: - preprocess_* (value) - preprocess_*_Unc (uncertainty)

    using nv.add_lidar_da(), with attributes copied from the input channel and Signal_Type preserved.

Outputs in _lidar#

For each processed channel:

  • preprocess_* Preprocessed signal \((P_{\\text{raw}} - B) \\times S / G\).

  • preprocess_*_Unc Associated uncertainty (simplified quadratic sum of raw and background uncertainties).

Attributes - Signal_Type copied from the source channel. - long_name may be derived from variable naming conventions.

Assumptions#

  • Background, saturation, and geometrical factor variables have been previously created with consistent dimensions and attributes.

  • All variables share the same altitude and time coordinates.

  • self._ureg (unit registry) is initialized for Pint operations.

  • tf.xr_get_data_and_unc() supports fallback defaults (zero or unity) for missing data.

Errors & logging#

  • All input parameters are type-checked via tf.check_input_param().

  • Missing correction terms are replaced by default arrays (zeros or ones).

  • Logs debug entries via logging.

Example#

Typical usage:

from SignalCreation.Lidar import Lidar

lidar = Lidar("Configuration_files/Parameters/example_station.xml", "2024-03-28")
lidar.import_lidar_data()
lidar.background()
lidar.saturation()
lidar.geometrical_factor()

# Apply full preprocessing chain
lidar.apply_preprocess()

# Inspect output variables
print([v for v in lidar._lidar.data_vars if v.startswith("preprocess")])

Notes#

  • The formula used assumes independent errors between raw signal and background.

  • You can customize prefixes (e.g., preprocess_prefix="Preproc") to avoid overwriting variables.

  • Future versions may implement full propagation of all partial uncertainties.

See also#