LIDAR signal filtering

This page documents SignalCreation.Lidar.Lidar.filtering(), and details the available filter kernels, the ways to set the number of coefficients over altitude, and the NDACC-compliant resolution methodologies implemented in SignalCreation.Lidar.Filter.Filter.

Overview

• Selects input variables by prefix (default: preprocess) and by Signal_Type (default: [100,119]).

• Builds a Filter object with the requested filter name.

• Computes the number of coefficients per altitude via one of several strategies (fixed, DIAL polynomial, linear).

• Applies the filter to value and uncertainty (optional PR2 and/or log-space transform).

• Computes resolution using NDACC transfer function (DF) and/or impulse response (IR) methods, and also the originator method.

• Saves filtered data, uncertainties, number-of-points, and resolution arrays to the dataset.

Function signature

def filtering(
    self,
    list_data_name: list | None = None,
    error_suffix: None | str = None,
    input_prefix: str | None = None,
    prefix_output: str | None = None,
    filter_output: str | None = None,
    resolution_output_origin: str | None = None,
    resolution_output_ndacc_df: str | None = None,
    resolution_output_ndacc_ir: str | None = None,
    resolution_methodology: str | list | None = None,
    list_signal_type: list | np.ndarray | None = None,
    name_param_tag: str = "Filtering",
    use_pr2: bool = False,
    use_log: bool = False,
) -> None

Configuring the filter

### Available filter names

These map to static methods in Filter and determine the kernel shape.

Filter name (string)	Type	Notes
"Savitzky-Golay"	Low-pass	savgol_coeffs(window, poly=2, deriv=0)
"Savitzky-Golay-deriv"	Derivative	Antisymmetric coefficients (1st derivative)
"LSQ1"	Low-pass	Flat window / moving average
"Hann-windows"	Low-pass	Hann window (normalized)
"Hamming-windows"	Low-pass	Hamming window (normalized)
"Blackman-windows"	Low-pass	Blackman window (normalized)
"Ideal-trunc"	Low-pass (needs fcut)	Ideal sinc truncation with cutoff fcut
"Hann"	Low-pass (needs fcut)	Windowed-sinc, Hann
"Hamming"	Low-pass (needs fcut)	Windowed-sinc, Hamming
"Blackman"	Low-pass (needs fcut)	Windowed-sinc, Blackman
"Binomial"	Low-pass	Normalized binomial coefficients

Notes - Type: filters are detected as low-pass (symmetric kernel) or derivative (antisymmetric kernel). - Filters marked needs fcut accept a cutoff parameter fcut via the parameter dict used when computing coefficients. - Some implementations accept a generic atten key, but the current kernels use mainly fcut. - For derivative filters, an extra division by \(\Delta z\) (altitude sampling) is applied after convolution to yield the correct derivative units.

Setting the number of coefficients over altitude

The number of coefficients \(N(z)\) can vary with altitude. The following strategies are available via Filter.calculate_nb_coef_over_altitude(method, parameters)():

1. Fixed number of points ("fixed number points" / constant window)

   • Parameters: value (int, default 50).

   • Resulting window length is forced to odd internally (for symmetry).

2. DIAL polynomial ("dial poly")

   • Parameters: ican1, ican2, ncan1, ncan2, ncanmin, ncanmax (defaults: 400, 1200, 50, 400, 1, 200).

   • Window evolves quadratically between indices ican1 and ican2, bounded by ncanmin/ncanmax.

   • Final length is odd: \(N(z)=2\cdot \lfloor \mathrm{something}\rfloor + 1\).

3. Linear ("linear")

   • Parameters: alt_min, alt_max, n_min, n_max (defaults: 0 m, 50 km, 1, 400).

   • Linearly interpolates the window length between altitude bounds, then rounds to the nearest odd integer.

After computing \(N(z)\), Filter.calculate_coefficients() pre-computes unique kernels used during the convolution.

Applying the filter

• Inputs: value and uncertainty are fetched with tf.xr_get_data_and_unc().

• Optional transforms: - use_pr2=True → apply to \(P r^2\) (and its uncertainty). - use_log=True → apply to \(\log(\cdot)\); uncertainty handled via tf.xr_log().

• Convolution: altitude-dependent kernel is applied per-altitude; if any NaN exists in the window, the corresponding output altitude is set to NaN.

• Units: if the input has pint units, they are preserved; derivative filters divide by \(\Delta z\) automatically.

Resolution methodologies

Three resolution definitions are provided and may be saved:

1. NDACC — transfer function (DF) Uses transfer_function(). The magnitude of the transfer function \(H(f)\) is computed from the kernel, the cutoff frequency \(f_c\) is defined by \(H(f_c)=0.5\), and the resolution is

   \[\mathrm{Resolution}_{\mathrm{DF}} = \frac{1}{2 f_c} \times \Delta z\]

   where \(\Delta z\) is the vertical sampling (pint-aware). The method also supports returning the full transfer function matrix for diagnostics. Files can be cached to NetCDF when enabled.

2. NDACC — impulse response (IR) Uses impulse_response(). The impulse response \(h\) is normalized to 1 at peak, and the resolution is computed from the FWHM via linear interpolation around 0.5; equivalently

   \[\mathrm{Resolution}_{\mathrm{IR}} = \mathrm{FWHM}(h) \times \Delta z.\]

3. Originator method Filter.get_resol_init_matlab_software() implements the legacy (MATLAB) formula:

   \[\mathrm{Resolution}_{\mathrm{orig}} = \frac{\Delta z}{\left(\frac{N/2}{a}\right)^{1/b}},\quad a=0.664,\; b=-1.046,\]

   where \(N\) is the odd window length at each altitude.

Outputs and shapes - output="resolution" → 1D profile over altitude with units of altitude. - output="matrix" → 2D matrix:

• DF: dims ['altitude','frequency'] with frequency coordinate (scaled by \(\Delta z\)).

• IR: dims ['altitude','nf'] with the impulse-response vector.

Caching and performance - If use_ndacc_resol_file=True, DF/IR results can be read/written to NetCDF files (filenames configurable via filenames_ndacc_resol = {'df': ..., 'ir': ...}) to speed up repeated runs. - Multiprocessing is used for DF calculations when enabled (use_multiprocessing=True).

Processing steps (updated)

1. Discover inputs (by prefix and Signal_Type) if list_data_name not provided.

2. Instantiate Filter(filter_name) from XML Filtering/filter_name (default: "Savitzky-Golay").

3. Compute N(z) via Filter.calculate_nb_coef_over_altitude() using XML Filtering/mode_number_points and its child parameters; write back final parameters to XML.

4. Apply filter to value and uncertainty (with PR2/log options if requested).

5. Compute resolution according to Filtering/resolution_methodology (one or more of df, ir, origin) and save each under the dedicated prefixes (resolution_ndacc_df, resolution_ndacc_ir, resolution_origin).

6. Write outputs: filtered signal under prefix_output, number-of-points under filter_output, and attach attributes including Signal_Type, ancillary_variables, and Resolution list.

Example (XML)

<Filtering>
  <filter_name>Savitzky-Golay</filter_name>
  <mode_number_points>dial poly</mode_number_points>
  <mode_number_points>
    <ican1>400</ican1>
    <ican2>1200</ican2>
    <ncan1>50</ncan1>
    <ncan2>400</ncan2>
    <ncanmin>1</ncanmin>
    <ncanmax>200</ncanmax>
  </mode_number_points>
  <resolution_methodology options="df,ir,origin" multiple_options="True">df,ir</resolution_methodology>
</Filtering>

See also

• Importing LIDAR raw data

• Sky background subtraction

• Saturation / desaturation correction

• Geometrical factor loading

• Variable naming and metadata configuration