Flagging and spectral filtering
===============================

Flagging and filtering address different problems. Flagging identifies
unreliable frequency channels or spatial modes and removes them or gives them
zero weight. Spectral filtering subtracts a deliberately chosen range of
frequency-smooth structure from otherwise retained samples.

Flagger pipelines
-----------------

:class:`~ps_eor.flagger.FlaggerRunner` applies an ordered set of tests to a
data cube and a matched noise proxy, commonly Stokes I and Stokes V:

.. code-block:: python

   from ps_eor.flagger import FlaggerRunner

   runner = FlaggerRunner.load("flagger.ini")
   flagged_i, flagged_v = runner.run(i_cube.copy(), v_cube.copy())

   flagged_noise = runner.apply_last(noise_cube)
   runner.plot()

The runner derives every selection from the paired cubes and applies it to
both, keeping their sampling aligned. ``apply_last`` replays the combined
selection on another cube with the original geometry.

Available flaggers
~~~~~~~~~~~~~~~~~~

The ``type`` used in each INI section must match one of these concrete
flagger classes. ``UVDirectionFlagger``, ``UVWeightsFlagger``,
``UVSigmaClipFlagger``, and ``FreqsWeightsFlagger`` operate on
:class:`~ps_eor.datacube.CartDataCube`; ``LMThetaMaxFlagger`` operates on
:class:`~ps_eor.sphcube.SphDataCube`. The frequency sigma clipper supports
both geometries when clipping variance, while its SEFD statistic requires a
Cartesian cube.

.. list-table::
   :header-rows: 1
   :widths: 24 12 34 30

   * - Type
     - Selection
     - What it flags
     - Main settings
   * - :class:`~ps_eor.flagger.FixedFreqsFlagger`
     - Frequencies
     - Explicit channels or MHz ranges. A variance test can optionally retain
       listed ranges that do not look anomalous.
     - ``freqs``, ``ratio_min_v_var``,
       ``ratio_min_v_var_poly_sub``
   * - :class:`~ps_eor.flagger.FreqsSigmaClipFlagger`
     - Frequencies
     - Channels with anomalously high SEFD or variance across spatial modes,
       measured from Stokes I or the noise-proxy cube.
     - ``stokes``, ``sefd``, ``nsigma``,
       ``detrend_poly_deg``, ``detrend_nsigma_clip``
   * - :class:`~ps_eor.flagger.FreqsWeightsFlagger`
     - Frequencies
     - Channels whose median effective weight falls below a fitted frequency
       trend.
     - ``ratio``, ``trend_poly_deg``
   * - :class:`~ps_eor.flagger.UVDirectionFlagger`
     - UV modes
     - UV cells along a specified angular direction, for removing line-like
       gridding artefacts.
     - ``direction_deg``, ``extend``
   * - :class:`~ps_eor.flagger.UVWeightsFlagger`
     - UV modes
     - UV samples whose minimum weight across frequency is too small relative
       to the best-sampled mode.
     - ``threshold``
   * - :class:`~ps_eor.flagger.UVSigmaClipFlagger`
     - UV modes
     - UV samples with anomalously high SEFD or variance after detrending
       against baseline length. It can use I, V, dI, or dV.
     - ``stokes``, ``sefd``, ``nsigma``,
       ``detrend_poly_deg``, ``detrend_nsigma_clip``
   * - :class:`~ps_eor.flagger.LMThetaMaxFlagger`
     - Spherical modes
     - Spherical-harmonic modes beyond a variance-derived angular boundary.
     - ``th_in``, ``relative_threshold``

All types also accept ``action`` and a display ``name``. When loading an INI
file, the section name becomes the display name and the ``[flagger]`` section
sets the execution order:

.. code-block:: ini

   [flagger]
   pipeline = bad_channels, low_weights, noisy_uv

   [bad_channels]
   type = FixedFreqsFlagger
   freqs = 115.0-115.2, 121.5
   action = filter

   [low_weights]
   type = FreqsWeightsFlagger
   ratio = 0.6
   action = zero_weight

   [noisy_uv]
   type = UVSigmaClipFlagger
   stokes = dV
   sefd = true
   nsigma = 5
   action = zero_weight

The order matters because each flagger sees the output of the preceding
selection.

Each flagger uses one of two actions:

``filter``
   Remove selected channels or spatial modes. This makes subsequent arrays
   smaller and requires the same selection to be applied to every related
   cube.

``zero_weight``
   Keep the original grid but set the corresponding weights to zero. This is
   often convenient when a regular grid must be retained.

Flaggers and runners can modify their inputs. Work on copies unless the
selection is intended to become the new canonical data product.

High-pass filtering
-------------------

The FIR filters in :mod:`ps_eor.filtering` return a new cube. A fixed
frequency scale can be applied to every baseline:

.. code-block:: python

   from ps_eor.filtering import high_pass_cube

   filtered = high_pass_cube(cube, scale_channels=20)

For visibility data, a baseline-dependent horizon scale is often more useful:

.. code-block:: python

   from ps_eor.filtering import high_pass_horizon

   filtered = high_pass_horizon(cube, geometry_factor=0.7)

``geometry_factor`` sets the projected fraction of the horizon delay. The
filter scale also depends on channel width and baseline length. Filtering is
therefore a model choice with a signal-transfer function, not a replacement
for identifying invalid samples. Its signal-transfer function can be measured
with injected signals.