Flagging and spectral filtering

Flagging and filtering address different problems. Flagging identifies unreliable frequency channels or spatial modes and removes them or gives them zero weight. Spectral filtering subtracts a deliberately chosen range of frequency-smooth structure from otherwise retained samples.

Flagger pipelines

FlaggerRunner applies an ordered set of tests to a data cube and a matched noise proxy, commonly Stokes I and Stokes V:

from ps_eor.flagger import FlaggerRunner

runner = FlaggerRunner.load("flagger.ini")
flagged_i, flagged_v = runner.run(i_cube.copy(), v_cube.copy())

flagged_noise = runner.apply_last(noise_cube)
runner.plot()

The runner derives every selection from the paired cubes and applies it to both, keeping their sampling aligned. apply_last replays the combined selection on another cube with the original geometry.

Available flaggers

The type used in each INI section must match one of these concrete flagger classes. UVDirectionFlagger, UVWeightsFlagger, UVSigmaClipFlagger, and FreqsWeightsFlagger operate on CartDataCube; LMThetaMaxFlagger operates on SphDataCube. The frequency sigma clipper supports both geometries when clipping variance, while its SEFD statistic requires a Cartesian cube.

Type	Selection	What it flags	Main settings
FixedFreqsFlagger	Frequencies	Explicit channels or MHz ranges. A variance test can optionally retain listed ranges that do not look anomalous.	freqs, ratio_min_v_var, ratio_min_v_var_poly_sub
FreqsSigmaClipFlagger	Frequencies	Channels with anomalously high SEFD or variance across spatial modes, measured from Stokes I or the noise-proxy cube.	stokes, sefd, nsigma, detrend_poly_deg, detrend_nsigma_clip
FreqsWeightsFlagger	Frequencies	Channels whose median effective weight falls below a fitted frequency trend.	ratio, trend_poly_deg
UVDirectionFlagger	UV modes	UV cells along a specified angular direction, for removing line-like gridding artefacts.	direction_deg, extend
UVWeightsFlagger	UV modes	UV samples whose minimum weight across frequency is too small relative to the best-sampled mode.	threshold
UVSigmaClipFlagger	UV modes	UV samples with anomalously high SEFD or variance after detrending against baseline length. It can use I, V, dI, or dV.	stokes, sefd, nsigma, detrend_poly_deg, detrend_nsigma_clip
LMThetaMaxFlagger	Spherical modes	Spherical-harmonic modes beyond a variance-derived angular boundary.	th_in, relative_threshold

All types also accept action and a display name. When loading an INI file, the section name becomes the display name and the [flagger] section sets the execution order:

[flagger]
pipeline = bad_channels, low_weights, noisy_uv

[bad_channels]
type = FixedFreqsFlagger
freqs = 115.0-115.2, 121.5
action = filter

[low_weights]
type = FreqsWeightsFlagger
ratio = 0.6
action = zero_weight

[noisy_uv]
type = UVSigmaClipFlagger
stokes = dV
sefd = true
nsigma = 5
action = zero_weight

The order matters because each flagger sees the output of the preceding selection.

Each flagger uses one of two actions:

filter

Remove selected channels or spatial modes. This makes subsequent arrays smaller and requires the same selection to be applied to every related cube.

zero_weight

Keep the original grid but set the corresponding weights to zero. This is often convenient when a regular grid must be retained.

Flaggers and runners can modify their inputs. Work on copies unless the selection is intended to become the new canonical data product.

High-pass filtering

The FIR filters in ps_eor.filtering return a new cube. A fixed frequency scale can be applied to every baseline:

from ps_eor.filtering import high_pass_cube

filtered = high_pass_cube(cube, scale_channels=20)

For visibility data, a baseline-dependent horizon scale is often more useful:

from ps_eor.filtering import high_pass_horizon

filtered = high_pass_horizon(cube, geometry_factor=0.7)

geometry_factor sets the projected fraction of the horizon delay. The filter scale also depends on channel width and baseline length. Filtering is therefore a model choice with a signal-transfer function, not a replacement for identifying invalid samples. Its signal-transfer function can be measured with injected signals.