Conversation
Mirror the PDVD freqmask change for PDHD: - PDHD::OneChannelNoise::apply() (ProtoduneHD.cxx) now multiplies the forward FFT spectrum by the per-channel noise spectrum from chndb (m_noisedb->noise(ch)) before DC zeroing and IFFT. Empty spectrum = no-op. - pdhd/chndb-base.jsonnet accepts use_freqmask=true parameter, exposed as local freqmask_enabled for future channel_info[] entries. The matching use_freqmask TLA is threaded in from pdhd/wct-nf-sp.jsonnet (--tla-code use_freqmask=false to disable at run time). - Pre-existing U/V freqmask entries (notches at bins 169-173 / 513-516) were silently parsed but never applied. Cleared to freqmasks: [] pending re-analysis; previous values preserved in inline comments. Docs: update 05-detector-specific.md cross-detector table and add a ProtoDUNE-HD freqmask note. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…88-2195 and 2480-2485 Harmonics n=2..12 of f0=23.5 kHz (47-282 kHz), delta=1.0 kHz notches with automatic negative-freq mirrors via freqmasks_mirror. Diagnosed from run 040475 evt 0 fft_w averaged over each cluster; both clusters share the same spectral pattern. Gated on use_freqmask TLA (default on). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Real PDVD data frames are 8000 ticks after the 512->500 ns Resampler (7813 raw ticks * 512/500 -> 8000). The chndb was previously sized to nsamples=6000 (from daq.nticks), causing a size mismatch that silently blocked the new freqmask consumer in PDVD::OneChannelNoise. Override params.nf.nsamples=8000 in protodunevd/params.jsonnet so the noise spectrum (and the freqmask) match the actual FFT size. Also add a debug-level log in PDVD::OneChannelNoise::apply() that prints the channel number and number of zeroed bins whenever a non-trivial freqmask is applied (only fires with -L debug). Verified: run 040475 evt 0 anode 0 shows exactly 14 debug lines for channels 2188-2195 and 2480-2485, each zeroing 198/8000 FFT bins (11 harmonics * 2 halves * 9 bins/notch). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When OmniChannelNoiseDB is configured with an nsamples that differs from the actual frame size, PDVD/PDHD::OneChannelNoise silently skipped the freqmask (spec.size() != spectrum.size() guard). Now both emit a m_log->warn() per masked channel so the mismatch is immediately visible in the log instead of a silent no-op. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…me size OmnibusNoiseFilter now pushes the actual frame nticks into OmniChannelNoiseDB (via a new set_nsamples() method) on the first frame. If the runtime frame size differs from the jsonnet-configured nsamples, all frequency-domain spectra (freqmasks, rcrc, config, response) are rebuilt with the correct size and a single info-level log is emitted. This prevents the silent mismatch where params.nf.nsamples=6000 but the Resampler emits 8000-tick frames, causing freqmasks to be skipped by the spec.size() == spectrum.size() guard. Only OmniChannelNoiseDB supports this auto-correction; other chndb implementations are unaffected (the dynamic_cast is a no-op). The params.nf.nsamples=8000 override in PDVD params.jsonnet is kept for correct freqbinner() sizing in chndb-base.jsonnet, but is no longer load-bearing at runtime. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds {value, flo, fhi} entry form to parse_freqmasks that resolves to
bin indices at runtime using the live m_tick/m_nsamples and auto-mirrors
to the conjugate-frequency bins for real-FFT correctness. This fixes
the case where the chndb is configured with one nsamples but the runtime
frame size differs (eg PDVD with mixed 6400/8000-tick frames): the masks
now hit the intended physical frequencies on every frame size.
Adds wc.freqmasks_phys(meanfreqs, delta) jsonnet helper that emits the
new entry form with no tick/nsamples argument.
Switches PDVD chndb-base.jsonnet to the new helper. Existing
{value, lobin, hibin} entries (sbnd, pdsp, uboone, iceberg,
dune10kt-1x2x6, pcbro-50liter) keep their current behavior bit-identical.
Verified: PDVD run 040475 (8000-tick frames) zeroes 198/8000 bins per
masked W channel (same as before); run 039324 (6400-tick frames) now
zeroes 162/6400 bins (was 99/6400 with the mirror silently truncated
and the notch centers landing on wrong physical frequencies).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Updates the detector-specific examination doc to reflect:
- The new wc.freqmasks_phys helper and {value, flo, fhi} schema as the
preferred form for new freqmask entries.
- The OmniChannelNoiseDB auto-rebuild on runtime frame-size mismatch.
- The loud-skip warning in PDVD/PDHD OneChannelNoise.
- The currently-populated W-plane harmonic mask on PDVD anode 0.
- The fact that the legacy {lobin, hibin} schema is still accepted
(no auto-mirror) for back-compat with sbnd/pdsp/uboone/etc.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The U/V plane channel_info comments still recommended the legacy freqbinner(...).freqmasks_mirror(...) helper (which bakes bins at jsonnet eval time and is wrong when the runtime FFT size differs). Reworded to recommend wc.freqmasks_phys([freqs], delta), which resolves to bins at runtime and auto-mirrors conjugate bins. Also notes in params.jsonnet that PDHD's nf.nsamples (inherited from daq.nticks=6000) no longer needs to exactly match the post-Resampler frame size (today: 5999); OmnibusNoiseFilter pushes the actual size into OmniChannelNoiseDB on first frame and spectra are rebuilt. Verified: run 027409 anode 0 now logs OmniChannelNoiseDB: nsamples 6000 -> 5999, rebuilding spectra with zero mask SKIPPED warnings. No code or runtime behavior change. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The runtime-nticks auto-rebuild in OmniChannelNoiseDB and the new
{value, flo, fhi} freqmasks schema do not require config changes for
SBND or uBooNE — both have been re-verified as bit-identical under
the new code path.
SBND: single frame size (3400 ticks) so the rebuild is a no-op; the
harmonic_freqs list is already empty (all candidates commented out).
Adds a note steering future maintainers toward wc.freqmasks_phys()
when notches are re-enabled from data analysis.
uBooNE: Microboone::OneChannelNoise uses a half-complex fwd_r2c FFT,
so only bins 0..nticks/2 (~4797) are consumed. The intentional
daq.nticks=9595 vs nf.nsamples=9592 offset was re-verified safe: the
auto-rebuild widens spectra to 9595 but the notch bins (169-173,
513-516) land at the same physical frequencies (±0.03%) and the
consumer reads only the first 4798 bins regardless. Adds a comment
block in channel_info[] documenting these invariants.
Also updates sigproc/docs/examination/05-detector-specific.md with
MicroBooNE and SBND freqmask-status notes.
No code or runtime behavior change.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Same examination as for SBND and uBooNE: pdsp uses OmniChannelNoiseDB
in all NF flows so the runtime auto-rebuild dynamic_cast hits, but
pdsp has a single production frame size (daq.nticks=nf.nsamples=6000,
no Resampler upstream of NF) so the rebuild is idempotent.
The U/V plane legacy {value, lobin, hibin} freqmasks at lines 227-231
and 246-250 stay physically correct because the rebuild lands at the
same 6000-bin size. Note also that the current pdOneChannelNoise
consumer (Protodune.cxx, gated on iplane==2) reads noise(ch) only for
W-plane channels, so the U/V freqmasks here are effectively dead in
today's code path — a separate latent issue, not introduced by the
auto-rebuild.
Adds an inline comment block at the top of channel_info[] documenting
these invariants and steering future maintainers toward
wc.freqmasks_phys() when notches are reactivated. Also updates
sigproc/docs/examination/05-detector-specific.md.
No code or runtime behavior change.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Audited dune10kt-1x2x6, dune-vd, dune-vd-coldbox, dunevd-crp2, icarus,
iceberg, and pcbro-50liter against the new OmniChannelNoiseDB runtime
auto-rebuild and the {value, flo, fhi} freqmasks schema. All seven
instantiate OmniChannelNoiseDB so the dynamic_cast hits in every flow.
Findings:
- dune10kt-1x2x6 and pcbro-50liter share the same legacy U/V freqmask
pattern as pdsp ({lobin,hibin} notches at 169-173, 513-516) — dead
config because the consumer is gated on iplane==2 (W-only).
- dune-vd inherits chndbs from dune10kt-1x2x6 but its nf.jsonnet has
the active filter commented out — no live consumer.
- dune-vd-coldbox, dunevd-crp2, icarus all have empty freqmasks; their
consumers either don't read noise(ch) at all (DuneCrp, Icarus) or
have nothing to rebuild against.
- iceberg is the most interesting case: Microboone::OneChannelNoise is
the active consumer and reads noise(ch) for ALL channels (no plane
gating), and the wcls-sim-drift-simchannel-splusn-{correlated,
uncorrelated}.jsonnet variants override RUN_NTICKS — exactly the
scenario the auto-rebuild guards against. Active freqmasks are
empty so no behavior change today.
In all cases the runtime rebuild is a harmless no-op (configured
nsamples already matches runtime nticks). Added in-line comments to
each chndb-base.jsonnet documenting these invariants and recommending
wc.freqmasks_phys([freqs], delta) when notches are reactivated. Also
added a summary table to sigproc/docs/examination/05-detector-specific.md.
No code or runtime behavior change.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Commit 41e0273 commented out elec.gain and adc.resolution in cfg/pgrapher/common/params.jsonnet because larsoft injects them via fcl. Standalone wcsonnet has no fcl injection, so re-inject the historical PDSP defaults (14 mV/fC, 12-bit) in the test config to restore standalone evaluation without disturbing the fcl-driven path. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Latent jsonnet syntax error from f1f6e70 — the default channel_info entry was missing a trailing comma after the channels: std.range(...) field, so any standalone evaluation of the SBND chndb tree errored with "Expected a comma before next field". Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…e notches Two changes in chndb-base.jsonnet: 1. The W-plane harmonic freqmask block (channels 2188-2195, 2480-2485) was being emitted into channel_info for every anode, but those channels live in anode 0's W range only. When configuring any other anode, OmniChannelNoiseDB::update_channels threw "no db info for channel 2188" before NF could even start. Tighten both `channels` and `freqmasks` gates to `n == 0`. 2. Add a top-anode (n >= 4) freqmask block covering 23.5 kHz (+/-0.5 kHz) and 711 kHz (+/-2 kHz) on all U+V+W channels of each top anode. Lines diagnosed from run 040475 evt 0 anode-4 magnify FFT histograms: 23.5 kHz is the fundamental LF pickup (max ~20 ADC), 711 kHz is the 709.6/712.1 kHz RF doublet (max ~6 ADC on W). Coherent noise removal handles weaker U-plane harmonics (70.5/117.5 kHz), so those are left un-masked to keep the notch list short. Verified: ./run_nf_sp_evt.sh 040475 0 -a4 succeeds and PDVDfreqmask debug log reports 1536/1536 anode-4 channels with 34 zeroed FFT bins each (matching 23.5 + 711 kHz notches plus conjugate mirrors at nsamples=6000). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The run-027409 evt-0 APA-0 correlation audit showed that the first 3 offline channels of every FEMB block (on U and V) carry the bulk common mode of the *previous* block, not their own. Both bulk-begin and bulk-end of block k correlate equally with edge(k+1) at mean(r)≈+0.45 pre-NF, and the edge-edge correlation survives NF at r≈+0.41/+0.44 (U/V), confirming a pure grouping offset rather than physical crosstalk. Add a `coh_group_shift` parameter (default 3) to chndb-base.jsonnet that cyclically rotates the U and V group boundaries by that many offline channels. W groups are unchanged (audit confirmed W is clean). Setting coh_group_shift=0 restores the original definition exactly (verified bit-identical via jsonnet evaluation). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Three changes:
1. cfg/pdhd/chndb-resp.jsonnet: replace byte-identical SBND copy with
PDHD-specific FR⊗ER kernel generated via
wirecell-sigproc track-response -d pdhd --export-jsonnet ...
FR: dune-garfield-1d565.json.bz2, gain=14 mV/fC, postgain=1.0,
ADC/mV=11.70, 500 ns tick. New peaks: U +193/−206 ADC, V +216/−247 ADC.
2. cfg/pdhd/chndb-base.jsonnet:
- Add scale_resp(arr) = std.map(x*gain_scale, arr) so the kernel
tracks runtime params.elec.gain (extVar elecGain).
- Update response_offset: U 120→127, V 124→132 (argmin of new arrays).
- Add follow-up TODO: thresholds (decon_limit etc.) need re-tuning for
the new ~4× larger kernel scale.
3. sigproc/src/ProtoduneHD.cxx: uncomment + extend debug prints in
SignalProtection (first 5 calls: mean/rms/respec.size/res_offset and
decon-path ON/OFF) and CoherentNoiseSub::apply (once per 800 channels:
response_offset, gain_correction, respec.size).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…kit into apply-pointcloud
…ane decon
U: pad_window_front 20->10, decon_limit 0.02->0.01*gs, roi_min_max_ratio 3.0->0.8.
V: decon_limit1 0.08->0.07*gs, roi_min_max_ratio 1.5->0.8; APA 0 (n==0) sets
response={} and response_offset=0 to fall through PDHD::SignalProtection's
deconvolution gate because that V plane is hardware-faulty (collection-like).
sigproc/src/ProtoduneHD.cxx: drop the debug prints and medians/medians_decon
dump added earlier in this session.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace the single dead-code chndb-resp.jsonnet with two live files: - chndb-resp-bot.jsonnet: bottom CRP (cold ER, gain 7.8 mV/fC, postgain 1.1365, ADC/mV 11.70); response_offset = 239 (U) / 245 (V) - chndb-resp-top.jsonnet: top CRP (JsonElecResponse, postgain 1.52, ADC/mV 8.192); response_offset = 240 (U) / 243 (V) Both kernels are generated via `wirecell-sigproc track-response` with a 160 µs convolution window (PDVD FR file native length is ~132.5 µs; without padding the bipolar induction tail wraps in the FFT convolution). chndb-base.jsonnet is updated to import both files, select the correct kernel based on anode index (n<4 → bottom, n>=4 → top), and apply gain_scale = if n>=4 then 1.0 else params.elec.gain / (7.8*wc.mV/wc.fC) so runtime FE gain overrides propagate correctly. This activates the SignalProtection deconvolution gate for U/V channels on all PDVD anodes. Threshold tuning (decon_limit, roi_min_max_ratio) is a follow-up step. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…e from decon params PDVD: pad_window_front/back 20→10; decon_limit W=0.05, U/V=0.01(bot)/0.02(top); decon_limit1 W=0.08, U/V=0.07; adc_limit=60*gs; min_adc_limit=200*gs (new). decon_limit/decon_limit1 operate on deconvolved output (gain-normalised) so gain_scale is not applied. PDHD: remove gain_scale from all decon_limit/decon_limit1 entries (same reasoning); update comment to distinguish ADC-domain vs deconvolved-domain thresholds. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace the three hardcoded software filters in PDHDCoherentNoiseSub
and PDVDCoherentNoiseSub (filter_time, filter_low, filter_low_loose)
with the same IFilterWaveform components SP already uses:
- time-filter <- per-plane HfFilter Wiener_tight_{U,V,W}
(_APA1 variants on PDHD anode 0)
- filter_low <- LfFilter ROI_tighter_lf
- filter_low_loose <- LfFilter ROI_loose_lf
The four MicroBooNE-era hardcoded notches (107/178/214/250 kHz) were
dropped -- they were never re-tuned for PDHD/PDVD; if needed, they
should come from per-detector chndb freqmasks_phys instead.
PDVD nf.jsonnet has a per-CRP dispatch hook (ident < 4) so future
top/bot divergence is a one-line jsonnet change.
Verified end-to-end: PDHD anode 0, PDVD anode 0, PDVD anode 4 all
run cleanly with zero errors.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add an opt-in validation hook to PDHDCoherentNoiseSub /
PDVDCoherentNoiseSub: when jsonnet field data.debug_dump_path is
non-empty, apply() emits one .npz per channel group under
<path>/apa<N>/<plane>_g<gid>.npz capturing
- median, medians_decon (circular-shifted by +res_offset so the
deconvolved view is aligned in original time),
- signal_bool (post-pad_window) and signal_bool_raw (threshold
crossings only),
- the ROI list with per-ROI median scalars (max/min/ratio_obs and
the median's accept decision computed against decon_limit1 and
roi_min_max_ratio),
- the per-(channel, ROI) accept matrix that Subtract_WScaling
actually used,
- per-channel scaling_coef + ave_coef,
- every knob in scope (protection_factor, min_adc_limit,
upper_adc_limit, upper_decon_limit, decon_limit1,
roi_min_max_ratio, pad_front, pad_back, the chosen ADC and
deconv thresholds, RMS, mean, and the SP filter names).
The struct (sigproc/inc/WireCellSigProc/CoherentNoiseDump.h) and the
cnpy-based writer are detector-agnostic and shared by both PDHD and
PDVD. Storage: util/cnpy.h npz_save with append mode (one .npz per
group). Off-state cost: one .empty() check per apply() group.
SignalProtection / Subtract_WScaling gain a final default-nullptr
CoherentNoiseDump* arg; the hot path is bit-identical when
debug_dump_path is empty (verified: dump-OFF and dump-ON inner *.npy
arrays are np.array_equal across adjacent runs).
Wired through wct-nf-sp.jsonnet TLAs debug_dump_path /
debug_dump_groups in nf.jsonnet for both PDHD and PDVD.
…d: drop verbose debug logs
PDHD: adc_limit 30·gs, min_adc_limit 100·gs, pad_window 20/20 across U/V/W.
PDVD bottom: adc_limit 15·gs (U/V) / 20·gs (W); decon_limit 0.001 (U/V) /
0.005 (W); decon_limit1 0.007 (U/V) / 0.008 (W); min_adc_limit 50·gs.
PDVD top: adc_limit 20 (all planes); decon_limit 0.002 (U/V) / 0.005 (W);
decon_limit1 0.007 (U/V) / 0.008 (W); min_adc_limit 60.
pad_window_{front,back} = 20 for both detectors.
sigproc/pdvd: drop two per-channel m_log->debug() blocks
(PDVDfreqmask … zeroed bins; PDVDOneChannelNoise … noisy=…) that were
spamming the log at debug level on every NF frame.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The plan key included `bool aligned = (src&15)|(dst&15)` — a function of runtime heap buffer addresses. Under ASLR (and even without it, due to allocator-state variance between runs) this bit toggled between runs, selecting different cached FFTW codelets (SIMD vs scalar) that produce ULP-level FP differences. These differences propagated through NF coherent subtraction and through SP deconvolution FFTs, producing pervasive noise-level drift in NF output and up to ~434 ADC structural differences in SP output. Fix: add FFTW_UNALIGNED to every plan-creation call (7 sites) so plans are alignment-independent, and remove the `aligned` bit from make_key() so a single cached plan is reused for any buffer alignment. Verified bit-identical NF and SP output across repeated runs with both ASLR enabled and disabled (PDHD anodes 0 and 1). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…bility assessment Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New component L1SPFilterPD (Strategy B, per-ROI polarity detection):
- Full fork of L1SPFilter; original MicroBooNE source files are unchanged
- 2-column LASSO with polarity-keyed basis: {bipolar, +unipolar} or
{bipolar, -unipolar} selected by the per-ROI asymmetry trigger
- Trigger is stubbed (always pass-through) until calibration cuts are
determined from a dedicated event-scan + tagging analysis
- Layer 4 cross-channel cleaning removed (uBooNE shorted-wire rationale
does not apply to PDHD/PDVD geometry)
- Propagation generalised from bool to int active_polarity (±1/0) to
support bidirectional polarity propagation in forward+reverse passes
- unique_ptr<linterp<double>> for response interpolators; init_resp() lazy
- anonymous helpers build_G / lasso_solve for the segmented LASSO
- L1SPFilterPD.md: implementation reference, config table, pending work
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… mode and PDHD/PDVD wiring - L1SPFilterPD: add dump_mode (NPZ per event) + process_planes filter (U+V, skips W) via IAnodePlane lookup; dump-mode is a full pass-through that writes per-ROI asymmetry stats (temp_sum, temp1_sum, max/min, prev/next gap) without touching the signal traces - cfg/pdhd/sp.jsonnet, cfg/protodunevd/sp.jsonnet: extend make_sigproc with optional l1sp_pd_mode/dump_path/planes params (default OFF, bit-identical) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Make this new one, as #444 seems not working for the tracking purpose