Skip to content

Issue Report #352

@GernotMaier

Description

@GernotMaier

Issue Report

(used GPT-5.4 and Opus 4.7)

Scope: Full pipeline audit across EventDisplay_v4 (C++), Eventdisplay_AnalysisScripts_VTS
(shell orchestration), Eventdisplay_AnalysisFiles_VTS (auxiliary files), and Eventdisplay-ML
(XGB/Python ML). Log files and release-test artifacts were also reviewed.

Severity labels: HIGH — incorrect physics output or silent job failure; MEDIUM
degraded reliability, reproducibility, or correctness in edge cases; LOW — cosmetic,
misleading diagnostics, or minor fragility.


Pipeline overview (reference)

Stage IRF chain entry Data-analysis entry
Image reconstruction IRF.evndisp_MC.shevndisp ANALYSIS.evndisp.shevndisp
Stereo / table lookup IRF.mscw_energy_MC.shmscw_energy ANALYSIS.mscw_energy.shmscw_energy
Angular reco (ML) DispBDT / XGB training (IRF.train*) DispBDT XMLs consumed by mscw_energy
Gamma/hadron (ML) BDT / XGB training, optimisation, effective areas TMVA/XGB friend trees consumed by anasum
Science analysis ANALYSIS.anasum*.shanasum
DL3 export ANALYSIS.v2dl3.shv2dl3

C++ class chains:

  • evndisp: VReadRunParameterVEventLoopVTraceHandler/VFitTraceHandler, VCalibrator, VImageAnalyzer, VArrayAnalyzer, VPedestalCalculator
  • mscw_energy: VTableLookupRunParameterVTableLookupCData, VDispAnalyzer, VTMVADispAnalyzer, VEmissionHeightCalculator
  • anasum: VAnaSumRunParameterVAnaSumVStereoAnalysisCData, VGammaHadronCuts, VTMVAEvaluator
  • makeEffectiveArea: VGammaHadronCutsCDataVInstrumentResponseFunctionVInstrumentResponseFunctionData

A — C++ algorithm defects

A1 · HIGH · Out-of-bounds read in pulse-timing extraction

Location: EventDisplay_v4/src/VTraceHandler.cpp:346-358
getPulseTiming() interpolates on the rising edge using fpTrace[i+1]. When the pulse maximum
lands on the last sample (maxpos = size-1), i+1 is out of bounds.
Impact: Truncated or late pulses produce undefined timing values in the signal characterisation
path.

A2 · HIGH · Sliding-window charge branch returns FADC[1] instead of integrated charge

Location: EventDisplay_v4/src/VTraceHandler.cpp:745-757
calculateTraceSum_slidingWindow() computes charge and updates the integration-window metadata
but then returns FADC[1].
Impact: When the raw sliding-window branch is active, charge extraction is silently wrong.

A3 · MEDIUM · Fitted-trace T0 mixes pedestal-subtracted and absolute amplitude conventions

Location: EventDisplay_v4/src/VFitTraceHandler.cpp:433-442
getTraceMax() returns a pedestal-shifted peak, but getTraceTZero() solves for
-(imax/2 + fPed), re-adding the pedestal to the half-height target.
Impact: Fitted pulse-arrival times are biased whenever the pedestal is non-zero.

A4 · HIGH · Low-gain average-T0 assignment always writes the high-gain field

Location: EventDisplay_v4/src/VCalibrationData.cpp:365-373
setAverageTZero(bool iLowGain) unconditionally assigns fAverageTZero_highgain, even when
iLowGain == true.
Impact: The low-gain timing calibration path corrupts the high-gain average-T0 used later in
analysis.

A5 · HIGH · Low-gain average-T0 calculation uses the high-gain sum path

Location: EventDisplay_v4/src/VCalibrator.cpp:899-903
calculateAverageTZero(bool iLowGain) calls calcSums(..., iLowGainOnly=false) unconditionally.
Impact: Low-gain timing constants are derived from the wrong charge-extraction mode.

A6 · HIGH · Shower-core validity check tests ximp twice, never yimp

Location: EventDisplay_v4/src/VArrayAnalyzer.cpp:944-951
fillShowerCore() evaluates !isnormal(ximp) || !isnormal(ximp); the duplicate check means an
invalid yimp is never screened out.
Impact: NaN or invalid core-Y solutions propagate into core distances, energy reconstruction
inputs, and event-selection variables.

A7 · HIGH · Method-4 stereo pair-angle rejection applies cut to the wrong telescope pair

Location: EventDisplay_v4/src/VArrayAnalyzer.cpp:1241-1248
The per-pair rejection computes the current pair in mm[0]/mm[1] but evaluates the cut using
m[0]/m[1].
Impact: For events with more than two images, pair-angle rejection is applied to the wrong
telescopes, biasing the weighted line intersections used for direction reconstruction.

A8 · HIGH · Log-likelihood image fit can be initialised with invalid centroid bounds

Location: EventDisplay_v4/src/VImageParameterCalculation.cpp:1542-1555,1588-1601
For edge/degenerate images with sigmaX <= 0 or sigmaY <= 0, only the upper Minuit bound is
set; the lower bound retains its sentinel value before being passed to the minimiser.
Impact: Edge images can enter the recovery LL fit with invalid parameter bounds, degrading
parameterisation in the regime where the fit is intended to stabilise results.

A9 · MEDIUM · Muon-ring image-size correction divides by zero at the first grid point

Location: EventDisplay_v4/src/VImageParameterCalculation.cpp:737-748
xi_tmp = i / numSteps is zero when i == 0, but kTest_tmp = ... / xi_tmp is evaluated before
any guard.
Impact: Muon calibration correction can produce inf/NaN candidates and an unstable
correction factor.

A10 · MEDIUM · Emission-height pair weights undefined for small image sizes

Location: EventDisplay_v4/src/VEmissionHeightCalculator.cpp:60-64
The weight formula 1./((1./log10(size[i])) + (1./log10(size[j]))) is not guarded against
size <= 1.
Impact: Small-image events receive singular, negative, or ill-defined pair weights in the
emission-height estimate.

A11 · MEDIUM · VEmissionHeightCalculator accumulates telescope positions across reinitalisations

Location: EventDisplay_v4/src/VEmissionHeightCalculator.cpp:100-133
Both setTelescopePositions() overloads only push_back() and never clear previous contents.
Impact: Reusing the same calculator instance duplicates telescope geometry, making pairwise
shower-height geometry wrong.

A12 · HIGH · Core-resolution IRF formula uses Xcore twice instead of Xcore and Ycore

Location: EventDisplay_v4/src/VInstrumentResponseFunctionData.cpp:472-485
The core-resolution error is computed as sqrt(Xcore²+Xcore²) - sqrt(MCxcore²+MCycore²).
Impact: Reconstructed core-error quantities are biased whenever Ycore differs from Xcore;
this is a direct physics error in the IRF-resolution path.

A13 · HIGH · Gamma/hadron TMVA training has an off-by-one zenith-bin bounds check

Location: EventDisplay_v4/src/trainTMVAforGammaHadronSeparation.cpp:255-266
The check uses < iZenithBin instead of <= iZenithBin, so iZenithBin == size() passes and is
then dereferenced.
Impact: Certain bin configurations index beyond the configured zenith-bin vectors during
training.

A14 · HIGH · Reconstruction-quality TMVA training configured as a classification factory

Location: EventDisplay_v4/src/trainTMVAforGammaHadronSeparation.cpp:343-360
The non-gamma/hadron branch adds a regression target but creates the TMVA::Factory with the
default (classification) options string.
Impact: The code path described as reconstruction-quality training is algorithmically
mismatched to the TMVA factory configuration.

A15 · MEDIUM · Angular-reconstruction training silently rescales the requested train/test split

Location: EventDisplay_v4/src/trainTMVAforAngularReconstruction.cpp:125-138
After deriving ntrain and ntest from the requested fraction, both are multiplied by 0.8.
Impact: Effective train/test statistics differ from the configured values, reducing
reproducibility.

A16 · MEDIUM · Reduced-tree subsampling normalised by file count, not event population

Location: EventDisplay_v4/src/trainTMVAforGammaHadronSeparation.cpp:147-157
The retained fraction is divided by iTreeVector.size() (number of input files), not by event
counts.
Impact: Training preselection depends on file granularity and can distort class composition
across uneven file splits.

A17 · HIGH · Sliding-window charge extraction reads one past the end of the FADC array

Location: EventDisplay_v4/src/VTraceHandler.cpp:725-783
iSearchEnd is clamped to n - iIntegrationWindow + 1. At the final loop iteration
(i = iSearchEnd - 1), the expression FADC[i + iIntegrationWindow] evaluates to FADC[n],
which is one past the end of the valid array range [0, n-1].
Impact: Undefined behaviour on every waveform where the sliding window reaches its rightmost
position; corrupts charge reconstruction silently on affected events.

A18 · MEDIUM · VTraceHandler::getQuickMax uses the previous call's trace maximum for saturation counting

Location: EventDisplay_v4/src/VTraceHandler.cpp:420-426
nMax is computed as (int)(fDynamicRange * tmax) before tmax is reset to −10000. If the
caller passes an uninitialised or stale value, the saturation counter n255 is thresholded
against the wrong maximum.
Impact: The number of saturated samples can be over- or under-estimated, misclassifying
pulses as saturated or unsaturated and biasing image-cleaning amplitude thresholds.


B — C++ infrastructure defects

B1 · MEDIUM · CData::getXGBTree() assumes .root is present in the filename

Location: EventDisplay_v4/src/CData.cpp:1073-1086
The XGB friend-file name is rewritten via string::replace(find(".root"), ...) without checking
whether .root exists in the name.
Impact: The class API is brittle and can generate an invalid replacement if reused with a
non-standard input name.

B2 · MEDIUM · XGB gamma/hadron C++ interface hard-wired to Is_Gamma_70

Location: EventDisplay_v4/src/CData.cpp:1118-1127; EventDisplay_v4/src/VGammaHadronCuts.cpp:727-731,1118-1128
The C++ consumer binds only Gamma_Prediction and Is_Gamma_70; the classification operating
point is baked into the upstream friend-tree producer, not enforced by VGammaHadronCuts.
Impact: The 70% efficiency threshold cannot be adjusted from cut files or runtime parameters,
discarding all the optimisation flexibility of the XGB classifier.

B3 · MEDIUM · VTMVAEvaluator recovers feature names by brittle XML string slicing

Location: EventDisplay_v4/src/VTMVAEvaluator.cpp:62-116
Variable recovery depends on exact XML formatting and field ordering (NVar, Expression,
Label, SpecIndex) rather than a structured parser.
Impact: A TMVA XML layout change can silently alter which variables the evaluator reads,
turning the deployed BDT interface into a schema-fragile contract.

B4 · MEDIUM · TMVA weight grid initialisation tolerates missing interior bins with warnings

Location: EventDisplay_v4/src/VTMVAEvaluator.cpp:190-279
Missing weight files at non-edge bins emit warnings and allow the evaluator to continue.
Impact: Incomplete TMVA weight grids survive initialisation, making production behaviour
depend on partial bin coverage instead of a strict consistency check.

B5 · LOW · VTMVARunData sanity message does not match the actual condition

Location: EventDisplay_v4/src/VTMVARunData.cpp:795-800
Prints "need at least two energy bins" while the code only rejects < 1 bin.
Impact: Misleading diagnostics when debugging training-configuration problems.

B6 · MEDIUM · VInstrumentResponseFunction mutates shared MC azimuth in place

Location: EventDisplay_v4/src/VInstrumentResponseFunction.cpp:193-199
MC azimuth normalisation is applied as fData->MCaz -= 360. on the shared event object.
Impact: Weak implementation pattern; mutates shared state instead of using a local normalised
copy.

B7 · MEDIUM · makeEffectiveArea IRF duplication is order-dependent

Location: EventDisplay_v4/src/makeEffectiveArea.cpp:162-208,194-208
IRF products are duplicated by index via getDuplicationID() and only work if the source has
already been initialised earlier in the vector.
Impact: A future refactor changing the internal IRF ordering will silently break resolution
product generation.

B8 · MEDIUM · copyMCHistograms() keeps the first ROOT file open

Location: EventDisplay_v4/src/makeEffectiveArea.cpp:370-401
The first MChistos object is borrowed directly from the first input file, which is never closed
inside the helper.
Impact: File-handle lifetime is coupled to the returned object, risking resource leaks or
dangling-file state.

B9 · MEDIUM · anasum contains a stale unreachable analysisType == 5 branch

Location: EventDisplay_v4/src/anasum.cpp:47-54,106-108
analysisType is explicitly limited to 3 and 4, but the VAnaSum::doStereoAnalysis() handoff
still tests analysisType == 5.
Impact: Dead code path; sign of an incomplete control-flow change.

B10 · MEDIUM · Epoch reevaluation at runtime means analysis behaviour changes with epoch-file updates

Location: EventDisplay_v4/src/VReadRunParameter.cpp:463-470; EventDisplay_v4/src/printRunParameter.cpp:100-117
printRunParameter updated-runinfo and evndisp -epochfile reevaluate epoch and atmosphere from
VERITAS.Epochs.runparameter each run rather than relying solely on the file-internal metadata.
Impact: Reprocessing the same intermediate ROOT file with an updated epoch file can silently
change the science output.

B11 · HIGH · VOnOff mutates the background histogram after using it, corrupting reflected-region normalisation

Location: EventDisplay_v4/src/VOnOff.cpp:159-161
doOnOffforParameterHistograms computes the excess as
hTemp->Add(hon, hoff, 1., -1.*i_norm_alpha) (which consumes hoff at its original scale) and
then immediately calls hoff->Scale(i_norm_alpha). Any caller that reuses hoff after this call
sees a rescaled background histogram.
Impact: The reflected-background normalisation in the anasum analysis path (via VAnaSum.cpp:556)
can be doubly-scaled, producing wrong excess counts and significance values.

B12 · HIGH · makeEffectiveArea comments that the MC run header is optional but hard-exits when it is missing

Location: EventDisplay_v4/src/makeEffectiveArea.cpp:118; EventDisplay_v4/src/VInstrumentResponseFunctionRunParameter.cpp:430-437
The comment at line 118 states "might not be there, no problem", yet readMCRunHeader() calls
exit(EXIT_FAILURE) when no MC_runheader object is found in the input file.
Impact: Misleading documentation causes operators to expect graceful handling; actual behaviour
is a hard crash with a non-zero exit code when the MC header is absent (e.g., for older files).

B13 · HIGH · VInstrumentResponseFunctionRunParameter silently ignores unknown run-parameter keys

Location: EventDisplay_v4/src/VInstrumentResponseFunctionRunParameter.cpp:109-358
The parser is a chain of else if branches ending at RERUN_STEREO_RECONSTRUCTION_3TEL with
no terminal else and no warning message. Unknown or misspelled keys are silently dropped.
Impact: Typos in makeEffectiveArea parameter files produce no diagnostic and default values
are used; an operator has no way to detect that a setting was not applied.

B14 · MEDIUM · printRunParameter exits with code 0 on file-open failure

Location: EventDisplay_v4/src/printRunParameter.cpp:330-335
new TFile(argv[1]) can return a zombie object; the code prints "exiting..." and calls exit(0).
Impact: Shell wrappers that check only the exit code (C6, several IRF helpers) will interpret
the empty output as a missing field rather than a fatal error, silently propagating bad metadata
through the pipeline.

B15 · LOW · VTableLookupRunParameter error message says "2=read" but the code only accepts "0=read"

Location: EventDisplay_v4/src/VTableLookupRunParameter.cpp:162-176
The -fill option accepts 0 (read) or 1 (fill); any other value triggers the message
"unknown parameter, choose 1=fill or 2=read lookup tables", which is incorrect.
Impact: Operators following the error message will pass -fill=2, which also triggers the
error in a confusing loop.


C — Shell orchestration bugs

C1 · HIGH · ANALYSIS.anasum.sh never submits any jobs

Location: Eventdisplay_AnalysisScripts_VTS/scripts/ANALYSIS.anasum.sh:90-92
Line 90 assigns the helper path as a string; the subsequent eval only echoes it back. Every
other equivalent script correctly uses backticks to execute the helper:

# Broken (line 90):
SUBC="$(dirname "$0")/helper_scripts/UTILITY.readSubmissionCommand.sh"
SUBC=`eval "echo \"$SUBC\""`    # still a string, never executed

# Correct (evndisp.sh:126):
SUBC=`$(dirname "$0")/helper_scripts/UTILITY.readSubmissionCommand.sh`

All *qsub*/*condor*/*sbatch*/*simple* tests fail, so generated job scripts are written
but never submitted.
Impact: The full-runlist anasum path silently drops all jobs with no error message.

C2 · HIGH · get_disp_dir() checks an undefined variable, always selects the wrong DispBDT directory

Location: Eventdisplay_AnalysisScripts_VTS/scripts/helper_scripts/ANALYSIS.mscw_energy_sub.sh:136-152
The function checks ${SIMTYPE} == "CARE_RedHV" to decide between 55-deg and 60-deg DispBDT
fallback models. SIMTYPE is never assigned in this script; only SIMTYPE_RUN is set (lines
82–93). A secondary issue: the current value since v491 is CARE_RedHV_Feb2024, not
CARE_RedHV, so the comparison would fail even if the variable were correct. The proper guard
should use HVSETTINGS == "obsLowHV".
Impact: For all RedHV (and high-zenith UV) runs, mscw_energy loads 60-deg DispBDT models
instead of the 55-deg fallback, degrading direction reconstruction quality for runs at zenith ≥ 58°.

C3 · MEDIUM · FORCEDATMO atmosphere override is silently ignored in cluster submissions

Location: Eventdisplay_AnalysisScripts_VTS/scripts/ANALYSIS.mscw_energy.sh:63; helper_scripts/ANALYSIS.mscw_energy_sub.sh:71
The parent script captures FORCEDATMO=$6 locally. The sed-template substitution block does not
inject FORCEDATMO into the generated job script, and the job script references
${FORCEDATMO:-...} as an unexported shell variable, which is empty in batch environments.
Impact: The documented [ATM] override parameter does nothing in production; users
attempting to force an atmosphere ID get silently wrong results.

C4 · MEDIUM · mscw_energy_sub.sh missing atmosphere-ID normalisation for V4/V5 data

Location: Eventdisplay_AnalysisScripts_VTS/scripts/helper_scripts/ANALYSIS.mscw_energy_sub.sh:68-71; ANALYSIS.anasum_sub.sh:74-83
mscw_energy_sub.sh calls printRunParameter … updated-runinfo, which may return a modernised
6x atmosphere ID for old V4/V5 observations. anasum_sub.sh compensates with
prepare_atmo_string() (${ATMO/6/2} for V4/V5); mscw_energy_sub.sh has no equivalent
conversion.
Impact: mscw_energy exits with "table file not found" for V4/V5 GRISU data when the epoch
files return a modernised atmosphere ID.

C5 · MEDIUM · DISPBDT guard in mscw_energy_sub.sh is missing $

Location: Eventdisplay_AnalysisScripts_VTS/scripts/helper_scripts/ANALYSIS.mscw_energy_sub.sh:193
if [[ DISPBDT != "NOTSET" ]]; then references the literal string DISPBDT instead of
$DISPBDT, so the condition is never conditionally guarded.
Impact: The log reports DispBDT XML provenance even when that branch is not actually active.

C6 · MEDIUM · printRunParameter -elevation field is parsed positionally in the helper

Location: Eventdisplay_AnalysisScripts_VTS/scripts/helper_scripts/ANALYSIS.mscw_energy_sub.sh:127; EventDisplay_v4/src/printRunParameter.cpp:340-343
The helper extracts the zenith angle by awk field number from a human-readable CLI printout
rather than from a stable machine-readable interface.
Impact: Fragile table and DispBDT directory selection; breaks silently if the printout format
changes.

C7 · MEDIUM · mscw_energy hard-fails when a TMVA XML family is missing, no soft fallback

Location: EventDisplay_v4/src/VTMVADispAnalyzer.cpp:97-105,164-170
When TMVA XML prefixes are provided and one family is missing, the code exits without graceful
degradation.
Impact: Missing or incompletely copied DispBDT XMLs stop the run.

C8 · MEDIUM · anasum default input directory hard-wired to RecID0 with no consistency check

Location: Eventdisplay_AnalysisScripts_VTS/scripts/ANALYSIS.anasum.sh:47-49; helper_scripts/ANALYSIS.anasum_sub.sh:213-226
The default input directory is .../RecID0; no validation that the mscw.root files were
produced with the requested reconstruction ID.
Impact: A workflow that reconstructs with a non-zero RecID can run anasum against stale
RecID0 products, producing plausible but inconsistent science outputs.

C9 · HIGH · ANALYSIS.evndisp_sub.sh consumes DBTEXT-derived files without existence checks

Location: Eventdisplay_AnalysisScripts_VTS/scripts/helper_scripts/ANALYSIS.evndisp_sub.sh:97-107,137-166
After unpacking the DBTEXT tarball, the helper immediately cats ${TMP_LASERRUN} and parses
${RUN}.runinfo without checking that either file exists or is readable.
Impact: Missing or corrupt DBTEXT tarballs silently propagate empty laser-run metadata or
wrong run dates into CVBF lookup and calibration selection.

C10 · HIGH · Broken TMVA bin-parsing regex in training and optimisation scripts

Location: Eventdisplay_AnalysisScripts_VTS/scripts/IRF.trainTMVAforGammaHadronSeparation.sh:107-109; IRF.optimizeTMVAforGammaHadronSeparation.sh:91-93
grep "^* ENERGYBINS" and the matching sed expressions are invalid or broken (* is not
a valid regex anchor in this position).
Impact: Training and optimisation bin parsing can fail or misbehave, directly affecting
gamma/hadron separation products.

C11 · MEDIUM · Cut optimisation continues with missing effective-area input

Location: Eventdisplay_AnalysisScripts_VTS/scripts/IRF.optimizeTMVAforGammaHadronSeparation.sh:81-83
Logs an error on missing effective-area file but does not exit.
Impact: Optimisation jobs may run with incomplete inputs, producing invalid cut tuning.

C12 · MEDIUM · DL3 jobs install Python dependencies at runtime per job

Location: Eventdisplay_AnalysisScripts_VTS/scripts/helper_scripts/ANALYSIS.v2dl3_sub.sh:56-58
Each job activates the base environment and runs pip install -e ....
Impact: Slow, non-reproducible, and environment-dependent production.

C13 · MEDIUM · ANALYSIS.anasum_combine.sh lacks a completion/integrity guard before merge

Location: Eventdisplay_AnalysisScripts_VTS/scripts/ANALYSIS.anasum_combine.sh:31-36,85-115
Assumes parallel jobs are finished; no strong completeness validation before combining.
Impact: Partial combined outputs can be produced silently.

C14 · MEDIUM · Data-side mscw_energy selects IRF products purely from filename templates

Location: Eventdisplay_AnalysisScripts_VTS/scripts/helper_scripts/ANALYSIS.mscw_energy_sub.sh:120-179,182-188
The helper derives the IRF table path and DispBDT directory from printRunParameter output and
string templates, then runs mscw_energy without verifying that the chosen XMLs and tables
match the input evndisp.root configuration.
Impact: Stale or mismatched IRF products can be applied to otherwise valid evndisp files,
yielding apparently normal mscw.root outputs built with the wrong reconstruction tables.

C15 · MEDIUM · DBTEXT run extraction is fragile against schema drift in .runinfo files

Location: Eventdisplay_AnalysisScripts_VTS/scripts/db_scripts/db_run.sh:69-115,165-185
db_run.sh re-parses .runinfo headers by name; if a header is absent, the helper falls back
to empty or stale column indexes.
Impact: Downstream scripts can derive wrong time windows, source IDs, and DB queries from
superficially valid DBTEXT files.

C16 · MEDIUM · Existing DBTEXT tarballs suppress regeneration of laser-calibration side files

Location: Eventdisplay_AnalysisScripts_VTS/scripts/db_scripts/db_run.sh:294-297
If <run>.tar.gz exists, the script skips read_laser_calibration entirely rather than
checking whether .laserrun, .gain, .toffset, and related files are current inside the
tarball.
Impact: Stale or incomplete DBTEXT tarballs freeze outdated calibration metadata and prevent
later evndisp analyses from refreshing it.

C17 · MEDIUM · FITS converter crashes on missing DQM-like DBTEXT files before the None check

Location: Eventdisplay_AnalysisScripts_VTS/scripts/db_scripts/db_write_fits.py:425-433
convert_table_comment_to_ascii(table) is called for dqm files before checking
if table is not None.
Impact: One missing DBTEXT component aborts FITS generation and drops all DQM metadata for
the run.

C18 · MEDIUM · DQM FITS combination requires identical schemas across all runs

Location: Eventdisplay_AnalysisScripts_VTS/scripts/db_scripts/db_combine_dqm_fits.py:40-54
Uses vstack(..., join_type="exact"), so any column drift makes the whole merge fail.
Impact: Long-run DQM aggregation is brittle against minor schema evolution.

C19 · MEDIUM · DQM rate/dead-time extraction does not guard malformed scaler intervals

Location: Eventdisplay_AnalysisScripts_VTS/scripts/db_scripts/db_write_fits.py:101-138
np.diff() and scaler divisions are performed with no protection against zero time_diffs or
zero TenMHzScaler increments.
Impact: Invalid scaler records can inject inf/NaN values into exported DQM quantities.

C20 · LOW · ANALYSIS.v2dl3.sh has typo LOGIDR and undefined variable AFILE

Location: Eventdisplay_AnalysisScripts_VTS/scripts/ANALYSIS.v2dl3.sh:53-58,93-106,117-122
Impact: Stale temp files and misleading job logs.

C21 · LOW · ANALYSIS.anasum_allcuts.sh uses relative ./ANALYSIS... calls

Location: Eventdisplay_AnalysisScripts_VTS/scripts/ANALYSIS.anasum_allcuts.sh:66-76
Impact: Fails unless launched from the scripts directory.

C22 · LOW · IRF packaging script has misspelled function name and undefined $F in logging

Location: Eventdisplay_AnalysisFiles_VTS/transfer/pack_irfs_for_ucla.sh:161-185
Impact: Fragile packaging and misleading archive diagnostics.


D — IRF production chain

D1 · HIGH · IRF.production.sh can resubmit the full TMVA grid multiple times

Location: Eventdisplay_AnalysisScripts_VTS/scripts/IRF.production.sh:337-387
The TRAINTMVA/OPTIMIZETMVA block contains inner loops that re-iterate over the full EPOCH
and ATMOS sets while already inside outer loops over the same sets.
Impact: TMVA training and optimisation jobs are submitted len(EPOCH) × len(ATMOS) times,
wasting batch capacity and mixing results.

D2 · HIGH · Inner TMVA block uses ${EPOCH:0:2} instead of ${VX:0:2} for epoch selection

Location: Eventdisplay_AnalysisScripts_VTS/scripts/IRF.production.sh:353-366
Inside the for VX in $EPOCH loop, size-cut file grep and TMVA runparameter selection still use
the outer ${EPOCH:0:2} instead of the current iteration variable ${VX:0:2}.
Impact: Multi-epoch submissions select the wrong size-cut row and TMVA runparameter template
for all epochs beyond the first.

D3 · HIGH · IRF.mscw_energy_MC_sub.sh quoted glob fails to copy uncompressed ROOT files

Location: Eventdisplay_AnalysisScripts_VTS/scripts/helper_scripts/IRF.mscw_energy_MC_sub.sh:133-136
cp -v "${INDIR}/*[0-9].root" "$DDIR" — the glob is inside double quotes and is not expanded
by the shell.
Impact: If upstream evndisp products are uncompressed .root files, the MC mscw_energy
stage fails before building its file list.

D4 · HIGH · Job-launch scripts generate one filename and submit another

Location: Eventdisplay_AnalysisScripts_VTS/scripts/IRF.generate_lookup_table_parts.sh:86-124; IRF.mscw_energy_MC.sh:97-139
Both scripts write the runnable script to $FSCRIPT but submit $FSCRIPT.sh, pointing to a
non-existent pathname.
Impact: Core IRF production stages — lookup-table and MC mscw_energy generation — fail
before creating their outputs, blocking all downstream steps.

D5 · HIGH · Gamma/hadron TMVA training ignores the documented UUID argument

Location: Eventdisplay_AnalysisScripts_VTS/scripts/IRF.trainTMVAforGammaHadronSeparation.sh:18-19,46,60
The help text places the optional UUID as argument 8, but the script reads it from $12.
Impact: Repeated training campaigns reuse the same submit/log directory, mixing products and
overwriting job state.

D6 · MEDIUM · Lookup-table combine checks the wrong path before overwriting

Location: Eventdisplay_AnalysisScripts_VTS/scripts/IRF.combine_lookup_table_parts.sh:77-80
Guard tests [[ -f $OFILE ]] instead of [[ -f "$ODIR/$OFILE" ]].
Impact: Existing combined lookup tables in the output directory are not reliably detected
before overwrite or merge attempts.

D7 · MEDIUM · zstd availability checks are malformed in several IRF helpers

Location: IRF.compress_evndisp_MC_sub.sh:46-54; IRF.evndisp_MC_sub.sh:323-330; IRF.lookup_table_parallel_sub.sh:67-79; IRF.mscw_energy_MC_sub.sh:137-149
command -v zstd /dev/null is not the standard existence test.
Impact: Compression/decompression branches can report false missing-tool failures.

D8 · MEDIUM · IRF.evndisp_MC_sub.sh passes literal glob to zstd when no files match

Location: Eventdisplay_AnalysisScripts_VTS/scripts/helper_scripts/IRF.evndisp_MC_sub.sh:88-103
The loop runs over "$V4N"/*.root.zst without checking whether any files match; the literal
pattern is then passed to zstd.
Impact: Spurious decompression failures during duplicate-processing detection.

D9 · MEDIUM · IRF.optimizeTMVAforGammaHadronSeparation_sub.sh uses MVADIR before definition

Location: Eventdisplay_AnalysisScripts_VTS/scripts/helper_scripts/IRF.optimizeTMVAforGammaHadronSeparation_sub.sh:54-69
rm -f ${MVADIR}/rates.log executes before MVADIR is assigned later in the file.
Impact: The cleanup step targets the wrong directory, potentially using an inherited
environment value.

D10 · MEDIUM · IRF.production.sh mixes robust $(dirname "$0")/... calls with fragile ./IRF... calls

Location: Eventdisplay_AnalysisScripts_VTS/scripts/IRF.production.sh:311,319,330,372,379
Impact: IRF production fails if launched from outside the scripts directory.

D11 · MEDIUM · makeEffectiveArea IRF resolution duplication is order-dependent in C++

Location: EventDisplay_v4/src/makeEffectiveArea.cpp:162-208
(See also B7.) IRF products are duplicated by index and only work if the source IRF was
initialised earlier in the vector.
Impact: Any refactor changing internal IRF ordering silently breaks resolution product
generation.

D12 · LOW · TMVA gamma/hadron preselection applies an unjustified 10× random retention factor

Location: EventDisplay_v4/src/trainTMVAforGammaHadronSeparation.cpp:143-185
After computing the required retained fraction, the code multiplies it by 10. before capping
at unity.
Impact: Makes the effective preselection behaviour opaque and bin-statistics dependent.


E — XGB / ML integration

E1 · HIGH · XGB data-analysis step not triggered automatically before XGB anasum cuts

Location: Eventdisplay_AnalysisScripts_VTS/scripts/ANALYSIS.dispXGB.sh:11-95; ANALYSIS.anasum_allcuts.sh:6-10,57-71; ANALYSIS.anasum_parallel_from_runlist.sh:133-145
ANALYSIS.dispXGB.sh exists as a standalone script, and the anasum wrappers now expose XGB
cut aliases (moderate2telXGB, etc.), but the documented workflow goes straight from
ANALYSIS.mscw_energy.sh to anasum. The wrappers do not trigger ANALYSIS.dispXGB.sh
automatically before selecting *-XGB-BDT cuts.
Impact: Users can request XGB cuts while analysing plain mscw files with no
.xgb_stereo.root/.xgb_gh.root sidecars, making XGB analyses a fragile manual side-band
instead of a reproducible pipeline stage.

E2 · HIGH · IRF.mscw_energy_MC_sub.sh XGB branch uses stale CLI and wrong run-parameter keys

Location: Eventdisplay_AnalysisScripts_VTS/scripts/helper_scripts/IRF.mscw_energy_MC_sub.sh:237-271,322-347; Eventdisplay-ML/src/eventdisplay_ml/config.py:175-230; EventDisplay_v4/src/VInstrumentResponseFunctionRunParameter.cpp:249-263
The helper calls eventdisplay-ml-apply-xgb-stereo --input-file --model-dir --output-file, but
the current CLI expects --input_file --model_prefix --output_file. It also writes
* XGBFILESUFFIX into the effective-area parameter file, while EventDisplay_v4 now parses
only XGBSTEREOFILESUFFIX and XGBGAMMAHADRONFILESUFFIX.
Impact: The legacy in-subscript XGB branch cannot drive the current Python CLI or pass its
outputs to the current C++ consumer; any workflow relying on this path is broken or silently
stale.

E3 · HIGH · XGB stereo training ignores the requested reconstruction ID

Location: Eventdisplay_AnalysisScripts_VTS/scripts/IRF.trainXGBforAngularReconstructionBinned.sh:48-56,76
The script accepts <Rec ID> as a required argument and carries it through naming and logging,
but hard-codes the training input directory to .../MSCW_RECID0_DISP.
Impact: Training jobs submitted for reconstruction IDs other than 0 silently use wrong MC
products.

E4 · HIGH · Non-XGB effective-area jobs still probe XGB sidecar files

Location: EventDisplay_v4/src/VStereoAnalysis.cpp:1965-1972; EventDisplay_v4/src/CData.cpp:28-38,1073-1127; logFiles/irfs/20deg_0.5wob_NOISE200.effarea.log:21-23
CData opens XGB friend files from non-empty suffix strings before checking whether the active
reconstruction methods require them. Confirmed in a production IRF log with
ENERGYRECONSTRUCTIONMETHOD 0 and a TMVA preselection cut file, yet the job still tries to open
.xgb_stereo.root/.xgb_gh.root.
Impact: Non-XGB jobs emit avoidable file-open errors and retain hidden dependencies on
artifacts that are not required.

E5 · HIGH · Eventdisplay-ML classification training and application can use different feature sets

Location: Eventdisplay-ML/src/eventdisplay_ml/data_processing.py:807-817; Eventdisplay-ML/src/eventdisplay_ml/models.py:441-452
Classification training switches to features_tmva_style(...) when tmva_style=True, but the
application path always uses features.features(analysis_type, training=False) and does not
mirror that branch.
Impact: A TMVA-style-trained classification model can be evaluated with the wrong input
feature set.

E6 · HIGH · Eventdisplay-ML config.py calls .get() on a list

Location: Eventdisplay-ML/src/eventdisplay_ml/config.py:162-165
configure_training() uses model_parameters.get("energy_bins_log10_tev", []).get("E_min");
energy_bins_log10_tev is list-structured and a list does not have .get().
Impact: Classification training/apply configuration can fail with AttributeError on
valid-looking parameter files in certain call paths.

E7 · MEDIUM · XGB training helper scratch directory uses undefined variable MSCW_FILE

Location: Eventdisplay_AnalysisScripts_VTS/scripts/helper_scripts/IRF.trainXGBforAngularReconstruction_sub.sh:15-18
TEMPDIR is derived from $(basename $MSCW_FILE .root), but MSCW_FILE is never defined; the
actual input is the list file LLIST.
Impact: Scratch-directory naming is accidental and environment-dependent, causing collisions
in batch production and making failed training jobs harder to debug.

E8 · MEDIUM · EVNDISP_MLSYS documented but helpers require a hard-coded conda environment name

Location: Eventdisplay_AnalysisScripts_VTS/scripts/helper_scripts/ANALYSIS.dispXGB_sub.sh:9-10,27-53; IRF.dispXGB_sub.sh:8-9; IRF.trainXGBforAngularReconstruction_sub.sh:9-10; IRF.trainXGBforGammaHadronSeparation_sub.sh:13-14; README.md:16-18,22-29
The public setup documents EVNDISP_MLSYS, but the helpers ignore it and require a pre-existing
conda environment named eventdisplay_ml.
Impact: A user can satisfy all documented environment variables and still fail at runtime.

E9 · MEDIUM · XGBoost hyperparameters are hard-coded in Python, not versioned in auxiliary parameter files

Location: Eventdisplay-ML/src/eventdisplay_ml/hyper_parameters.py:9-45
n_estimators, learning_rate, max_depth, subsample, colsample_bytree, etc. are embedded
in code; the shipped JSON files define only binning and tmva_style.
Impact: The effective ML model is not fully described by the analysis parameter files,
weakening reproducibility.

E10 · MEDIUM · Production XGB training helpers omit --random_state and --hyperparameter_config

Location: IRF.trainXGBforGammaHadronSeparation_sub.sh:59-67; IRF.trainXGBforAngularReconstruction_sub.sh:55-61
The helpers pass train_test_fraction and event limits but not a fixed random seed or explicit
hyperparameter file.
Impact: Routine retraining is not strictly reproducible even when input event lists are
unchanged.

E11 · MEDIUM · XGB command names are inconsistent between production scripts and documentation

Location: IRF.generalproduction.sh:18-29; IRF.production.sh:24-31,392-403; docs/changes/124.feature.md:1-10; README.md:83-159
IRF.generalproduction.sh advertises TRAINXGBANGRESBINNED; IRF.production.sh implements
TRAINXGBANGRES; the change note references a non-existent
IRF.trainXGBforAngularReconstruction.sh; the main README omits the XGB production sequence
entirely.
Impact: Operators are left with conflicting command names and incomplete instructions.

E12 · MEDIUM · Legacy TMVA.BDT.V4.runparameter ships a malformed PREPARE_TRAINING_OPTIONS line

Location: Eventdisplay_AnalysisFiles_VTS/ParameterFiles/TMVA.BDT.V4.runparameter:68-76; EventDisplay_v4/src/VTMVARunData.cpp:423-435
The line lacks the leading * required for parsing and contains a double :: in the option
string.
Impact: Legacy V4 BDT configuration does not reliably describe the training setup it
appears to encode.

E13 · LOW · TMVA training options have drifted from the accepted backend behaviour

Location: logFiles/irfs/mvaAngRes_20deg-BDTDisp-Tel2.log:81-82
NegWeightTreatment=InverseBoostNegWeights is not supported for BoostType=Grad and is silently
replaced by Pray.
Impact: Effective training configuration no longer matches the nominal requested one.


F — Auxiliary files & configuration

F1 · HIGH · Epoch metadata and downloadable IRF manifests are out of sync

Location: Eventdisplay_AnalysisFiles_VTS/ParameterFiles/VERITAS.Epochs.runparameter:47-53; Eventdisplay_AnalysisFiles_VTS/transfer/file_list_V6.dat:304-358
VERITAS.Epochs.runparameter defines V6_2024_2024s, V6_2024_2025w, V6_2025_2025s, and
V6_2025_2026w, but the shipped V6 download manifest stops at V6_2023_2024w.
Impact: A fresh checkout following the README cannot fetch the IRF tarballs needed for the
newer epochs that the active scripts may select at runtime.

F2 · HIGH · Future epoch V6_2025_2025s is defined twice in the epoch file

Location: Eventdisplay_AnalysisFiles_VTS/ParameterFiles/VERITAS.Epochs.runparameter:48-53
The second entry covers 2026-05-01 to 2026-11-24 while a TODO states it should become
V6_2026_2026s.
Impact: Future epoch lookup becomes ambiguous and can map runs to the wrong gain/atmosphere
configuration.

F3 · HIGH · Documented download path does not reproduce the full current auxiliary-file set

Location: Eventdisplay_AnalysisFiles_VTS/README.md:10-19; transfer/file_list_V6.dat:1-45,170-186; auxiliary_files.txt:19632-19645,20236-20262
The download manifests expose only classic DispBDTs and effective areas; the fully unpacked
inventory in auxiliary_files.txt additionally contains a DispXGB tree and XGB
effective-area products.
Impact: Two users following the official instructions end up with materially different
auxiliary installations.

F4 · HIGH · Unpacked auxiliary inventory is effectively AP-only despite scripts advertising multi-type support

Location: Eventdisplay_AnalysisScripts_VTS/README.md:58-81; IRF.production.sh:91-120; auxiliary_files.txt:19629-19631,21112-21169
Scripts and documentation present AP, NN, TS, and CC as supported analysis types; the
V6 file list even advertises DispBDTs_NN_* and EffectiveAreas_nominalHV_NN_* tarballs. The
unpacked inventory shows only an empty DispBDTs/NN directory; all table and effective-area
roots are -AP.root products.
Impact: The supposedly complete auxiliary area cannot support NN (or TS/CC) analyses.

F5 · HIGH · XGB instrument-file coverage is patchy while wrappers expose XGB as a generic AP mode

Location: auxiliary_files.txt:19632-19645,19697,19734,19933,20010,20177,20236-20262
Most XGB model directories are ATM61 only; XGB effective areas are confined to
V6_2016_2017 for ATM61/62, far narrower than the generic XGB hooks now exposed by the AP
wrappers.
Impact: The auxiliary-file area does not yet support XGB as a broadly deployable production
mode across the same epoch space as the classic AP products.

F6 · MEDIUM · XGB effective-area naming is internally inconsistent

Location: auxiliary_files.txt:20245-20248,20698; ANALYSIS.anasum_parallel_from_runlist.sh:133-145
The same tree contains both ...ModerateXGB...root and ...Moderate-XGB-BDT...root names;
the analysis wrapper resolves moderate2telXGB to NTel2-PointSource-Moderate-XGB-BDT.
Impact: Shell wrappers, cut files, and effective-area selection logic can disagree on which
product family is canonical.

F7 · MEDIUM · Maintainer README is stale and contradicts the transfer scripts

Location: Eventdisplay_AnalysisFiles_VTS/README.md:29-30,77-89; transfer/pack_irfs_for_ucla.sh:11; transfer/put_irfs_to_ucla.sh:27
The README documents a non-existent EVNDISP.reconstruction.runparameter, refers generically
to tar_packages, and instructs maintainers to move packages to archive before calling an
upload script that only scans tar_packages_${VERSION}/.
Impact: Release maintenance instructions are no longer self-consistent.

F8 · MEDIUM · download_irfs.sh continues into extract/remove after a failed transfer

Location: Eventdisplay_AnalysisFiles_VTS/download_irfs.sh:36-49
tar --keep-newer-files and rm -v execute immediately after the copy/bbftp call without
checking the transfer exit status.
Impact: Failed or partial downloads trigger extraction and cleanup, making recovery harder.

F9 · MEDIUM · copy_dispBDTs.sh and copy_dispXGB.sh silently overwrite SIMTYPE in a loop

Location: Eventdisplay_AnalysisFiles_VTS/DispBDTs/copy_dispBDTs.sh:11-19; DispXGB/copy_dispXGB.sh:11-18
Both scripts assign SIMTYPE repeatedly, so only the last hard-coded value survives.
copy_dispBDTs.sh also prints ${IRVERSION} instead of ${IRFVERSION}.
Impact: IRF-maintenance copies silently pull the wrong model family with misleading
provenance information.

F10 · LOW · V4/V5 gamma/hadron copy script has dead atmosphere-selection code

Location: Eventdisplay_AnalysisFiles_VTS/GammaHadronBDTs/copy_GammaHadron_V4_V5_BDTs.sh:21-31
Loop iterates over ATM21 ATM22 but the branch condition checks for ATM62; the
summer/winter split is never exercised.
Impact: Any future atmosphere-specific V4/V5 handling based on this template will be wrong.

F11 · LOW · Global DB fallback is present in text only, not in parsed configuration

Location: Eventdisplay_AnalysisFiles_VTS/ParameterFiles/EVNDISP.global.runparameter:9-16
Only the romulus.ucsc.edu line is active (*-prefixed); the remus.ucsc.edu fallback is
unstarred and ignored.
Impact: No real failover at the parameter-file level if the primary DB endpoint is
unavailable.

F12 · LOW · Backup content mixed into the main instrument-file tree

Location: auxiliary_files.txt:19298
A DispBDTs/AP.back subtree sits alongside nominal production content.
Impact: Ambiguity about which products are authoritative; risk of copying the wrong tree.


G — Release tests

G1 · HIGH · EventDisplay_Release_v492 is still a skeleton, not an auditable release-test record

  • Expected. v492 is in early development.

Location: EventDisplay_Release_v492/README.md:1-3
The checkout contains only metadata plus runparameter/V6.AP.runparameter.dat; none of the
expected result trees (Crab/, mc_data_comparison/, irf_plotting/, SourceTests/,
AP/radialAcceptances/) are present.
Impact: The v492 artifact does not provide the evidence needed to review the release tests
the codebase expects to exist.

G2 · HIGH · Release-test output naming is inconsistent

Location: EventDisplay_ReleaseTests_code/README.md:10-15; release_tests/montecarlo/energythresholds/plot.C:150-152
Most scripts use EventDisplay_Release_<version>, but the README points to
EventDisplay_ReleaseTests_v492 and the energy-threshold macro writes to
EventDisplay_ReleaseTests_<version>/energythresholds/.
Impact: Release products are scattered between differently named trees, making results look
missing and breaking documentation links.

G3 · HIGH · MC/data comparison applies the summer-atmosphere filter with an undefined variable

Location: EventDisplay_ReleaseTests_code/release_tests/montecarlo/mc_data_comparison/compareDatawithMC.sh:138-146
The winter branch checks ${atm}, but the summer branch checks ${A} before A is assigned.
Impact: Summer-epoch filtering silently uses the wrong atmosphere combinations, weakening one
of the main release-validation checks.

G4 · HIGH · Crab run-list generation: WOBBB typo leaves zero-wobble classification undefined

Location: EventDisplay_ReleaseTests_code/release_tests/sources/Crab/runlist_generator_from_anasum_log.sh:127-132,156-165
When both wobble offsets are zero, the script assigns WOBBB="0" instead of WOBB="0", but
later decisions key on WOBB.
Impact: Zero-wobble runs inherit a stale wobble state and are grouped into the wrong run
lists, contaminating downstream source-test summaries.

G5 · HIGH · Source-test plotting macros are pinned to the v491 library and data layout

Location: release_tests/sources/Crab/plot_lightcurves.C:19; plot_energy_spectra.C:20; plot_skymaps.C:17; plot_sensitivity.C:11,27-40; plot_sensitivity_compare_cuts.C:16,44-62
These macros hard-code /.../EventDisplay_v491-al9/lib/libVAnaSum.so and for sensitivity plots
hard-code v491/v490 result directories.
Impact: Running on v492 products loads the wrong code and compares against fixed historical
paths.

G6 · MEDIUM · IRF plotting is hard-coded to a single slice instead of broad parameter coverage

Location: EventDisplay_ReleaseTests_code/release_tests/montecarlo/irf_plotting/irf_plotting.sh:16-25,57,75-89
The driver hard-codes COMPAREVERSION="v492", a fixed sim/cut family, and only runs the body
for V6_2016_2017 / ATM61.
Impact: The nominal IRF plotting step no longer spans the stated parameter space and is weak
as a release-regression check.

G7 · MEDIUM · Shipped v492 run-parameter format encodes a fragile cut-name duality

Location: EventDisplay_Release_v492/runparameter/V6.AP.runparameter.dat:35-39; release_tests/sources/Crab/anasum_from_runlists.sh:34-40
Each * CUT line stores both a full IRF cut identifier and a short alias; consumers rely on the
fourth field specifically.
Impact: Easy for future scripts to read the wrong cut token and silently select the wrong
inputs.


H — Performance & efficiency

H1 · HIGH · evndisp keeps time-sliced pedestal analysis enabled during normal data reconstruction

Location: EventDisplay_v4/src/VEventLoop.cpp:490-495,1442-1461; EventDisplay_v4/src/VPedestalCalculator.cpp:30-199,280-381; logFiles/analysis/64080.log:71,598-599
On the exercised run, evndisp processes ~205 events/s (543 481 events in 44:04). The active
path still performs pedestal-variance bookkeeping for every telescope during standard
reconstruction: VPedestalCalculator::doAnalysis() loops over hit channels, recomputes trace
sums for every summation window, and fills per-pixel histograms.
Impact: A calibration-style side analysis remains in the hottest reconstruction loop and is a
plausible first-order contributor to the 16× throughput gap between evndisp (~205 ev/s) and
mscw_energy (~3 229 ev/s).

H2 · MEDIUM · evndisp repeats invariant low-gain/sum-window validation in the per-event loop

Location: EventDisplay_v4/src/VEventLoop.cpp:1204-1292
The analysis path rechecks availability of low-gain multipliers for sumwindow_1,
sumwindow_2, and sumwindow_pass1 on every event for every telescope, even though these are
run-level constants after initialisation.
Impact: Avoidable branch-heavy work repeated millions of times in long runs.

H3 · HIGH · anasum double-scans mscw.root and reinitialises TMVA twice per run

Location: EventDisplay_v4/src/VAnaSum.cpp:464-465; EventDisplay_v4/src/VStereoAnalysis.cpp:264-349,434-448; EventDisplay_v4/src/VTMVAEvaluator.cpp:366-511; logFiles/analysis/64080.anasum.log:97-103,237-243
In the reflected-background path, ON and OFF both point to the same mscw.root, yet VAnaSum
runs separate passes. Each pass reads 174 937 entries and rebuilds the gamma/hadron TMVA
readers from disk.
Impact: For every run, anasum roughly doubles tree I/O and repeats expensive TMVA and
effective-area initialisation that could be shared.

H4 · MEDIUM · anasum uses a linear TMVA-bin search with recomputed mean energies per event

Location: EventDisplay_v4/src/VTMVAEvaluator.cpp:714-745,802-839
Each gamma/hadron evaluation calls getDataBin(), which scans all loaded bins and recomputes
the mean energy of each candidate before evaluating the selected reader.
Impact: Avoidable per-event classifier overhead in the inner anasum loop, especially with
many energy/zenith bins.

H5 · MEDIUM · Multi-run anasum jobs scale as O(runs × total entries in chain)

Location: EventDisplay_v4/src/VAnaSum.cpp:464-465; EventDisplay_v4/src/VStereoAnalysis.cpp:434-450
fillHistograms() loops over all chain entries and then filters by runNumber == irun; each
run-specific pass re-reads entries from all other runs.
Impact: Runtime can grow toward O(runs × total_entries) instead of near-linear.

H6 · MEDIUM · evndisp reads pedestal calibration inputs redundantly during initialisation

Location: logFiles/analysis/64080.log:247-252,263-268,275-280,291-296
The same telescope-level pedestal products are read twice in succession during calibration setup.
Impact: Duplicated calibration I/O in the slowest stage of the data pipeline.

H7 · MEDIUM · mscw_energy repeatedly rebuilds TMVA datasets instead of amortising setup

Location: logFiles/analysis/64080.mscw.log:156-169; logFiles/irfs/20deg_0.5wob_NOISE200.mscw.log:261-277
Both data-side and MC-side logs contain repeated Rebuilding Dataset Default messages.
Impact: Avoidable setup churn, expensive when multiplied across large IRF productions.


Cross-cutting issues (confirmed from logs)

X1 · HIGH · v2dl3 silently sets quality flag to 0 when Eventdisplay logfile is missing

Location: logFiles/analysis/64080.v2dl3.log:14
v2dl3 reports "Eventdisplay logfile not found in anasum root file. Quality flag set to 0."
Impact: Downstream DL3 products lose quality metadata without a loud failure.

X2 · MEDIUM · anasum continues under missing TMVA-weight warnings

Location: logFiles/analysis/64080.anasum.log:243-249,541-547; EventDisplay_v4/src/VTMVAEvaluator.cpp:190-279
VTMVAEvaluator reports "TMVA root file not found or incomplete file (0)" and continues;
see also B4.
Impact: Incomplete classifier inputs survive initialisation silently.

X3 · MEDIUM · anasum reinitialises the same TMVA state twice, doubling warning noise

Location: logFiles/analysis/64080.anasum.log:237-247,535-545
See also H3. The same missing/incomplete low-energy TMVA file warning is emitted once for ON
and once again for OFF.
Impact: Duplicated warnings make it harder to distinguish a new problem from a repeated one.

X4 · LOW · Data run contains large numbers of GPS-faulted events; pipeline continues without escalation

Location: logFiles/analysis/64080.log:565-580,587-596
Missing telescope events and GPS faults on telescopes 1 and 4 are logged and skipped.
Impact: Run-quality issues are not escalated; pipeline favours "continue with warnings" over
explicit run-quality gating.


Hyperparameter locations (reference)

Model Location
TMVA gamma/hadron BDT Eventdisplay_AnalysisFiles_VTS/ParameterFiles/TMVA.BDT.runparameter:13-15
TMVA DispBDT Eventdisplay_AnalysisFiles_VTS/ParameterFiles/TMVA.BDTDisp.runparameter:1-2
XGB regression/classification Eventdisplay-ML/src/eventdisplay_ml/hyper_parameters.py:9-45 (hard-coded in Python)
XGB binning / style Eventdisplay_AnalysisFiles_VTS/ParameterFiles/XGB-stereo-parameter.json; XGB-classify-parameter.json

Priority action list

Immediate (HIGH severity, silent wrong-physics or silent job failure)

  1. Fix ANALYSIS.anasum.sh SUBC initialisation (anasum.sh:90-92) — jobs are silently never submitted (C1)
  2. Fix get_disp_dir() to check HVSETTINGS == "obsLowHV" instead of undefined SIMTYPE (mscw_energy_sub.sh:136-152) — wrong DispBDT directory for all RedHV/UV runs at high zenith (C2)
  3. Fix shower-core yimp validity check (VArrayAnalyzer.cpp:944-951) — NaN core-Y propagates into reconstruction (A6)
  4. Fix method-4 pair-angle rejection variable (VArrayAnalyzer.cpp:1241-1248) — wrong telescope pairs used in cut (A7)
  5. Fix low-gain T0 assignment and calculation (VCalibrationData.cpp:365-373; VCalibrator.cpp:899-903) — high-gain state corrupted (A4, A5)
  6. Fix core-resolution formula (VInstrumentResponseFunctionData.cpp:472-485) — Xcore used twice, Ycore ignored (A12)
  7. Fix TMVA zenith-bin off-by-one (trainTMVAforGammaHadronSeparation.cpp:255-266) — out-of-bounds vector access (A13)
  8. Fix TMVA training factory type for reconstruction-quality path (trainTMVAforGammaHadronSeparation.cpp:343-360) — classification factory used instead of regression (A14)
  9. Fix IRF job-launch filenames (IRF.generate_lookup_table_parts.sh:86-124; IRF.mscw_energy_MC.sh:97-139) — submitted path does not exist (D4)
  10. Fix IRF.production.sh double-loop and epoch variable (IRF.production.sh:337-387,353-366) — TMVA grid submitted multiple times with wrong epoch variable (D1, D2)
  11. Fix IRF.mscw_energy_MC_sub.sh quoted glob and stale XGB CLI (IRF.mscw_energy_MC_sub.sh:133-136,237-271) — files not copied; Python CLI mismatch (D3, E2)
  12. Propagate FORCEDATMO via sed template (ANALYSIS.mscw_energy.sh:63) — ATM override is a no-op in batch (C3)
  13. Add atmosphere-ID normalisation for V4/V5 in mscw_energy_sub.sh consistent with prepare_atmo_string() — table lookup fails for V4/V5 GRISU data (C4)
  14. Fix ANALYSIS.evndisp_sub.sh DBTEXT file existence checks — missing files silently inject wrong metadata (C9)
  15. Fix broken TMVA bin-parsing regex (IRF.trainTMVAforGammaHadronSeparation.sh:107-109) — bin parsing fails (C10)
  16. Fix sliding-window FADC out-of-bounds read (VTraceHandler.cpp:783) — FADC[n] accessed on every event where the window reaches the rightmost search position; UB corrupts charge reconstruction (A17)
  17. Fix VOnOff::doOnOffforParameterHistograms background mutation (VOnOff.cpp:159-161) — hoff->Scale after Add double-scales background, corrupting reflected-region excess counts and significance in anasum (B11)
  18. Fix VInstrumentResponseFunctionRunParameter silent key drop (VInstrumentResponseFunctionRunParameter.cpp:109-358) — add a terminal else { cerr << "unknown key: " << temp << endl; } to catch parameter typos (B13)
  19. Fix printRunParameter exit code on file-open failure (printRunParameter.cpp:330-335) — replace exit(0) with exit(EXIT_FAILURE) so calling shell scripts detect errors (B14)

Short-term (MEDIUM / HIGH, reproducibility and efficiency)

  1. Gate time-sliced pedestal analysis in the standard evndisp analysis loop — primary throughput bottleneck (H1)
  2. Avoid double mscw.root scan and double TMVA initialisation in anasum — doubles tree I/O (H3)
  3. Fix Eventdisplay-ML train/apply feature mismatch for tmva_style=True classification (E5)
  4. Fix config.py .get() on listAttributeError on valid parameter files (E6)
  5. Wire ANALYSIS.dispXGB.sh into the default anasum workflow, or add a guard that prevents *-XGB-BDT cut selection without XGB sidecars (E1)
  6. Fix XGB stereo training RecID hard-coding (IRF.trainXGBforAngularReconstructionBinned.sh:48-56) (E3)
  7. Gate XGB friend-file probing in CData to the active reconstruction methods (E4)
  8. Synchronise epoch file, download manifests, and downloadable IRF tarballs (F1, F2, F3)
  9. Add end-to-end IRF → analysis consistency test — shell and C++ interface drift surface only in production
  10. Fix makeEffectiveArea "no problem" comment vs. exit(EXIT_FAILURE) mismatch (makeEffectiveArea.cpp:118; VInstrumentResponseFunctionRunParameter.cpp:437) — either remove the misleading comment or implement graceful fallback (B12)
  11. Fix VTableLookupRunParameter error message (VTableLookupRunParameter.cpp:175) — change "2=read" to "0=read" (B15)
  12. Fix VTraceHandler::getQuickMax tmax initialisation (VTraceHandler.cpp:420-426) — initialise tmax to 0 or the expected maximum before computing nMax (A18)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions