Issue Report
(used GPT-5.4 and Opus 4.7)
Scope: Full pipeline audit across EventDisplay_v4 (C++), Eventdisplay_AnalysisScripts_VTS
(shell orchestration), Eventdisplay_AnalysisFiles_VTS (auxiliary files), and Eventdisplay-ML
(XGB/Python ML). Log files and release-test artifacts were also reviewed.
Severity labels: HIGH — incorrect physics output or silent job failure; MEDIUM —
degraded reliability, reproducibility, or correctness in edge cases; LOW — cosmetic,
misleading diagnostics, or minor fragility.
Pipeline overview (reference)
| Stage |
IRF chain entry |
Data-analysis entry |
| Image reconstruction |
IRF.evndisp_MC.sh → evndisp |
ANALYSIS.evndisp.sh → evndisp |
| Stereo / table lookup |
IRF.mscw_energy_MC.sh → mscw_energy |
ANALYSIS.mscw_energy.sh → mscw_energy |
| Angular reco (ML) |
DispBDT / XGB training (IRF.train*) |
DispBDT XMLs consumed by mscw_energy |
| Gamma/hadron (ML) |
BDT / XGB training, optimisation, effective areas |
TMVA/XGB friend trees consumed by anasum |
| Science analysis |
— |
ANALYSIS.anasum*.sh → anasum |
| DL3 export |
— |
ANALYSIS.v2dl3.sh → v2dl3 |
C++ class chains:
evndisp: VReadRunParameter → VEventLoop → VTraceHandler/VFitTraceHandler, VCalibrator, VImageAnalyzer, VArrayAnalyzer, VPedestalCalculator
mscw_energy: VTableLookupRunParameter → VTableLookup → CData, VDispAnalyzer, VTMVADispAnalyzer, VEmissionHeightCalculator
anasum: VAnaSumRunParameter → VAnaSum → VStereoAnalysis → CData, VGammaHadronCuts, VTMVAEvaluator
makeEffectiveArea: VGammaHadronCuts → CData → VInstrumentResponseFunction → VInstrumentResponseFunctionData
A — C++ algorithm defects
A1 · HIGH · Out-of-bounds read in pulse-timing extraction
Location: EventDisplay_v4/src/VTraceHandler.cpp:346-358
getPulseTiming() interpolates on the rising edge using fpTrace[i+1]. When the pulse maximum
lands on the last sample (maxpos = size-1), i+1 is out of bounds.
Impact: Truncated or late pulses produce undefined timing values in the signal characterisation
path.
A2 · HIGH · Sliding-window charge branch returns FADC[1] instead of integrated charge
Location: EventDisplay_v4/src/VTraceHandler.cpp:745-757
calculateTraceSum_slidingWindow() computes charge and updates the integration-window metadata
but then returns FADC[1].
Impact: When the raw sliding-window branch is active, charge extraction is silently wrong.
A3 · MEDIUM · Fitted-trace T0 mixes pedestal-subtracted and absolute amplitude conventions
Location: EventDisplay_v4/src/VFitTraceHandler.cpp:433-442
getTraceMax() returns a pedestal-shifted peak, but getTraceTZero() solves for
-(imax/2 + fPed), re-adding the pedestal to the half-height target.
Impact: Fitted pulse-arrival times are biased whenever the pedestal is non-zero.
A4 · HIGH · Low-gain average-T0 assignment always writes the high-gain field
Location: EventDisplay_v4/src/VCalibrationData.cpp:365-373
setAverageTZero(bool iLowGain) unconditionally assigns fAverageTZero_highgain, even when
iLowGain == true.
Impact: The low-gain timing calibration path corrupts the high-gain average-T0 used later in
analysis.
A5 · HIGH · Low-gain average-T0 calculation uses the high-gain sum path
Location: EventDisplay_v4/src/VCalibrator.cpp:899-903
calculateAverageTZero(bool iLowGain) calls calcSums(..., iLowGainOnly=false) unconditionally.
Impact: Low-gain timing constants are derived from the wrong charge-extraction mode.
A6 · HIGH · Shower-core validity check tests ximp twice, never yimp
Location: EventDisplay_v4/src/VArrayAnalyzer.cpp:944-951
fillShowerCore() evaluates !isnormal(ximp) || !isnormal(ximp); the duplicate check means an
invalid yimp is never screened out.
Impact: NaN or invalid core-Y solutions propagate into core distances, energy reconstruction
inputs, and event-selection variables.
A7 · HIGH · Method-4 stereo pair-angle rejection applies cut to the wrong telescope pair
Location: EventDisplay_v4/src/VArrayAnalyzer.cpp:1241-1248
The per-pair rejection computes the current pair in mm[0]/mm[1] but evaluates the cut using
m[0]/m[1].
Impact: For events with more than two images, pair-angle rejection is applied to the wrong
telescopes, biasing the weighted line intersections used for direction reconstruction.
A8 · HIGH · Log-likelihood image fit can be initialised with invalid centroid bounds
Location: EventDisplay_v4/src/VImageParameterCalculation.cpp:1542-1555,1588-1601
For edge/degenerate images with sigmaX <= 0 or sigmaY <= 0, only the upper Minuit bound is
set; the lower bound retains its sentinel value before being passed to the minimiser.
Impact: Edge images can enter the recovery LL fit with invalid parameter bounds, degrading
parameterisation in the regime where the fit is intended to stabilise results.
A9 · MEDIUM · Muon-ring image-size correction divides by zero at the first grid point
Location: EventDisplay_v4/src/VImageParameterCalculation.cpp:737-748
xi_tmp = i / numSteps is zero when i == 0, but kTest_tmp = ... / xi_tmp is evaluated before
any guard.
Impact: Muon calibration correction can produce inf/NaN candidates and an unstable
correction factor.
A10 · MEDIUM · Emission-height pair weights undefined for small image sizes
Location: EventDisplay_v4/src/VEmissionHeightCalculator.cpp:60-64
The weight formula 1./((1./log10(size[i])) + (1./log10(size[j]))) is not guarded against
size <= 1.
Impact: Small-image events receive singular, negative, or ill-defined pair weights in the
emission-height estimate.
A11 · MEDIUM · VEmissionHeightCalculator accumulates telescope positions across reinitalisations
Location: EventDisplay_v4/src/VEmissionHeightCalculator.cpp:100-133
Both setTelescopePositions() overloads only push_back() and never clear previous contents.
Impact: Reusing the same calculator instance duplicates telescope geometry, making pairwise
shower-height geometry wrong.
A12 · HIGH · Core-resolution IRF formula uses Xcore twice instead of Xcore and Ycore
Location: EventDisplay_v4/src/VInstrumentResponseFunctionData.cpp:472-485
The core-resolution error is computed as sqrt(Xcore²+Xcore²) - sqrt(MCxcore²+MCycore²).
Impact: Reconstructed core-error quantities are biased whenever Ycore differs from Xcore;
this is a direct physics error in the IRF-resolution path.
A13 · HIGH · Gamma/hadron TMVA training has an off-by-one zenith-bin bounds check
Location: EventDisplay_v4/src/trainTMVAforGammaHadronSeparation.cpp:255-266
The check uses < iZenithBin instead of <= iZenithBin, so iZenithBin == size() passes and is
then dereferenced.
Impact: Certain bin configurations index beyond the configured zenith-bin vectors during
training.
A14 · HIGH · Reconstruction-quality TMVA training configured as a classification factory
Location: EventDisplay_v4/src/trainTMVAforGammaHadronSeparation.cpp:343-360
The non-gamma/hadron branch adds a regression target but creates the TMVA::Factory with the
default (classification) options string.
Impact: The code path described as reconstruction-quality training is algorithmically
mismatched to the TMVA factory configuration.
A15 · MEDIUM · Angular-reconstruction training silently rescales the requested train/test split
Location: EventDisplay_v4/src/trainTMVAforAngularReconstruction.cpp:125-138
After deriving ntrain and ntest from the requested fraction, both are multiplied by 0.8.
Impact: Effective train/test statistics differ from the configured values, reducing
reproducibility.
A16 · MEDIUM · Reduced-tree subsampling normalised by file count, not event population
Location: EventDisplay_v4/src/trainTMVAforGammaHadronSeparation.cpp:147-157
The retained fraction is divided by iTreeVector.size() (number of input files), not by event
counts.
Impact: Training preselection depends on file granularity and can distort class composition
across uneven file splits.
A17 · HIGH · Sliding-window charge extraction reads one past the end of the FADC array
Location: EventDisplay_v4/src/VTraceHandler.cpp:725-783
iSearchEnd is clamped to n - iIntegrationWindow + 1. At the final loop iteration
(i = iSearchEnd - 1), the expression FADC[i + iIntegrationWindow] evaluates to FADC[n],
which is one past the end of the valid array range [0, n-1].
Impact: Undefined behaviour on every waveform where the sliding window reaches its rightmost
position; corrupts charge reconstruction silently on affected events.
A18 · MEDIUM · VTraceHandler::getQuickMax uses the previous call's trace maximum for saturation counting
Location: EventDisplay_v4/src/VTraceHandler.cpp:420-426
nMax is computed as (int)(fDynamicRange * tmax) before tmax is reset to −10000. If the
caller passes an uninitialised or stale value, the saturation counter n255 is thresholded
against the wrong maximum.
Impact: The number of saturated samples can be over- or under-estimated, misclassifying
pulses as saturated or unsaturated and biasing image-cleaning amplitude thresholds.
B — C++ infrastructure defects
B1 · MEDIUM · CData::getXGBTree() assumes .root is present in the filename
Location: EventDisplay_v4/src/CData.cpp:1073-1086
The XGB friend-file name is rewritten via string::replace(find(".root"), ...) without checking
whether .root exists in the name.
Impact: The class API is brittle and can generate an invalid replacement if reused with a
non-standard input name.
B2 · MEDIUM · XGB gamma/hadron C++ interface hard-wired to Is_Gamma_70
Location: EventDisplay_v4/src/CData.cpp:1118-1127; EventDisplay_v4/src/VGammaHadronCuts.cpp:727-731,1118-1128
The C++ consumer binds only Gamma_Prediction and Is_Gamma_70; the classification operating
point is baked into the upstream friend-tree producer, not enforced by VGammaHadronCuts.
Impact: The 70% efficiency threshold cannot be adjusted from cut files or runtime parameters,
discarding all the optimisation flexibility of the XGB classifier.
B3 · MEDIUM · VTMVAEvaluator recovers feature names by brittle XML string slicing
Location: EventDisplay_v4/src/VTMVAEvaluator.cpp:62-116
Variable recovery depends on exact XML formatting and field ordering (NVar, Expression,
Label, SpecIndex) rather than a structured parser.
Impact: A TMVA XML layout change can silently alter which variables the evaluator reads,
turning the deployed BDT interface into a schema-fragile contract.
B4 · MEDIUM · TMVA weight grid initialisation tolerates missing interior bins with warnings
Location: EventDisplay_v4/src/VTMVAEvaluator.cpp:190-279
Missing weight files at non-edge bins emit warnings and allow the evaluator to continue.
Impact: Incomplete TMVA weight grids survive initialisation, making production behaviour
depend on partial bin coverage instead of a strict consistency check.
B5 · LOW · VTMVARunData sanity message does not match the actual condition
Location: EventDisplay_v4/src/VTMVARunData.cpp:795-800
Prints "need at least two energy bins" while the code only rejects < 1 bin.
Impact: Misleading diagnostics when debugging training-configuration problems.
B6 · MEDIUM · VInstrumentResponseFunction mutates shared MC azimuth in place
Location: EventDisplay_v4/src/VInstrumentResponseFunction.cpp:193-199
MC azimuth normalisation is applied as fData->MCaz -= 360. on the shared event object.
Impact: Weak implementation pattern; mutates shared state instead of using a local normalised
copy.
B7 · MEDIUM · makeEffectiveArea IRF duplication is order-dependent
Location: EventDisplay_v4/src/makeEffectiveArea.cpp:162-208,194-208
IRF products are duplicated by index via getDuplicationID() and only work if the source has
already been initialised earlier in the vector.
Impact: A future refactor changing the internal IRF ordering will silently break resolution
product generation.
B8 · MEDIUM · copyMCHistograms() keeps the first ROOT file open
Location: EventDisplay_v4/src/makeEffectiveArea.cpp:370-401
The first MChistos object is borrowed directly from the first input file, which is never closed
inside the helper.
Impact: File-handle lifetime is coupled to the returned object, risking resource leaks or
dangling-file state.
B9 · MEDIUM · anasum contains a stale unreachable analysisType == 5 branch
Location: EventDisplay_v4/src/anasum.cpp:47-54,106-108
analysisType is explicitly limited to 3 and 4, but the VAnaSum::doStereoAnalysis() handoff
still tests analysisType == 5.
Impact: Dead code path; sign of an incomplete control-flow change.
B10 · MEDIUM · Epoch reevaluation at runtime means analysis behaviour changes with epoch-file updates
Location: EventDisplay_v4/src/VReadRunParameter.cpp:463-470; EventDisplay_v4/src/printRunParameter.cpp:100-117
printRunParameter updated-runinfo and evndisp -epochfile reevaluate epoch and atmosphere from
VERITAS.Epochs.runparameter each run rather than relying solely on the file-internal metadata.
Impact: Reprocessing the same intermediate ROOT file with an updated epoch file can silently
change the science output.
B11 · HIGH · VOnOff mutates the background histogram after using it, corrupting reflected-region normalisation
Location: EventDisplay_v4/src/VOnOff.cpp:159-161
doOnOffforParameterHistograms computes the excess as
hTemp->Add(hon, hoff, 1., -1.*i_norm_alpha) (which consumes hoff at its original scale) and
then immediately calls hoff->Scale(i_norm_alpha). Any caller that reuses hoff after this call
sees a rescaled background histogram.
Impact: The reflected-background normalisation in the anasum analysis path (via VAnaSum.cpp:556)
can be doubly-scaled, producing wrong excess counts and significance values.
B12 · HIGH · makeEffectiveArea comments that the MC run header is optional but hard-exits when it is missing
Location: EventDisplay_v4/src/makeEffectiveArea.cpp:118; EventDisplay_v4/src/VInstrumentResponseFunctionRunParameter.cpp:430-437
The comment at line 118 states "might not be there, no problem", yet readMCRunHeader() calls
exit(EXIT_FAILURE) when no MC_runheader object is found in the input file.
Impact: Misleading documentation causes operators to expect graceful handling; actual behaviour
is a hard crash with a non-zero exit code when the MC header is absent (e.g., for older files).
B13 · HIGH · VInstrumentResponseFunctionRunParameter silently ignores unknown run-parameter keys
Location: EventDisplay_v4/src/VInstrumentResponseFunctionRunParameter.cpp:109-358
The parser is a chain of else if branches ending at RERUN_STEREO_RECONSTRUCTION_3TEL with
no terminal else and no warning message. Unknown or misspelled keys are silently dropped.
Impact: Typos in makeEffectiveArea parameter files produce no diagnostic and default values
are used; an operator has no way to detect that a setting was not applied.
B14 · MEDIUM · printRunParameter exits with code 0 on file-open failure
Location: EventDisplay_v4/src/printRunParameter.cpp:330-335
new TFile(argv[1]) can return a zombie object; the code prints "exiting..." and calls exit(0).
Impact: Shell wrappers that check only the exit code (C6, several IRF helpers) will interpret
the empty output as a missing field rather than a fatal error, silently propagating bad metadata
through the pipeline.
B15 · LOW · VTableLookupRunParameter error message says "2=read" but the code only accepts "0=read"
Location: EventDisplay_v4/src/VTableLookupRunParameter.cpp:162-176
The -fill option accepts 0 (read) or 1 (fill); any other value triggers the message
"unknown parameter, choose 1=fill or 2=read lookup tables", which is incorrect.
Impact: Operators following the error message will pass -fill=2, which also triggers the
error in a confusing loop.
C — Shell orchestration bugs
C1 · HIGH · ANALYSIS.anasum.sh never submits any jobs
Location: Eventdisplay_AnalysisScripts_VTS/scripts/ANALYSIS.anasum.sh:90-92
Line 90 assigns the helper path as a string; the subsequent eval only echoes it back. Every
other equivalent script correctly uses backticks to execute the helper:
# Broken (line 90):
SUBC="$(dirname "$0")/helper_scripts/UTILITY.readSubmissionCommand.sh"
SUBC=`eval "echo \"$SUBC\""` # still a string, never executed
# Correct (evndisp.sh:126):
SUBC=`$(dirname "$0")/helper_scripts/UTILITY.readSubmissionCommand.sh`
All *qsub*/*condor*/*sbatch*/*simple* tests fail, so generated job scripts are written
but never submitted.
Impact: The full-runlist anasum path silently drops all jobs with no error message.
C2 · HIGH · get_disp_dir() checks an undefined variable, always selects the wrong DispBDT directory
Location: Eventdisplay_AnalysisScripts_VTS/scripts/helper_scripts/ANALYSIS.mscw_energy_sub.sh:136-152
The function checks ${SIMTYPE} == "CARE_RedHV" to decide between 55-deg and 60-deg DispBDT
fallback models. SIMTYPE is never assigned in this script; only SIMTYPE_RUN is set (lines
82–93). A secondary issue: the current value since v491 is CARE_RedHV_Feb2024, not
CARE_RedHV, so the comparison would fail even if the variable were correct. The proper guard
should use HVSETTINGS == "obsLowHV".
Impact: For all RedHV (and high-zenith UV) runs, mscw_energy loads 60-deg DispBDT models
instead of the 55-deg fallback, degrading direction reconstruction quality for runs at zenith ≥ 58°.
C3 · MEDIUM · FORCEDATMO atmosphere override is silently ignored in cluster submissions
Location: Eventdisplay_AnalysisScripts_VTS/scripts/ANALYSIS.mscw_energy.sh:63; helper_scripts/ANALYSIS.mscw_energy_sub.sh:71
The parent script captures FORCEDATMO=$6 locally. The sed-template substitution block does not
inject FORCEDATMO into the generated job script, and the job script references
${FORCEDATMO:-...} as an unexported shell variable, which is empty in batch environments.
Impact: The documented [ATM] override parameter does nothing in production; users
attempting to force an atmosphere ID get silently wrong results.
C4 · MEDIUM · mscw_energy_sub.sh missing atmosphere-ID normalisation for V4/V5 data
Location: Eventdisplay_AnalysisScripts_VTS/scripts/helper_scripts/ANALYSIS.mscw_energy_sub.sh:68-71; ANALYSIS.anasum_sub.sh:74-83
mscw_energy_sub.sh calls printRunParameter … updated-runinfo, which may return a modernised
6x atmosphere ID for old V4/V5 observations. anasum_sub.sh compensates with
prepare_atmo_string() (${ATMO/6/2} for V4/V5); mscw_energy_sub.sh has no equivalent
conversion.
Impact: mscw_energy exits with "table file not found" for V4/V5 GRISU data when the epoch
files return a modernised atmosphere ID.
C5 · MEDIUM · DISPBDT guard in mscw_energy_sub.sh is missing $
Location: Eventdisplay_AnalysisScripts_VTS/scripts/helper_scripts/ANALYSIS.mscw_energy_sub.sh:193
if [[ DISPBDT != "NOTSET" ]]; then references the literal string DISPBDT instead of
$DISPBDT, so the condition is never conditionally guarded.
Impact: The log reports DispBDT XML provenance even when that branch is not actually active.
C6 · MEDIUM · printRunParameter -elevation field is parsed positionally in the helper
Location: Eventdisplay_AnalysisScripts_VTS/scripts/helper_scripts/ANALYSIS.mscw_energy_sub.sh:127; EventDisplay_v4/src/printRunParameter.cpp:340-343
The helper extracts the zenith angle by awk field number from a human-readable CLI printout
rather than from a stable machine-readable interface.
Impact: Fragile table and DispBDT directory selection; breaks silently if the printout format
changes.
C7 · MEDIUM · mscw_energy hard-fails when a TMVA XML family is missing, no soft fallback
Location: EventDisplay_v4/src/VTMVADispAnalyzer.cpp:97-105,164-170
When TMVA XML prefixes are provided and one family is missing, the code exits without graceful
degradation.
Impact: Missing or incompletely copied DispBDT XMLs stop the run.
C8 · MEDIUM · anasum default input directory hard-wired to RecID0 with no consistency check
Location: Eventdisplay_AnalysisScripts_VTS/scripts/ANALYSIS.anasum.sh:47-49; helper_scripts/ANALYSIS.anasum_sub.sh:213-226
The default input directory is .../RecID0; no validation that the mscw.root files were
produced with the requested reconstruction ID.
Impact: A workflow that reconstructs with a non-zero RecID can run anasum against stale
RecID0 products, producing plausible but inconsistent science outputs.
C9 · HIGH · ANALYSIS.evndisp_sub.sh consumes DBTEXT-derived files without existence checks
Location: Eventdisplay_AnalysisScripts_VTS/scripts/helper_scripts/ANALYSIS.evndisp_sub.sh:97-107,137-166
After unpacking the DBTEXT tarball, the helper immediately cats ${TMP_LASERRUN} and parses
${RUN}.runinfo without checking that either file exists or is readable.
Impact: Missing or corrupt DBTEXT tarballs silently propagate empty laser-run metadata or
wrong run dates into CVBF lookup and calibration selection.
C10 · HIGH · Broken TMVA bin-parsing regex in training and optimisation scripts
Location: Eventdisplay_AnalysisScripts_VTS/scripts/IRF.trainTMVAforGammaHadronSeparation.sh:107-109; IRF.optimizeTMVAforGammaHadronSeparation.sh:91-93
grep "^* ENERGYBINS" and the matching sed expressions are invalid or broken (* is not
a valid regex anchor in this position).
Impact: Training and optimisation bin parsing can fail or misbehave, directly affecting
gamma/hadron separation products.
C11 · MEDIUM · Cut optimisation continues with missing effective-area input
Location: Eventdisplay_AnalysisScripts_VTS/scripts/IRF.optimizeTMVAforGammaHadronSeparation.sh:81-83
Logs an error on missing effective-area file but does not exit.
Impact: Optimisation jobs may run with incomplete inputs, producing invalid cut tuning.
C12 · MEDIUM · DL3 jobs install Python dependencies at runtime per job
Location: Eventdisplay_AnalysisScripts_VTS/scripts/helper_scripts/ANALYSIS.v2dl3_sub.sh:56-58
Each job activates the base environment and runs pip install -e ....
Impact: Slow, non-reproducible, and environment-dependent production.
C13 · MEDIUM · ANALYSIS.anasum_combine.sh lacks a completion/integrity guard before merge
Location: Eventdisplay_AnalysisScripts_VTS/scripts/ANALYSIS.anasum_combine.sh:31-36,85-115
Assumes parallel jobs are finished; no strong completeness validation before combining.
Impact: Partial combined outputs can be produced silently.
C14 · MEDIUM · Data-side mscw_energy selects IRF products purely from filename templates
Location: Eventdisplay_AnalysisScripts_VTS/scripts/helper_scripts/ANALYSIS.mscw_energy_sub.sh:120-179,182-188
The helper derives the IRF table path and DispBDT directory from printRunParameter output and
string templates, then runs mscw_energy without verifying that the chosen XMLs and tables
match the input evndisp.root configuration.
Impact: Stale or mismatched IRF products can be applied to otherwise valid evndisp files,
yielding apparently normal mscw.root outputs built with the wrong reconstruction tables.
C15 · MEDIUM · DBTEXT run extraction is fragile against schema drift in .runinfo files
Location: Eventdisplay_AnalysisScripts_VTS/scripts/db_scripts/db_run.sh:69-115,165-185
db_run.sh re-parses .runinfo headers by name; if a header is absent, the helper falls back
to empty or stale column indexes.
Impact: Downstream scripts can derive wrong time windows, source IDs, and DB queries from
superficially valid DBTEXT files.
C16 · MEDIUM · Existing DBTEXT tarballs suppress regeneration of laser-calibration side files
Location: Eventdisplay_AnalysisScripts_VTS/scripts/db_scripts/db_run.sh:294-297
If <run>.tar.gz exists, the script skips read_laser_calibration entirely rather than
checking whether .laserrun, .gain, .toffset, and related files are current inside the
tarball.
Impact: Stale or incomplete DBTEXT tarballs freeze outdated calibration metadata and prevent
later evndisp analyses from refreshing it.
C17 · MEDIUM · FITS converter crashes on missing DQM-like DBTEXT files before the None check
Location: Eventdisplay_AnalysisScripts_VTS/scripts/db_scripts/db_write_fits.py:425-433
convert_table_comment_to_ascii(table) is called for dqm files before checking
if table is not None.
Impact: One missing DBTEXT component aborts FITS generation and drops all DQM metadata for
the run.
C18 · MEDIUM · DQM FITS combination requires identical schemas across all runs
Location: Eventdisplay_AnalysisScripts_VTS/scripts/db_scripts/db_combine_dqm_fits.py:40-54
Uses vstack(..., join_type="exact"), so any column drift makes the whole merge fail.
Impact: Long-run DQM aggregation is brittle against minor schema evolution.
C19 · MEDIUM · DQM rate/dead-time extraction does not guard malformed scaler intervals
Location: Eventdisplay_AnalysisScripts_VTS/scripts/db_scripts/db_write_fits.py:101-138
np.diff() and scaler divisions are performed with no protection against zero time_diffs or
zero TenMHzScaler increments.
Impact: Invalid scaler records can inject inf/NaN values into exported DQM quantities.
C20 · LOW · ANALYSIS.v2dl3.sh has typo LOGIDR and undefined variable AFILE
Location: Eventdisplay_AnalysisScripts_VTS/scripts/ANALYSIS.v2dl3.sh:53-58,93-106,117-122
Impact: Stale temp files and misleading job logs.
C21 · LOW · ANALYSIS.anasum_allcuts.sh uses relative ./ANALYSIS... calls
Location: Eventdisplay_AnalysisScripts_VTS/scripts/ANALYSIS.anasum_allcuts.sh:66-76
Impact: Fails unless launched from the scripts directory.
C22 · LOW · IRF packaging script has misspelled function name and undefined $F in logging
Location: Eventdisplay_AnalysisFiles_VTS/transfer/pack_irfs_for_ucla.sh:161-185
Impact: Fragile packaging and misleading archive diagnostics.
D — IRF production chain
D1 · HIGH · IRF.production.sh can resubmit the full TMVA grid multiple times
Location: Eventdisplay_AnalysisScripts_VTS/scripts/IRF.production.sh:337-387
The TRAINTMVA/OPTIMIZETMVA block contains inner loops that re-iterate over the full EPOCH
and ATMOS sets while already inside outer loops over the same sets.
Impact: TMVA training and optimisation jobs are submitted len(EPOCH) × len(ATMOS) times,
wasting batch capacity and mixing results.
D2 · HIGH · Inner TMVA block uses ${EPOCH:0:2} instead of ${VX:0:2} for epoch selection
Location: Eventdisplay_AnalysisScripts_VTS/scripts/IRF.production.sh:353-366
Inside the for VX in $EPOCH loop, size-cut file grep and TMVA runparameter selection still use
the outer ${EPOCH:0:2} instead of the current iteration variable ${VX:0:2}.
Impact: Multi-epoch submissions select the wrong size-cut row and TMVA runparameter template
for all epochs beyond the first.
D3 · HIGH · IRF.mscw_energy_MC_sub.sh quoted glob fails to copy uncompressed ROOT files
Location: Eventdisplay_AnalysisScripts_VTS/scripts/helper_scripts/IRF.mscw_energy_MC_sub.sh:133-136
cp -v "${INDIR}/*[0-9].root" "$DDIR" — the glob is inside double quotes and is not expanded
by the shell.
Impact: If upstream evndisp products are uncompressed .root files, the MC mscw_energy
stage fails before building its file list.
D4 · HIGH · Job-launch scripts generate one filename and submit another
Location: Eventdisplay_AnalysisScripts_VTS/scripts/IRF.generate_lookup_table_parts.sh:86-124; IRF.mscw_energy_MC.sh:97-139
Both scripts write the runnable script to $FSCRIPT but submit $FSCRIPT.sh, pointing to a
non-existent pathname.
Impact: Core IRF production stages — lookup-table and MC mscw_energy generation — fail
before creating their outputs, blocking all downstream steps.
D5 · HIGH · Gamma/hadron TMVA training ignores the documented UUID argument
Location: Eventdisplay_AnalysisScripts_VTS/scripts/IRF.trainTMVAforGammaHadronSeparation.sh:18-19,46,60
The help text places the optional UUID as argument 8, but the script reads it from $12.
Impact: Repeated training campaigns reuse the same submit/log directory, mixing products and
overwriting job state.
D6 · MEDIUM · Lookup-table combine checks the wrong path before overwriting
Location: Eventdisplay_AnalysisScripts_VTS/scripts/IRF.combine_lookup_table_parts.sh:77-80
Guard tests [[ -f $OFILE ]] instead of [[ -f "$ODIR/$OFILE" ]].
Impact: Existing combined lookup tables in the output directory are not reliably detected
before overwrite or merge attempts.
D7 · MEDIUM · zstd availability checks are malformed in several IRF helpers
Location: IRF.compress_evndisp_MC_sub.sh:46-54; IRF.evndisp_MC_sub.sh:323-330; IRF.lookup_table_parallel_sub.sh:67-79; IRF.mscw_energy_MC_sub.sh:137-149
command -v zstd /dev/null is not the standard existence test.
Impact: Compression/decompression branches can report false missing-tool failures.
D8 · MEDIUM · IRF.evndisp_MC_sub.sh passes literal glob to zstd when no files match
Location: Eventdisplay_AnalysisScripts_VTS/scripts/helper_scripts/IRF.evndisp_MC_sub.sh:88-103
The loop runs over "$V4N"/*.root.zst without checking whether any files match; the literal
pattern is then passed to zstd.
Impact: Spurious decompression failures during duplicate-processing detection.
D9 · MEDIUM · IRF.optimizeTMVAforGammaHadronSeparation_sub.sh uses MVADIR before definition
Location: Eventdisplay_AnalysisScripts_VTS/scripts/helper_scripts/IRF.optimizeTMVAforGammaHadronSeparation_sub.sh:54-69
rm -f ${MVADIR}/rates.log executes before MVADIR is assigned later in the file.
Impact: The cleanup step targets the wrong directory, potentially using an inherited
environment value.
D10 · MEDIUM · IRF.production.sh mixes robust $(dirname "$0")/... calls with fragile ./IRF... calls
Location: Eventdisplay_AnalysisScripts_VTS/scripts/IRF.production.sh:311,319,330,372,379
Impact: IRF production fails if launched from outside the scripts directory.
D11 · MEDIUM · makeEffectiveArea IRF resolution duplication is order-dependent in C++
Location: EventDisplay_v4/src/makeEffectiveArea.cpp:162-208
(See also B7.) IRF products are duplicated by index and only work if the source IRF was
initialised earlier in the vector.
Impact: Any refactor changing internal IRF ordering silently breaks resolution product
generation.
D12 · LOW · TMVA gamma/hadron preselection applies an unjustified 10× random retention factor
Location: EventDisplay_v4/src/trainTMVAforGammaHadronSeparation.cpp:143-185
After computing the required retained fraction, the code multiplies it by 10. before capping
at unity.
Impact: Makes the effective preselection behaviour opaque and bin-statistics dependent.
E — XGB / ML integration
E1 · HIGH · XGB data-analysis step not triggered automatically before XGB anasum cuts
Location: Eventdisplay_AnalysisScripts_VTS/scripts/ANALYSIS.dispXGB.sh:11-95; ANALYSIS.anasum_allcuts.sh:6-10,57-71; ANALYSIS.anasum_parallel_from_runlist.sh:133-145
ANALYSIS.dispXGB.sh exists as a standalone script, and the anasum wrappers now expose XGB
cut aliases (moderate2telXGB, etc.), but the documented workflow goes straight from
ANALYSIS.mscw_energy.sh to anasum. The wrappers do not trigger ANALYSIS.dispXGB.sh
automatically before selecting *-XGB-BDT cuts.
Impact: Users can request XGB cuts while analysing plain mscw files with no
.xgb_stereo.root/.xgb_gh.root sidecars, making XGB analyses a fragile manual side-band
instead of a reproducible pipeline stage.
E2 · HIGH · IRF.mscw_energy_MC_sub.sh XGB branch uses stale CLI and wrong run-parameter keys
Location: Eventdisplay_AnalysisScripts_VTS/scripts/helper_scripts/IRF.mscw_energy_MC_sub.sh:237-271,322-347; Eventdisplay-ML/src/eventdisplay_ml/config.py:175-230; EventDisplay_v4/src/VInstrumentResponseFunctionRunParameter.cpp:249-263
The helper calls eventdisplay-ml-apply-xgb-stereo --input-file --model-dir --output-file, but
the current CLI expects --input_file --model_prefix --output_file. It also writes
* XGBFILESUFFIX into the effective-area parameter file, while EventDisplay_v4 now parses
only XGBSTEREOFILESUFFIX and XGBGAMMAHADRONFILESUFFIX.
Impact: The legacy in-subscript XGB branch cannot drive the current Python CLI or pass its
outputs to the current C++ consumer; any workflow relying on this path is broken or silently
stale.
E3 · HIGH · XGB stereo training ignores the requested reconstruction ID
Location: Eventdisplay_AnalysisScripts_VTS/scripts/IRF.trainXGBforAngularReconstructionBinned.sh:48-56,76
The script accepts <Rec ID> as a required argument and carries it through naming and logging,
but hard-codes the training input directory to .../MSCW_RECID0_DISP.
Impact: Training jobs submitted for reconstruction IDs other than 0 silently use wrong MC
products.
E4 · HIGH · Non-XGB effective-area jobs still probe XGB sidecar files
Location: EventDisplay_v4/src/VStereoAnalysis.cpp:1965-1972; EventDisplay_v4/src/CData.cpp:28-38,1073-1127; logFiles/irfs/20deg_0.5wob_NOISE200.effarea.log:21-23
CData opens XGB friend files from non-empty suffix strings before checking whether the active
reconstruction methods require them. Confirmed in a production IRF log with
ENERGYRECONSTRUCTIONMETHOD 0 and a TMVA preselection cut file, yet the job still tries to open
.xgb_stereo.root/.xgb_gh.root.
Impact: Non-XGB jobs emit avoidable file-open errors and retain hidden dependencies on
artifacts that are not required.
E5 · HIGH · Eventdisplay-ML classification training and application can use different feature sets
Location: Eventdisplay-ML/src/eventdisplay_ml/data_processing.py:807-817; Eventdisplay-ML/src/eventdisplay_ml/models.py:441-452
Classification training switches to features_tmva_style(...) when tmva_style=True, but the
application path always uses features.features(analysis_type, training=False) and does not
mirror that branch.
Impact: A TMVA-style-trained classification model can be evaluated with the wrong input
feature set.
E6 · HIGH · Eventdisplay-ML config.py calls .get() on a list
Location: Eventdisplay-ML/src/eventdisplay_ml/config.py:162-165
configure_training() uses model_parameters.get("energy_bins_log10_tev", []).get("E_min");
energy_bins_log10_tev is list-structured and a list does not have .get().
Impact: Classification training/apply configuration can fail with AttributeError on
valid-looking parameter files in certain call paths.
E7 · MEDIUM · XGB training helper scratch directory uses undefined variable MSCW_FILE
Location: Eventdisplay_AnalysisScripts_VTS/scripts/helper_scripts/IRF.trainXGBforAngularReconstruction_sub.sh:15-18
TEMPDIR is derived from $(basename $MSCW_FILE .root), but MSCW_FILE is never defined; the
actual input is the list file LLIST.
Impact: Scratch-directory naming is accidental and environment-dependent, causing collisions
in batch production and making failed training jobs harder to debug.
E8 · MEDIUM · EVNDISP_MLSYS documented but helpers require a hard-coded conda environment name
Location: Eventdisplay_AnalysisScripts_VTS/scripts/helper_scripts/ANALYSIS.dispXGB_sub.sh:9-10,27-53; IRF.dispXGB_sub.sh:8-9; IRF.trainXGBforAngularReconstruction_sub.sh:9-10; IRF.trainXGBforGammaHadronSeparation_sub.sh:13-14; README.md:16-18,22-29
The public setup documents EVNDISP_MLSYS, but the helpers ignore it and require a pre-existing
conda environment named eventdisplay_ml.
Impact: A user can satisfy all documented environment variables and still fail at runtime.
E9 · MEDIUM · XGBoost hyperparameters are hard-coded in Python, not versioned in auxiliary parameter files
Location: Eventdisplay-ML/src/eventdisplay_ml/hyper_parameters.py:9-45
n_estimators, learning_rate, max_depth, subsample, colsample_bytree, etc. are embedded
in code; the shipped JSON files define only binning and tmva_style.
Impact: The effective ML model is not fully described by the analysis parameter files,
weakening reproducibility.
E10 · MEDIUM · Production XGB training helpers omit --random_state and --hyperparameter_config
Location: IRF.trainXGBforGammaHadronSeparation_sub.sh:59-67; IRF.trainXGBforAngularReconstruction_sub.sh:55-61
The helpers pass train_test_fraction and event limits but not a fixed random seed or explicit
hyperparameter file.
Impact: Routine retraining is not strictly reproducible even when input event lists are
unchanged.
E11 · MEDIUM · XGB command names are inconsistent between production scripts and documentation
Location: IRF.generalproduction.sh:18-29; IRF.production.sh:24-31,392-403; docs/changes/124.feature.md:1-10; README.md:83-159
IRF.generalproduction.sh advertises TRAINXGBANGRESBINNED; IRF.production.sh implements
TRAINXGBANGRES; the change note references a non-existent
IRF.trainXGBforAngularReconstruction.sh; the main README omits the XGB production sequence
entirely.
Impact: Operators are left with conflicting command names and incomplete instructions.
E12 · MEDIUM · Legacy TMVA.BDT.V4.runparameter ships a malformed PREPARE_TRAINING_OPTIONS line
Location: Eventdisplay_AnalysisFiles_VTS/ParameterFiles/TMVA.BDT.V4.runparameter:68-76; EventDisplay_v4/src/VTMVARunData.cpp:423-435
The line lacks the leading * required for parsing and contains a double :: in the option
string.
Impact: Legacy V4 BDT configuration does not reliably describe the training setup it
appears to encode.
E13 · LOW · TMVA training options have drifted from the accepted backend behaviour
Location: logFiles/irfs/mvaAngRes_20deg-BDTDisp-Tel2.log:81-82
NegWeightTreatment=InverseBoostNegWeights is not supported for BoostType=Grad and is silently
replaced by Pray.
Impact: Effective training configuration no longer matches the nominal requested one.
F — Auxiliary files & configuration
F1 · HIGH · Epoch metadata and downloadable IRF manifests are out of sync
Location: Eventdisplay_AnalysisFiles_VTS/ParameterFiles/VERITAS.Epochs.runparameter:47-53; Eventdisplay_AnalysisFiles_VTS/transfer/file_list_V6.dat:304-358
VERITAS.Epochs.runparameter defines V6_2024_2024s, V6_2024_2025w, V6_2025_2025s, and
V6_2025_2026w, but the shipped V6 download manifest stops at V6_2023_2024w.
Impact: A fresh checkout following the README cannot fetch the IRF tarballs needed for the
newer epochs that the active scripts may select at runtime.
F2 · HIGH · Future epoch V6_2025_2025s is defined twice in the epoch file
Location: Eventdisplay_AnalysisFiles_VTS/ParameterFiles/VERITAS.Epochs.runparameter:48-53
The second entry covers 2026-05-01 to 2026-11-24 while a TODO states it should become
V6_2026_2026s.
Impact: Future epoch lookup becomes ambiguous and can map runs to the wrong gain/atmosphere
configuration.
F3 · HIGH · Documented download path does not reproduce the full current auxiliary-file set
Location: Eventdisplay_AnalysisFiles_VTS/README.md:10-19; transfer/file_list_V6.dat:1-45,170-186; auxiliary_files.txt:19632-19645,20236-20262
The download manifests expose only classic DispBDTs and effective areas; the fully unpacked
inventory in auxiliary_files.txt additionally contains a DispXGB tree and XGB
effective-area products.
Impact: Two users following the official instructions end up with materially different
auxiliary installations.
F4 · HIGH · Unpacked auxiliary inventory is effectively AP-only despite scripts advertising multi-type support
Location: Eventdisplay_AnalysisScripts_VTS/README.md:58-81; IRF.production.sh:91-120; auxiliary_files.txt:19629-19631,21112-21169
Scripts and documentation present AP, NN, TS, and CC as supported analysis types; the
V6 file list even advertises DispBDTs_NN_* and EffectiveAreas_nominalHV_NN_* tarballs. The
unpacked inventory shows only an empty DispBDTs/NN directory; all table and effective-area
roots are -AP.root products.
Impact: The supposedly complete auxiliary area cannot support NN (or TS/CC) analyses.
F5 · HIGH · XGB instrument-file coverage is patchy while wrappers expose XGB as a generic AP mode
Location: auxiliary_files.txt:19632-19645,19697,19734,19933,20010,20177,20236-20262
Most XGB model directories are ATM61 only; XGB effective areas are confined to
V6_2016_2017 for ATM61/62, far narrower than the generic XGB hooks now exposed by the AP
wrappers.
Impact: The auxiliary-file area does not yet support XGB as a broadly deployable production
mode across the same epoch space as the classic AP products.
F6 · MEDIUM · XGB effective-area naming is internally inconsistent
Location: auxiliary_files.txt:20245-20248,20698; ANALYSIS.anasum_parallel_from_runlist.sh:133-145
The same tree contains both ...ModerateXGB...root and ...Moderate-XGB-BDT...root names;
the analysis wrapper resolves moderate2telXGB to NTel2-PointSource-Moderate-XGB-BDT.
Impact: Shell wrappers, cut files, and effective-area selection logic can disagree on which
product family is canonical.
F7 · MEDIUM · Maintainer README is stale and contradicts the transfer scripts
Location: Eventdisplay_AnalysisFiles_VTS/README.md:29-30,77-89; transfer/pack_irfs_for_ucla.sh:11; transfer/put_irfs_to_ucla.sh:27
The README documents a non-existent EVNDISP.reconstruction.runparameter, refers generically
to tar_packages, and instructs maintainers to move packages to archive before calling an
upload script that only scans tar_packages_${VERSION}/.
Impact: Release maintenance instructions are no longer self-consistent.
F8 · MEDIUM · download_irfs.sh continues into extract/remove after a failed transfer
Location: Eventdisplay_AnalysisFiles_VTS/download_irfs.sh:36-49
tar --keep-newer-files and rm -v execute immediately after the copy/bbftp call without
checking the transfer exit status.
Impact: Failed or partial downloads trigger extraction and cleanup, making recovery harder.
F9 · MEDIUM · copy_dispBDTs.sh and copy_dispXGB.sh silently overwrite SIMTYPE in a loop
Location: Eventdisplay_AnalysisFiles_VTS/DispBDTs/copy_dispBDTs.sh:11-19; DispXGB/copy_dispXGB.sh:11-18
Both scripts assign SIMTYPE repeatedly, so only the last hard-coded value survives.
copy_dispBDTs.sh also prints ${IRVERSION} instead of ${IRFVERSION}.
Impact: IRF-maintenance copies silently pull the wrong model family with misleading
provenance information.
F10 · LOW · V4/V5 gamma/hadron copy script has dead atmosphere-selection code
Location: Eventdisplay_AnalysisFiles_VTS/GammaHadronBDTs/copy_GammaHadron_V4_V5_BDTs.sh:21-31
Loop iterates over ATM21 ATM22 but the branch condition checks for ATM62; the
summer/winter split is never exercised.
Impact: Any future atmosphere-specific V4/V5 handling based on this template will be wrong.
F11 · LOW · Global DB fallback is present in text only, not in parsed configuration
Location: Eventdisplay_AnalysisFiles_VTS/ParameterFiles/EVNDISP.global.runparameter:9-16
Only the romulus.ucsc.edu line is active (*-prefixed); the remus.ucsc.edu fallback is
unstarred and ignored.
Impact: No real failover at the parameter-file level if the primary DB endpoint is
unavailable.
F12 · LOW · Backup content mixed into the main instrument-file tree
Location: auxiliary_files.txt:19298
A DispBDTs/AP.back subtree sits alongside nominal production content.
Impact: Ambiguity about which products are authoritative; risk of copying the wrong tree.
G — Release tests
G1 · HIGH · EventDisplay_Release_v492 is still a skeleton, not an auditable release-test record
Location: EventDisplay_Release_v492/README.md:1-3
The checkout contains only metadata plus runparameter/V6.AP.runparameter.dat; none of the
expected result trees (Crab/, mc_data_comparison/, irf_plotting/, SourceTests/,
AP/radialAcceptances/) are present.
Impact: The v492 artifact does not provide the evidence needed to review the release tests
the codebase expects to exist.
G2 · HIGH · Release-test output naming is inconsistent
Location: EventDisplay_ReleaseTests_code/README.md:10-15; release_tests/montecarlo/energythresholds/plot.C:150-152
Most scripts use EventDisplay_Release_<version>, but the README points to
EventDisplay_ReleaseTests_v492 and the energy-threshold macro writes to
EventDisplay_ReleaseTests_<version>/energythresholds/.
Impact: Release products are scattered between differently named trees, making results look
missing and breaking documentation links.
G3 · HIGH · MC/data comparison applies the summer-atmosphere filter with an undefined variable
Location: EventDisplay_ReleaseTests_code/release_tests/montecarlo/mc_data_comparison/compareDatawithMC.sh:138-146
The winter branch checks ${atm}, but the summer branch checks ${A} before A is assigned.
Impact: Summer-epoch filtering silently uses the wrong atmosphere combinations, weakening one
of the main release-validation checks.
G4 · HIGH · Crab run-list generation: WOBBB typo leaves zero-wobble classification undefined
Location: EventDisplay_ReleaseTests_code/release_tests/sources/Crab/runlist_generator_from_anasum_log.sh:127-132,156-165
When both wobble offsets are zero, the script assigns WOBBB="0" instead of WOBB="0", but
later decisions key on WOBB.
Impact: Zero-wobble runs inherit a stale wobble state and are grouped into the wrong run
lists, contaminating downstream source-test summaries.
G5 · HIGH · Source-test plotting macros are pinned to the v491 library and data layout
Location: release_tests/sources/Crab/plot_lightcurves.C:19; plot_energy_spectra.C:20; plot_skymaps.C:17; plot_sensitivity.C:11,27-40; plot_sensitivity_compare_cuts.C:16,44-62
These macros hard-code /.../EventDisplay_v491-al9/lib/libVAnaSum.so and for sensitivity plots
hard-code v491/v490 result directories.
Impact: Running on v492 products loads the wrong code and compares against fixed historical
paths.
G6 · MEDIUM · IRF plotting is hard-coded to a single slice instead of broad parameter coverage
Location: EventDisplay_ReleaseTests_code/release_tests/montecarlo/irf_plotting/irf_plotting.sh:16-25,57,75-89
The driver hard-codes COMPAREVERSION="v492", a fixed sim/cut family, and only runs the body
for V6_2016_2017 / ATM61.
Impact: The nominal IRF plotting step no longer spans the stated parameter space and is weak
as a release-regression check.
G7 · MEDIUM · Shipped v492 run-parameter format encodes a fragile cut-name duality
Location: EventDisplay_Release_v492/runparameter/V6.AP.runparameter.dat:35-39; release_tests/sources/Crab/anasum_from_runlists.sh:34-40
Each * CUT line stores both a full IRF cut identifier and a short alias; consumers rely on the
fourth field specifically.
Impact: Easy for future scripts to read the wrong cut token and silently select the wrong
inputs.
H — Performance & efficiency
H1 · HIGH · evndisp keeps time-sliced pedestal analysis enabled during normal data reconstruction
Location: EventDisplay_v4/src/VEventLoop.cpp:490-495,1442-1461; EventDisplay_v4/src/VPedestalCalculator.cpp:30-199,280-381; logFiles/analysis/64080.log:71,598-599
On the exercised run, evndisp processes ~205 events/s (543 481 events in 44:04). The active
path still performs pedestal-variance bookkeeping for every telescope during standard
reconstruction: VPedestalCalculator::doAnalysis() loops over hit channels, recomputes trace
sums for every summation window, and fills per-pixel histograms.
Impact: A calibration-style side analysis remains in the hottest reconstruction loop and is a
plausible first-order contributor to the 16× throughput gap between evndisp (~205 ev/s) and
mscw_energy (~3 229 ev/s).
H2 · MEDIUM · evndisp repeats invariant low-gain/sum-window validation in the per-event loop
Location: EventDisplay_v4/src/VEventLoop.cpp:1204-1292
The analysis path rechecks availability of low-gain multipliers for sumwindow_1,
sumwindow_2, and sumwindow_pass1 on every event for every telescope, even though these are
run-level constants after initialisation.
Impact: Avoidable branch-heavy work repeated millions of times in long runs.
H3 · HIGH · anasum double-scans mscw.root and reinitialises TMVA twice per run
Location: EventDisplay_v4/src/VAnaSum.cpp:464-465; EventDisplay_v4/src/VStereoAnalysis.cpp:264-349,434-448; EventDisplay_v4/src/VTMVAEvaluator.cpp:366-511; logFiles/analysis/64080.anasum.log:97-103,237-243
In the reflected-background path, ON and OFF both point to the same mscw.root, yet VAnaSum
runs separate passes. Each pass reads 174 937 entries and rebuilds the gamma/hadron TMVA
readers from disk.
Impact: For every run, anasum roughly doubles tree I/O and repeats expensive TMVA and
effective-area initialisation that could be shared.
H4 · MEDIUM · anasum uses a linear TMVA-bin search with recomputed mean energies per event
Location: EventDisplay_v4/src/VTMVAEvaluator.cpp:714-745,802-839
Each gamma/hadron evaluation calls getDataBin(), which scans all loaded bins and recomputes
the mean energy of each candidate before evaluating the selected reader.
Impact: Avoidable per-event classifier overhead in the inner anasum loop, especially with
many energy/zenith bins.
H5 · MEDIUM · Multi-run anasum jobs scale as O(runs × total entries in chain)
Location: EventDisplay_v4/src/VAnaSum.cpp:464-465; EventDisplay_v4/src/VStereoAnalysis.cpp:434-450
fillHistograms() loops over all chain entries and then filters by runNumber == irun; each
run-specific pass re-reads entries from all other runs.
Impact: Runtime can grow toward O(runs × total_entries) instead of near-linear.
H6 · MEDIUM · evndisp reads pedestal calibration inputs redundantly during initialisation
Location: logFiles/analysis/64080.log:247-252,263-268,275-280,291-296
The same telescope-level pedestal products are read twice in succession during calibration setup.
Impact: Duplicated calibration I/O in the slowest stage of the data pipeline.
H7 · MEDIUM · mscw_energy repeatedly rebuilds TMVA datasets instead of amortising setup
Location: logFiles/analysis/64080.mscw.log:156-169; logFiles/irfs/20deg_0.5wob_NOISE200.mscw.log:261-277
Both data-side and MC-side logs contain repeated Rebuilding Dataset Default messages.
Impact: Avoidable setup churn, expensive when multiplied across large IRF productions.
Cross-cutting issues (confirmed from logs)
X1 · HIGH · v2dl3 silently sets quality flag to 0 when Eventdisplay logfile is missing
Location: logFiles/analysis/64080.v2dl3.log:14
v2dl3 reports "Eventdisplay logfile not found in anasum root file. Quality flag set to 0."
Impact: Downstream DL3 products lose quality metadata without a loud failure.
X2 · MEDIUM · anasum continues under missing TMVA-weight warnings
Location: logFiles/analysis/64080.anasum.log:243-249,541-547; EventDisplay_v4/src/VTMVAEvaluator.cpp:190-279
VTMVAEvaluator reports "TMVA root file not found or incomplete file (0)" and continues;
see also B4.
Impact: Incomplete classifier inputs survive initialisation silently.
X3 · MEDIUM · anasum reinitialises the same TMVA state twice, doubling warning noise
Location: logFiles/analysis/64080.anasum.log:237-247,535-545
See also H3. The same missing/incomplete low-energy TMVA file warning is emitted once for ON
and once again for OFF.
Impact: Duplicated warnings make it harder to distinguish a new problem from a repeated one.
X4 · LOW · Data run contains large numbers of GPS-faulted events; pipeline continues without escalation
Location: logFiles/analysis/64080.log:565-580,587-596
Missing telescope events and GPS faults on telescopes 1 and 4 are logged and skipped.
Impact: Run-quality issues are not escalated; pipeline favours "continue with warnings" over
explicit run-quality gating.
Hyperparameter locations (reference)
| Model |
Location |
| TMVA gamma/hadron BDT |
Eventdisplay_AnalysisFiles_VTS/ParameterFiles/TMVA.BDT.runparameter:13-15 |
| TMVA DispBDT |
Eventdisplay_AnalysisFiles_VTS/ParameterFiles/TMVA.BDTDisp.runparameter:1-2 |
| XGB regression/classification |
Eventdisplay-ML/src/eventdisplay_ml/hyper_parameters.py:9-45 (hard-coded in Python) |
| XGB binning / style |
Eventdisplay_AnalysisFiles_VTS/ParameterFiles/XGB-stereo-parameter.json; XGB-classify-parameter.json |
Priority action list
Immediate (HIGH severity, silent wrong-physics or silent job failure)
- Fix
ANALYSIS.anasum.sh SUBC initialisation (anasum.sh:90-92) — jobs are silently never submitted (C1)
- Fix
get_disp_dir() to check HVSETTINGS == "obsLowHV" instead of undefined SIMTYPE (mscw_energy_sub.sh:136-152) — wrong DispBDT directory for all RedHV/UV runs at high zenith (C2)
- Fix shower-core
yimp validity check (VArrayAnalyzer.cpp:944-951) — NaN core-Y propagates into reconstruction (A6)
- Fix method-4 pair-angle rejection variable (
VArrayAnalyzer.cpp:1241-1248) — wrong telescope pairs used in cut (A7)
- Fix low-gain T0 assignment and calculation (
VCalibrationData.cpp:365-373; VCalibrator.cpp:899-903) — high-gain state corrupted (A4, A5)
- Fix core-resolution formula (
VInstrumentResponseFunctionData.cpp:472-485) — Xcore used twice, Ycore ignored (A12)
- Fix TMVA zenith-bin off-by-one (
trainTMVAforGammaHadronSeparation.cpp:255-266) — out-of-bounds vector access (A13)
- Fix TMVA training factory type for reconstruction-quality path (
trainTMVAforGammaHadronSeparation.cpp:343-360) — classification factory used instead of regression (A14)
- Fix IRF job-launch filenames (
IRF.generate_lookup_table_parts.sh:86-124; IRF.mscw_energy_MC.sh:97-139) — submitted path does not exist (D4)
- Fix
IRF.production.sh double-loop and epoch variable (IRF.production.sh:337-387,353-366) — TMVA grid submitted multiple times with wrong epoch variable (D1, D2)
- Fix
IRF.mscw_energy_MC_sub.sh quoted glob and stale XGB CLI (IRF.mscw_energy_MC_sub.sh:133-136,237-271) — files not copied; Python CLI mismatch (D3, E2)
- Propagate
FORCEDATMO via sed template (ANALYSIS.mscw_energy.sh:63) — ATM override is a no-op in batch (C3)
- Add atmosphere-ID normalisation for V4/V5 in
mscw_energy_sub.sh consistent with prepare_atmo_string() — table lookup fails for V4/V5 GRISU data (C4)
- Fix
ANALYSIS.evndisp_sub.sh DBTEXT file existence checks — missing files silently inject wrong metadata (C9)
- Fix broken TMVA bin-parsing regex (
IRF.trainTMVAforGammaHadronSeparation.sh:107-109) — bin parsing fails (C10)
- Fix sliding-window FADC out-of-bounds read (
VTraceHandler.cpp:783) — FADC[n] accessed on every event where the window reaches the rightmost search position; UB corrupts charge reconstruction (A17)
- Fix
VOnOff::doOnOffforParameterHistograms background mutation (VOnOff.cpp:159-161) — hoff->Scale after Add double-scales background, corrupting reflected-region excess counts and significance in anasum (B11)
- Fix
VInstrumentResponseFunctionRunParameter silent key drop (VInstrumentResponseFunctionRunParameter.cpp:109-358) — add a terminal else { cerr << "unknown key: " << temp << endl; } to catch parameter typos (B13)
- Fix
printRunParameter exit code on file-open failure (printRunParameter.cpp:330-335) — replace exit(0) with exit(EXIT_FAILURE) so calling shell scripts detect errors (B14)
Short-term (MEDIUM / HIGH, reproducibility and efficiency)
- Gate time-sliced pedestal analysis in the standard
evndisp analysis loop — primary throughput bottleneck (H1)
- Avoid double
mscw.root scan and double TMVA initialisation in anasum — doubles tree I/O (H3)
- Fix Eventdisplay-ML train/apply feature mismatch for
tmva_style=True classification (E5)
- Fix
config.py .get() on list — AttributeError on valid parameter files (E6)
- Wire
ANALYSIS.dispXGB.sh into the default anasum workflow, or add a guard that prevents *-XGB-BDT cut selection without XGB sidecars (E1)
- Fix XGB stereo training RecID hard-coding (
IRF.trainXGBforAngularReconstructionBinned.sh:48-56) (E3)
- Gate XGB friend-file probing in
CData to the active reconstruction methods (E4)
- Synchronise epoch file, download manifests, and downloadable IRF tarballs (F1, F2, F3)
- Add end-to-end IRF → analysis consistency test — shell and C++ interface drift surface only in production
- Fix
makeEffectiveArea "no problem" comment vs. exit(EXIT_FAILURE) mismatch (makeEffectiveArea.cpp:118; VInstrumentResponseFunctionRunParameter.cpp:437) — either remove the misleading comment or implement graceful fallback (B12)
- Fix
VTableLookupRunParameter error message (VTableLookupRunParameter.cpp:175) — change "2=read" to "0=read" (B15)
- Fix
VTraceHandler::getQuickMax tmax initialisation (VTraceHandler.cpp:420-426) — initialise tmax to 0 or the expected maximum before computing nMax (A18)
Issue Report
(used GPT-5.4 and Opus 4.7)
Scope: Full pipeline audit across
EventDisplay_v4(C++),Eventdisplay_AnalysisScripts_VTS(shell orchestration),
Eventdisplay_AnalysisFiles_VTS(auxiliary files), andEventdisplay-ML(XGB/Python ML). Log files and release-test artifacts were also reviewed.
Severity labels:
HIGH— incorrect physics output or silent job failure;MEDIUM—degraded reliability, reproducibility, or correctness in edge cases;
LOW— cosmetic,misleading diagnostics, or minor fragility.
Pipeline overview (reference)
IRF.evndisp_MC.sh→evndispANALYSIS.evndisp.sh→evndispIRF.mscw_energy_MC.sh→mscw_energyANALYSIS.mscw_energy.sh→mscw_energyIRF.train*)mscw_energyanasumANALYSIS.anasum*.sh→anasumANALYSIS.v2dl3.sh→v2dl3C++ class chains:
evndisp:VReadRunParameter→VEventLoop→VTraceHandler/VFitTraceHandler,VCalibrator,VImageAnalyzer,VArrayAnalyzer,VPedestalCalculatormscw_energy:VTableLookupRunParameter→VTableLookup→CData,VDispAnalyzer,VTMVADispAnalyzer,VEmissionHeightCalculatoranasum:VAnaSumRunParameter→VAnaSum→VStereoAnalysis→CData,VGammaHadronCuts,VTMVAEvaluatormakeEffectiveArea:VGammaHadronCuts→CData→VInstrumentResponseFunction→VInstrumentResponseFunctionDataA — C++ algorithm defects
A1 · HIGH · Out-of-bounds read in pulse-timing extraction
Location:
EventDisplay_v4/src/VTraceHandler.cpp:346-358getPulseTiming()interpolates on the rising edge usingfpTrace[i+1]. When the pulse maximumlands on the last sample (
maxpos = size-1),i+1is out of bounds.Impact: Truncated or late pulses produce undefined timing values in the signal characterisation
path.
A2 · HIGH · Sliding-window charge branch returns
FADC[1]instead of integrated chargeLocation:
EventDisplay_v4/src/VTraceHandler.cpp:745-757calculateTraceSum_slidingWindow()computeschargeand updates the integration-window metadatabut then returns
FADC[1].Impact: When the raw sliding-window branch is active, charge extraction is silently wrong.
A3 · MEDIUM · Fitted-trace T0 mixes pedestal-subtracted and absolute amplitude conventions
Location:
EventDisplay_v4/src/VFitTraceHandler.cpp:433-442getTraceMax()returns a pedestal-shifted peak, butgetTraceTZero()solves for-(imax/2 + fPed), re-adding the pedestal to the half-height target.Impact: Fitted pulse-arrival times are biased whenever the pedestal is non-zero.
A4 · HIGH · Low-gain average-T0 assignment always writes the high-gain field
Location:
EventDisplay_v4/src/VCalibrationData.cpp:365-373setAverageTZero(bool iLowGain)unconditionally assignsfAverageTZero_highgain, even wheniLowGain == true.Impact: The low-gain timing calibration path corrupts the high-gain average-T0 used later in
analysis.
A5 · HIGH · Low-gain average-T0 calculation uses the high-gain sum path
Location:
EventDisplay_v4/src/VCalibrator.cpp:899-903calculateAverageTZero(bool iLowGain)callscalcSums(..., iLowGainOnly=false)unconditionally.Impact: Low-gain timing constants are derived from the wrong charge-extraction mode.
A6 · HIGH · Shower-core validity check tests
ximptwice, neveryimpLocation:
EventDisplay_v4/src/VArrayAnalyzer.cpp:944-951fillShowerCore()evaluates!isnormal(ximp) || !isnormal(ximp); the duplicate check means aninvalid
yimpis never screened out.Impact: NaN or invalid core-Y solutions propagate into core distances, energy reconstruction
inputs, and event-selection variables.
A7 · HIGH · Method-4 stereo pair-angle rejection applies cut to the wrong telescope pair
Location:
EventDisplay_v4/src/VArrayAnalyzer.cpp:1241-1248The per-pair rejection computes the current pair in
mm[0]/mm[1]but evaluates the cut usingm[0]/m[1].Impact: For events with more than two images, pair-angle rejection is applied to the wrong
telescopes, biasing the weighted line intersections used for direction reconstruction.
A8 · HIGH · Log-likelihood image fit can be initialised with invalid centroid bounds
Location:
EventDisplay_v4/src/VImageParameterCalculation.cpp:1542-1555,1588-1601For edge/degenerate images with
sigmaX <= 0orsigmaY <= 0, only the upper Minuit bound isset; the lower bound retains its sentinel value before being passed to the minimiser.
Impact: Edge images can enter the recovery LL fit with invalid parameter bounds, degrading
parameterisation in the regime where the fit is intended to stabilise results.
A9 · MEDIUM · Muon-ring image-size correction divides by zero at the first grid point
Location:
EventDisplay_v4/src/VImageParameterCalculation.cpp:737-748xi_tmp = i / numStepsis zero wheni == 0, butkTest_tmp = ... / xi_tmpis evaluated beforeany guard.
Impact: Muon calibration correction can produce
inf/NaNcandidates and an unstablecorrection factor.
A10 · MEDIUM · Emission-height pair weights undefined for small image sizes
Location:
EventDisplay_v4/src/VEmissionHeightCalculator.cpp:60-64The weight formula
1./((1./log10(size[i])) + (1./log10(size[j])))is not guarded againstsize <= 1.Impact: Small-image events receive singular, negative, or ill-defined pair weights in the
emission-height estimate.
A11 · MEDIUM ·
VEmissionHeightCalculatoraccumulates telescope positions across reinitalisationsLocation:
EventDisplay_v4/src/VEmissionHeightCalculator.cpp:100-133Both
setTelescopePositions()overloads onlypush_back()and never clear previous contents.Impact: Reusing the same calculator instance duplicates telescope geometry, making pairwise
shower-height geometry wrong.
A12 · HIGH · Core-resolution IRF formula uses
Xcoretwice instead ofXcoreandYcoreLocation:
EventDisplay_v4/src/VInstrumentResponseFunctionData.cpp:472-485The core-resolution error is computed as
sqrt(Xcore²+Xcore²) - sqrt(MCxcore²+MCycore²).Impact: Reconstructed core-error quantities are biased whenever
Ycorediffers fromXcore;this is a direct physics error in the IRF-resolution path.
A13 · HIGH · Gamma/hadron TMVA training has an off-by-one zenith-bin bounds check
Location:
EventDisplay_v4/src/trainTMVAforGammaHadronSeparation.cpp:255-266The check uses
< iZenithBininstead of<= iZenithBin, soiZenithBin == size()passes and isthen dereferenced.
Impact: Certain bin configurations index beyond the configured zenith-bin vectors during
training.
A14 · HIGH · Reconstruction-quality TMVA training configured as a classification factory
Location:
EventDisplay_v4/src/trainTMVAforGammaHadronSeparation.cpp:343-360The non-gamma/hadron branch adds a regression target but creates the
TMVA::Factorywith thedefault (classification) options string.
Impact: The code path described as reconstruction-quality training is algorithmically
mismatched to the TMVA factory configuration.
A15 · MEDIUM · Angular-reconstruction training silently rescales the requested train/test split
Location:
EventDisplay_v4/src/trainTMVAforAngularReconstruction.cpp:125-138After deriving
ntrainandntestfrom the requested fraction, both are multiplied by0.8.Impact: Effective train/test statistics differ from the configured values, reducing
reproducibility.
A16 · MEDIUM · Reduced-tree subsampling normalised by file count, not event population
Location:
EventDisplay_v4/src/trainTMVAforGammaHadronSeparation.cpp:147-157The retained fraction is divided by
iTreeVector.size()(number of input files), not by eventcounts.
Impact: Training preselection depends on file granularity and can distort class composition
across uneven file splits.
A17 · HIGH · Sliding-window charge extraction reads one past the end of the FADC array
Location:
EventDisplay_v4/src/VTraceHandler.cpp:725-783iSearchEndis clamped ton - iIntegrationWindow + 1. At the final loop iteration(
i = iSearchEnd - 1), the expressionFADC[i + iIntegrationWindow]evaluates toFADC[n],which is one past the end of the valid array range
[0, n-1].Impact: Undefined behaviour on every waveform where the sliding window reaches its rightmost
position; corrupts charge reconstruction silently on affected events.
A18 · MEDIUM ·
VTraceHandler::getQuickMaxuses the previous call's trace maximum for saturation countingLocation:
EventDisplay_v4/src/VTraceHandler.cpp:420-426nMaxis computed as(int)(fDynamicRange * tmax)beforetmaxis reset to −10000. If thecaller passes an uninitialised or stale value, the saturation counter
n255is thresholdedagainst the wrong maximum.
Impact: The number of saturated samples can be over- or under-estimated, misclassifying
pulses as saturated or unsaturated and biasing image-cleaning amplitude thresholds.
B — C++ infrastructure defects
B1 · MEDIUM ·
CData::getXGBTree()assumes.rootis present in the filenameLocation:
EventDisplay_v4/src/CData.cpp:1073-1086The XGB friend-file name is rewritten via
string::replace(find(".root"), ...)without checkingwhether
.rootexists in the name.Impact: The class API is brittle and can generate an invalid replacement if reused with a
non-standard input name.
B2 · MEDIUM · XGB gamma/hadron C++ interface hard-wired to
Is_Gamma_70Location:
EventDisplay_v4/src/CData.cpp:1118-1127;EventDisplay_v4/src/VGammaHadronCuts.cpp:727-731,1118-1128The C++ consumer binds only
Gamma_PredictionandIs_Gamma_70; the classification operatingpoint is baked into the upstream friend-tree producer, not enforced by
VGammaHadronCuts.Impact: The 70% efficiency threshold cannot be adjusted from cut files or runtime parameters,
discarding all the optimisation flexibility of the XGB classifier.
B3 · MEDIUM ·
VTMVAEvaluatorrecovers feature names by brittle XML string slicingLocation:
EventDisplay_v4/src/VTMVAEvaluator.cpp:62-116Variable recovery depends on exact XML formatting and field ordering (
NVar,Expression,Label,SpecIndex) rather than a structured parser.Impact: A TMVA XML layout change can silently alter which variables the evaluator reads,
turning the deployed BDT interface into a schema-fragile contract.
B4 · MEDIUM · TMVA weight grid initialisation tolerates missing interior bins with warnings
Location:
EventDisplay_v4/src/VTMVAEvaluator.cpp:190-279Missing weight files at non-edge bins emit warnings and allow the evaluator to continue.
Impact: Incomplete TMVA weight grids survive initialisation, making production behaviour
depend on partial bin coverage instead of a strict consistency check.
B5 · LOW ·
VTMVARunDatasanity message does not match the actual conditionLocation:
EventDisplay_v4/src/VTMVARunData.cpp:795-800Prints "need at least two energy bins" while the code only rejects
< 1bin.Impact: Misleading diagnostics when debugging training-configuration problems.
B6 · MEDIUM ·
VInstrumentResponseFunctionmutates shared MC azimuth in placeLocation:
EventDisplay_v4/src/VInstrumentResponseFunction.cpp:193-199MC azimuth normalisation is applied as
fData->MCaz -= 360.on the shared event object.Impact: Weak implementation pattern; mutates shared state instead of using a local normalised
copy.
B7 · MEDIUM ·
makeEffectiveAreaIRF duplication is order-dependentLocation:
EventDisplay_v4/src/makeEffectiveArea.cpp:162-208,194-208IRF products are duplicated by index via
getDuplicationID()and only work if the source hasalready been initialised earlier in the vector.
Impact: A future refactor changing the internal IRF ordering will silently break resolution
product generation.
B8 · MEDIUM ·
copyMCHistograms()keeps the first ROOT file openLocation:
EventDisplay_v4/src/makeEffectiveArea.cpp:370-401The first
MChistosobject is borrowed directly from the first input file, which is never closedinside the helper.
Impact: File-handle lifetime is coupled to the returned object, risking resource leaks or
dangling-file state.
B9 · MEDIUM ·
anasumcontains a stale unreachableanalysisType == 5branchLocation:
EventDisplay_v4/src/anasum.cpp:47-54,106-108analysisTypeis explicitly limited to 3 and 4, but theVAnaSum::doStereoAnalysis()handoffstill tests
analysisType == 5.Impact: Dead code path; sign of an incomplete control-flow change.
B10 · MEDIUM · Epoch reevaluation at runtime means analysis behaviour changes with epoch-file updates
Location:
EventDisplay_v4/src/VReadRunParameter.cpp:463-470;EventDisplay_v4/src/printRunParameter.cpp:100-117printRunParameter updated-runinfoandevndisp -epochfilereevaluate epoch and atmosphere fromVERITAS.Epochs.runparametereach run rather than relying solely on the file-internal metadata.Impact: Reprocessing the same intermediate ROOT file with an updated epoch file can silently
change the science output.
B11 · HIGH ·
VOnOffmutates the background histogram after using it, corrupting reflected-region normalisationLocation:
EventDisplay_v4/src/VOnOff.cpp:159-161doOnOffforParameterHistogramscomputes the excess ashTemp->Add(hon, hoff, 1., -1.*i_norm_alpha)(which consumeshoffat its original scale) andthen immediately calls
hoff->Scale(i_norm_alpha). Any caller that reuseshoffafter this callsees a rescaled background histogram.
Impact: The reflected-background normalisation in the
anasumanalysis path (viaVAnaSum.cpp:556)can be doubly-scaled, producing wrong excess counts and significance values.
B12 · HIGH ·
makeEffectiveAreacomments that the MC run header is optional but hard-exits when it is missingLocation:
EventDisplay_v4/src/makeEffectiveArea.cpp:118;EventDisplay_v4/src/VInstrumentResponseFunctionRunParameter.cpp:430-437The comment at line 118 states "might not be there, no problem", yet
readMCRunHeader()callsexit(EXIT_FAILURE)when noMC_runheaderobject is found in the input file.Impact: Misleading documentation causes operators to expect graceful handling; actual behaviour
is a hard crash with a non-zero exit code when the MC header is absent (e.g., for older files).
B13 · HIGH ·
VInstrumentResponseFunctionRunParametersilently ignores unknown run-parameter keysLocation:
EventDisplay_v4/src/VInstrumentResponseFunctionRunParameter.cpp:109-358The parser is a chain of
else ifbranches ending atRERUN_STEREO_RECONSTRUCTION_3TELwithno terminal
elseand no warning message. Unknown or misspelled keys are silently dropped.Impact: Typos in
makeEffectiveAreaparameter files produce no diagnostic and default valuesare used; an operator has no way to detect that a setting was not applied.
B14 · MEDIUM ·
printRunParameterexits with code 0 on file-open failureLocation:
EventDisplay_v4/src/printRunParameter.cpp:330-335new TFile(argv[1])can return a zombie object; the code prints "exiting..." and callsexit(0).Impact: Shell wrappers that check only the exit code (C6, several IRF helpers) will interpret
the empty output as a missing field rather than a fatal error, silently propagating bad metadata
through the pipeline.
B15 · LOW ·
VTableLookupRunParametererror message says "2=read" but the code only accepts "0=read"Location:
EventDisplay_v4/src/VTableLookupRunParameter.cpp:162-176The
-filloption accepts0(read) or1(fill); any other value triggers the message"unknown parameter, choose 1=fill or 2=read lookup tables", which is incorrect.
Impact: Operators following the error message will pass
-fill=2, which also triggers theerror in a confusing loop.
C — Shell orchestration bugs
C1 · HIGH ·
ANALYSIS.anasum.shnever submits any jobsLocation:
Eventdisplay_AnalysisScripts_VTS/scripts/ANALYSIS.anasum.sh:90-92Line 90 assigns the helper path as a string; the subsequent
evalonly echoes it back. Everyother equivalent script correctly uses backticks to execute the helper:
All
*qsub*/*condor*/*sbatch*/*simple*tests fail, so generated job scripts are writtenbut never submitted.
Impact: The full-runlist
anasumpath silently drops all jobs with no error message.C2 · HIGH ·
get_disp_dir()checks an undefined variable, always selects the wrong DispBDT directoryLocation:
Eventdisplay_AnalysisScripts_VTS/scripts/helper_scripts/ANALYSIS.mscw_energy_sub.sh:136-152The function checks
${SIMTYPE} == "CARE_RedHV"to decide between 55-deg and 60-deg DispBDTfallback models.
SIMTYPEis never assigned in this script; onlySIMTYPE_RUNis set (lines82–93). A secondary issue: the current value since v491 is
CARE_RedHV_Feb2024, notCARE_RedHV, so the comparison would fail even if the variable were correct. The proper guardshould use
HVSETTINGS == "obsLowHV".Impact: For all RedHV (and high-zenith UV) runs,
mscw_energyloads 60-deg DispBDT modelsinstead of the 55-deg fallback, degrading direction reconstruction quality for runs at zenith ≥ 58°.
C3 · MEDIUM ·
FORCEDATMOatmosphere override is silently ignored in cluster submissionsLocation:
Eventdisplay_AnalysisScripts_VTS/scripts/ANALYSIS.mscw_energy.sh:63;helper_scripts/ANALYSIS.mscw_energy_sub.sh:71The parent script captures
FORCEDATMO=$6locally. The sed-template substitution block does notinject
FORCEDATMOinto the generated job script, and the job script references${FORCEDATMO:-...}as an unexported shell variable, which is empty in batch environments.Impact: The documented
[ATM]override parameter does nothing in production; usersattempting to force an atmosphere ID get silently wrong results.
C4 · MEDIUM ·
mscw_energy_sub.shmissing atmosphere-ID normalisation for V4/V5 dataLocation:
Eventdisplay_AnalysisScripts_VTS/scripts/helper_scripts/ANALYSIS.mscw_energy_sub.sh:68-71;ANALYSIS.anasum_sub.sh:74-83mscw_energy_sub.shcallsprintRunParameter … updated-runinfo, which may return a modernised6x atmosphere ID for old V4/V5 observations.
anasum_sub.shcompensates withprepare_atmo_string()(${ATMO/6/2}for V4/V5);mscw_energy_sub.shhas no equivalentconversion.
Impact:
mscw_energyexits with "table file not found" for V4/V5 GRISU data when the epochfiles return a modernised atmosphere ID.
C5 · MEDIUM ·
DISPBDTguard inmscw_energy_sub.shis missing$Location:
Eventdisplay_AnalysisScripts_VTS/scripts/helper_scripts/ANALYSIS.mscw_energy_sub.sh:193if [[ DISPBDT != "NOTSET" ]]; thenreferences the literal stringDISPBDTinstead of$DISPBDT, so the condition is never conditionally guarded.Impact: The log reports DispBDT XML provenance even when that branch is not actually active.
C6 · MEDIUM ·
printRunParameter -elevationfield is parsed positionally in the helperLocation:
Eventdisplay_AnalysisScripts_VTS/scripts/helper_scripts/ANALYSIS.mscw_energy_sub.sh:127;EventDisplay_v4/src/printRunParameter.cpp:340-343The helper extracts the zenith angle by
awkfield number from a human-readable CLI printoutrather than from a stable machine-readable interface.
Impact: Fragile table and DispBDT directory selection; breaks silently if the printout format
changes.
C7 · MEDIUM ·
mscw_energyhard-fails when a TMVA XML family is missing, no soft fallbackLocation:
EventDisplay_v4/src/VTMVADispAnalyzer.cpp:97-105,164-170When TMVA XML prefixes are provided and one family is missing, the code exits without graceful
degradation.
Impact: Missing or incompletely copied DispBDT XMLs stop the run.
C8 · MEDIUM ·
anasumdefault input directory hard-wired toRecID0with no consistency checkLocation:
Eventdisplay_AnalysisScripts_VTS/scripts/ANALYSIS.anasum.sh:47-49;helper_scripts/ANALYSIS.anasum_sub.sh:213-226The default input directory is
.../RecID0; no validation that themscw.rootfiles wereproduced with the requested reconstruction ID.
Impact: A workflow that reconstructs with a non-zero RecID can run
anasumagainst staleRecID0products, producing plausible but inconsistent science outputs.C9 · HIGH ·
ANALYSIS.evndisp_sub.shconsumes DBTEXT-derived files without existence checksLocation:
Eventdisplay_AnalysisScripts_VTS/scripts/helper_scripts/ANALYSIS.evndisp_sub.sh:97-107,137-166After unpacking the DBTEXT tarball, the helper immediately
cats${TMP_LASERRUN}and parses${RUN}.runinfowithout checking that either file exists or is readable.Impact: Missing or corrupt DBTEXT tarballs silently propagate empty laser-run metadata or
wrong run dates into CVBF lookup and calibration selection.
C10 · HIGH · Broken TMVA bin-parsing regex in training and optimisation scripts
Location:
Eventdisplay_AnalysisScripts_VTS/scripts/IRF.trainTMVAforGammaHadronSeparation.sh:107-109;IRF.optimizeTMVAforGammaHadronSeparation.sh:91-93grep "^* ENERGYBINS"and the matchingsedexpressions are invalid or broken (*is nota valid regex anchor in this position).
Impact: Training and optimisation bin parsing can fail or misbehave, directly affecting
gamma/hadron separation products.
C11 · MEDIUM · Cut optimisation continues with missing effective-area input
Location:
Eventdisplay_AnalysisScripts_VTS/scripts/IRF.optimizeTMVAforGammaHadronSeparation.sh:81-83Logs an error on missing effective-area file but does not exit.
Impact: Optimisation jobs may run with incomplete inputs, producing invalid cut tuning.
C12 · MEDIUM · DL3 jobs install Python dependencies at runtime per job
Location:
Eventdisplay_AnalysisScripts_VTS/scripts/helper_scripts/ANALYSIS.v2dl3_sub.sh:56-58Each job activates the base environment and runs
pip install -e ....Impact: Slow, non-reproducible, and environment-dependent production.
C13 · MEDIUM ·
ANALYSIS.anasum_combine.shlacks a completion/integrity guard before mergeLocation:
Eventdisplay_AnalysisScripts_VTS/scripts/ANALYSIS.anasum_combine.sh:31-36,85-115Assumes parallel jobs are finished; no strong completeness validation before combining.
Impact: Partial combined outputs can be produced silently.
C14 · MEDIUM · Data-side
mscw_energyselects IRF products purely from filename templatesLocation:
Eventdisplay_AnalysisScripts_VTS/scripts/helper_scripts/ANALYSIS.mscw_energy_sub.sh:120-179,182-188The helper derives the IRF table path and DispBDT directory from
printRunParameteroutput andstring templates, then runs
mscw_energywithout verifying that the chosen XMLs and tablesmatch the input
evndisp.rootconfiguration.Impact: Stale or mismatched IRF products can be applied to otherwise valid
evndispfiles,yielding apparently normal
mscw.rootoutputs built with the wrong reconstruction tables.C15 · MEDIUM · DBTEXT run extraction is fragile against schema drift in
.runinfofilesLocation:
Eventdisplay_AnalysisScripts_VTS/scripts/db_scripts/db_run.sh:69-115,165-185db_run.shre-parses.runinfoheaders by name; if a header is absent, the helper falls backto empty or stale column indexes.
Impact: Downstream scripts can derive wrong time windows, source IDs, and DB queries from
superficially valid DBTEXT files.
C16 · MEDIUM · Existing DBTEXT tarballs suppress regeneration of laser-calibration side files
Location:
Eventdisplay_AnalysisScripts_VTS/scripts/db_scripts/db_run.sh:294-297If
<run>.tar.gzexists, the script skipsread_laser_calibrationentirely rather thanchecking whether
.laserrun,.gain,.toffset, and related files are current inside thetarball.
Impact: Stale or incomplete DBTEXT tarballs freeze outdated calibration metadata and prevent
later
evndispanalyses from refreshing it.C17 · MEDIUM · FITS converter crashes on missing DQM-like DBTEXT files before the
NonecheckLocation:
Eventdisplay_AnalysisScripts_VTS/scripts/db_scripts/db_write_fits.py:425-433convert_table_comment_to_ascii(table)is called fordqmfiles before checkingif table is not None.Impact: One missing DBTEXT component aborts FITS generation and drops all DQM metadata for
the run.
C18 · MEDIUM · DQM FITS combination requires identical schemas across all runs
Location:
Eventdisplay_AnalysisScripts_VTS/scripts/db_scripts/db_combine_dqm_fits.py:40-54Uses
vstack(..., join_type="exact"), so any column drift makes the whole merge fail.Impact: Long-run DQM aggregation is brittle against minor schema evolution.
C19 · MEDIUM · DQM rate/dead-time extraction does not guard malformed scaler intervals
Location:
Eventdisplay_AnalysisScripts_VTS/scripts/db_scripts/db_write_fits.py:101-138np.diff()and scaler divisions are performed with no protection against zerotime_diffsorzero
TenMHzScalerincrements.Impact: Invalid scaler records can inject
inf/NaNvalues into exported DQM quantities.C20 · LOW ·
ANALYSIS.v2dl3.shhas typoLOGIDRand undefined variableAFILELocation:
Eventdisplay_AnalysisScripts_VTS/scripts/ANALYSIS.v2dl3.sh:53-58,93-106,117-122Impact: Stale temp files and misleading job logs.
C21 · LOW ·
ANALYSIS.anasum_allcuts.shuses relative./ANALYSIS...callsLocation:
Eventdisplay_AnalysisScripts_VTS/scripts/ANALYSIS.anasum_allcuts.sh:66-76Impact: Fails unless launched from the scripts directory.
C22 · LOW · IRF packaging script has misspelled function name and undefined
$Fin loggingLocation:
Eventdisplay_AnalysisFiles_VTS/transfer/pack_irfs_for_ucla.sh:161-185Impact: Fragile packaging and misleading archive diagnostics.
D — IRF production chain
D1 · HIGH ·
IRF.production.shcan resubmit the full TMVA grid multiple timesLocation:
Eventdisplay_AnalysisScripts_VTS/scripts/IRF.production.sh:337-387The
TRAINTMVA/OPTIMIZETMVAblock contains inner loops that re-iterate over the fullEPOCHand
ATMOSsets while already inside outer loops over the same sets.Impact: TMVA training and optimisation jobs are submitted
len(EPOCH) × len(ATMOS)times,wasting batch capacity and mixing results.
D2 · HIGH · Inner TMVA block uses
${EPOCH:0:2}instead of${VX:0:2}for epoch selectionLocation:
Eventdisplay_AnalysisScripts_VTS/scripts/IRF.production.sh:353-366Inside the
for VX in $EPOCHloop, size-cut file grep and TMVA runparameter selection still usethe outer
${EPOCH:0:2}instead of the current iteration variable${VX:0:2}.Impact: Multi-epoch submissions select the wrong size-cut row and TMVA runparameter template
for all epochs beyond the first.
D3 · HIGH ·
IRF.mscw_energy_MC_sub.shquoted glob fails to copy uncompressed ROOT filesLocation:
Eventdisplay_AnalysisScripts_VTS/scripts/helper_scripts/IRF.mscw_energy_MC_sub.sh:133-136cp -v "${INDIR}/*[0-9].root" "$DDIR"— the glob is inside double quotes and is not expandedby the shell.
Impact: If upstream
evndispproducts are uncompressed.rootfiles, the MCmscw_energystage fails before building its file list.
D4 · HIGH · Job-launch scripts generate one filename and submit another
Location:
Eventdisplay_AnalysisScripts_VTS/scripts/IRF.generate_lookup_table_parts.sh:86-124;IRF.mscw_energy_MC.sh:97-139Both scripts write the runnable script to
$FSCRIPTbut submit$FSCRIPT.sh, pointing to anon-existent pathname.
Impact: Core IRF production stages — lookup-table and MC
mscw_energygeneration — failbefore creating their outputs, blocking all downstream steps.
D5 · HIGH · Gamma/hadron TMVA training ignores the documented UUID argument
Location:
Eventdisplay_AnalysisScripts_VTS/scripts/IRF.trainTMVAforGammaHadronSeparation.sh:18-19,46,60The help text places the optional UUID as argument 8, but the script reads it from
$12.Impact: Repeated training campaigns reuse the same submit/log directory, mixing products and
overwriting job state.
D6 · MEDIUM · Lookup-table combine checks the wrong path before overwriting
Location:
Eventdisplay_AnalysisScripts_VTS/scripts/IRF.combine_lookup_table_parts.sh:77-80Guard tests
[[ -f $OFILE ]]instead of[[ -f "$ODIR/$OFILE" ]].Impact: Existing combined lookup tables in the output directory are not reliably detected
before overwrite or merge attempts.
D7 · MEDIUM ·
zstdavailability checks are malformed in several IRF helpersLocation:
IRF.compress_evndisp_MC_sub.sh:46-54;IRF.evndisp_MC_sub.sh:323-330;IRF.lookup_table_parallel_sub.sh:67-79;IRF.mscw_energy_MC_sub.sh:137-149command -v zstd /dev/nullis not the standard existence test.Impact: Compression/decompression branches can report false missing-tool failures.
D8 · MEDIUM ·
IRF.evndisp_MC_sub.shpasses literal glob tozstdwhen no files matchLocation:
Eventdisplay_AnalysisScripts_VTS/scripts/helper_scripts/IRF.evndisp_MC_sub.sh:88-103The loop runs over
"$V4N"/*.root.zstwithout checking whether any files match; the literalpattern is then passed to
zstd.Impact: Spurious decompression failures during duplicate-processing detection.
D9 · MEDIUM ·
IRF.optimizeTMVAforGammaHadronSeparation_sub.shusesMVADIRbefore definitionLocation:
Eventdisplay_AnalysisScripts_VTS/scripts/helper_scripts/IRF.optimizeTMVAforGammaHadronSeparation_sub.sh:54-69rm -f ${MVADIR}/rates.logexecutes beforeMVADIRis assigned later in the file.Impact: The cleanup step targets the wrong directory, potentially using an inherited
environment value.
D10 · MEDIUM ·
IRF.production.shmixes robust$(dirname "$0")/...calls with fragile./IRF...callsLocation:
Eventdisplay_AnalysisScripts_VTS/scripts/IRF.production.sh:311,319,330,372,379Impact: IRF production fails if launched from outside the scripts directory.
D11 · MEDIUM ·
makeEffectiveAreaIRF resolution duplication is order-dependent in C++Location:
EventDisplay_v4/src/makeEffectiveArea.cpp:162-208(See also B7.) IRF products are duplicated by index and only work if the source IRF was
initialised earlier in the vector.
Impact: Any refactor changing internal IRF ordering silently breaks resolution product
generation.
D12 · LOW · TMVA gamma/hadron preselection applies an unjustified 10× random retention factor
Location:
EventDisplay_v4/src/trainTMVAforGammaHadronSeparation.cpp:143-185After computing the required retained fraction, the code multiplies it by
10.before cappingat unity.
Impact: Makes the effective preselection behaviour opaque and bin-statistics dependent.
E — XGB / ML integration
E1 · HIGH · XGB data-analysis step not triggered automatically before XGB
anasumcutsLocation:
Eventdisplay_AnalysisScripts_VTS/scripts/ANALYSIS.dispXGB.sh:11-95;ANALYSIS.anasum_allcuts.sh:6-10,57-71;ANALYSIS.anasum_parallel_from_runlist.sh:133-145ANALYSIS.dispXGB.shexists as a standalone script, and theanasumwrappers now expose XGBcut aliases (
moderate2telXGB, etc.), but the documented workflow goes straight fromANALYSIS.mscw_energy.shtoanasum. The wrappers do not triggerANALYSIS.dispXGB.shautomatically before selecting
*-XGB-BDTcuts.Impact: Users can request XGB cuts while analysing plain
mscwfiles with no.xgb_stereo.root/.xgb_gh.rootsidecars, making XGB analyses a fragile manual side-bandinstead of a reproducible pipeline stage.
E2 · HIGH ·
IRF.mscw_energy_MC_sub.shXGB branch uses stale CLI and wrong run-parameter keysLocation:
Eventdisplay_AnalysisScripts_VTS/scripts/helper_scripts/IRF.mscw_energy_MC_sub.sh:237-271,322-347;Eventdisplay-ML/src/eventdisplay_ml/config.py:175-230;EventDisplay_v4/src/VInstrumentResponseFunctionRunParameter.cpp:249-263The helper calls
eventdisplay-ml-apply-xgb-stereo --input-file --model-dir --output-file, butthe current CLI expects
--input_file --model_prefix --output_file. It also writes* XGBFILESUFFIXinto the effective-area parameter file, whileEventDisplay_v4now parsesonly
XGBSTEREOFILESUFFIXandXGBGAMMAHADRONFILESUFFIX.Impact: The legacy in-subscript XGB branch cannot drive the current Python CLI or pass its
outputs to the current C++ consumer; any workflow relying on this path is broken or silently
stale.
E3 · HIGH · XGB stereo training ignores the requested reconstruction ID
Location:
Eventdisplay_AnalysisScripts_VTS/scripts/IRF.trainXGBforAngularReconstructionBinned.sh:48-56,76The script accepts
<Rec ID>as a required argument and carries it through naming and logging,but hard-codes the training input directory to
.../MSCW_RECID0_DISP.Impact: Training jobs submitted for reconstruction IDs other than 0 silently use wrong MC
products.
E4 · HIGH · Non-XGB effective-area jobs still probe XGB sidecar files
Location:
EventDisplay_v4/src/VStereoAnalysis.cpp:1965-1972;EventDisplay_v4/src/CData.cpp:28-38,1073-1127;logFiles/irfs/20deg_0.5wob_NOISE200.effarea.log:21-23CDataopens XGB friend files from non-empty suffix strings before checking whether the activereconstruction methods require them. Confirmed in a production IRF log with
ENERGYRECONSTRUCTIONMETHOD 0and a TMVA preselection cut file, yet the job still tries to open.xgb_stereo.root/.xgb_gh.root.Impact: Non-XGB jobs emit avoidable file-open errors and retain hidden dependencies on
artifacts that are not required.
E5 · HIGH · Eventdisplay-ML classification training and application can use different feature sets
Location:
Eventdisplay-ML/src/eventdisplay_ml/data_processing.py:807-817;Eventdisplay-ML/src/eventdisplay_ml/models.py:441-452Classification training switches to
features_tmva_style(...)whentmva_style=True, but theapplication path always uses
features.features(analysis_type, training=False)and does notmirror that branch.
Impact: A TMVA-style-trained classification model can be evaluated with the wrong input
feature set.
E6 · HIGH · Eventdisplay-ML
config.pycalls.get()on a listLocation:
Eventdisplay-ML/src/eventdisplay_ml/config.py:162-165configure_training()usesmodel_parameters.get("energy_bins_log10_tev", []).get("E_min");energy_bins_log10_tevis list-structured and a list does not have.get().Impact: Classification training/apply configuration can fail with
AttributeErroronvalid-looking parameter files in certain call paths.
E7 · MEDIUM · XGB training helper scratch directory uses undefined variable
MSCW_FILELocation:
Eventdisplay_AnalysisScripts_VTS/scripts/helper_scripts/IRF.trainXGBforAngularReconstruction_sub.sh:15-18TEMPDIRis derived from$(basename $MSCW_FILE .root), butMSCW_FILEis never defined; theactual input is the list file
LLIST.Impact: Scratch-directory naming is accidental and environment-dependent, causing collisions
in batch production and making failed training jobs harder to debug.
E8 · MEDIUM ·
EVNDISP_MLSYSdocumented but helpers require a hard-coded conda environment nameLocation:
Eventdisplay_AnalysisScripts_VTS/scripts/helper_scripts/ANALYSIS.dispXGB_sub.sh:9-10,27-53;IRF.dispXGB_sub.sh:8-9;IRF.trainXGBforAngularReconstruction_sub.sh:9-10;IRF.trainXGBforGammaHadronSeparation_sub.sh:13-14;README.md:16-18,22-29The public setup documents
EVNDISP_MLSYS, but the helpers ignore it and require a pre-existingconda environment named
eventdisplay_ml.Impact: A user can satisfy all documented environment variables and still fail at runtime.
E9 · MEDIUM · XGBoost hyperparameters are hard-coded in Python, not versioned in auxiliary parameter files
Location:
Eventdisplay-ML/src/eventdisplay_ml/hyper_parameters.py:9-45n_estimators,learning_rate,max_depth,subsample,colsample_bytree, etc. are embeddedin code; the shipped JSON files define only binning and
tmva_style.Impact: The effective ML model is not fully described by the analysis parameter files,
weakening reproducibility.
E10 · MEDIUM · Production XGB training helpers omit
--random_stateand--hyperparameter_configLocation:
IRF.trainXGBforGammaHadronSeparation_sub.sh:59-67;IRF.trainXGBforAngularReconstruction_sub.sh:55-61The helpers pass
train_test_fractionand event limits but not a fixed random seed or explicithyperparameter file.
Impact: Routine retraining is not strictly reproducible even when input event lists are
unchanged.
E11 · MEDIUM · XGB command names are inconsistent between production scripts and documentation
Location:
IRF.generalproduction.sh:18-29;IRF.production.sh:24-31,392-403;docs/changes/124.feature.md:1-10;README.md:83-159IRF.generalproduction.shadvertisesTRAINXGBANGRESBINNED;IRF.production.shimplementsTRAINXGBANGRES; the change note references a non-existentIRF.trainXGBforAngularReconstruction.sh; the main README omits the XGB production sequenceentirely.
Impact: Operators are left with conflicting command names and incomplete instructions.
E12 · MEDIUM · Legacy
TMVA.BDT.V4.runparameterships a malformedPREPARE_TRAINING_OPTIONSlineLocation:
Eventdisplay_AnalysisFiles_VTS/ParameterFiles/TMVA.BDT.V4.runparameter:68-76;EventDisplay_v4/src/VTMVARunData.cpp:423-435The line lacks the leading
*required for parsing and contains a double::in the optionstring.
Impact: Legacy V4 BDT configuration does not reliably describe the training setup it
appears to encode.
E13 · LOW · TMVA training options have drifted from the accepted backend behaviour
Location:
logFiles/irfs/mvaAngRes_20deg-BDTDisp-Tel2.log:81-82NegWeightTreatment=InverseBoostNegWeightsis not supported forBoostType=Gradand is silentlyreplaced by
Pray.Impact: Effective training configuration no longer matches the nominal requested one.
F — Auxiliary files & configuration
F1 · HIGH · Epoch metadata and downloadable IRF manifests are out of sync
Location:
Eventdisplay_AnalysisFiles_VTS/ParameterFiles/VERITAS.Epochs.runparameter:47-53;Eventdisplay_AnalysisFiles_VTS/transfer/file_list_V6.dat:304-358VERITAS.Epochs.runparameterdefinesV6_2024_2024s,V6_2024_2025w,V6_2025_2025s, andV6_2025_2026w, but the shipped V6 download manifest stops atV6_2023_2024w.Impact: A fresh checkout following the README cannot fetch the IRF tarballs needed for the
newer epochs that the active scripts may select at runtime.
F2 · HIGH · Future epoch
V6_2025_2025sis defined twice in the epoch fileLocation:
Eventdisplay_AnalysisFiles_VTS/ParameterFiles/VERITAS.Epochs.runparameter:48-53The second entry covers
2026-05-01 to 2026-11-24while a TODO states it should becomeV6_2026_2026s.Impact: Future epoch lookup becomes ambiguous and can map runs to the wrong gain/atmosphere
configuration.
F3 · HIGH · Documented download path does not reproduce the full current auxiliary-file set
Location:
Eventdisplay_AnalysisFiles_VTS/README.md:10-19;transfer/file_list_V6.dat:1-45,170-186;auxiliary_files.txt:19632-19645,20236-20262The download manifests expose only classic DispBDTs and effective areas; the fully unpacked
inventory in
auxiliary_files.txtadditionally contains aDispXGBtree and XGBeffective-area products.
Impact: Two users following the official instructions end up with materially different
auxiliary installations.
F4 · HIGH · Unpacked auxiliary inventory is effectively AP-only despite scripts advertising multi-type support
Location:
Eventdisplay_AnalysisScripts_VTS/README.md:58-81;IRF.production.sh:91-120;auxiliary_files.txt:19629-19631,21112-21169Scripts and documentation present
AP,NN,TS, andCCas supported analysis types; theV6 file list even advertises
DispBDTs_NN_*andEffectiveAreas_nominalHV_NN_*tarballs. Theunpacked inventory shows only an empty
DispBDTs/NNdirectory; all table and effective-arearoots are
-AP.rootproducts.Impact: The supposedly complete auxiliary area cannot support NN (or TS/CC) analyses.
F5 · HIGH · XGB instrument-file coverage is patchy while wrappers expose XGB as a generic AP mode
Location:
auxiliary_files.txt:19632-19645,19697,19734,19933,20010,20177,20236-20262Most XGB model directories are
ATM61only; XGB effective areas are confined toV6_2016_2017forATM61/62, far narrower than the generic XGB hooks now exposed by the APwrappers.
Impact: The auxiliary-file area does not yet support XGB as a broadly deployable production
mode across the same epoch space as the classic AP products.
F6 · MEDIUM · XGB effective-area naming is internally inconsistent
Location:
auxiliary_files.txt:20245-20248,20698;ANALYSIS.anasum_parallel_from_runlist.sh:133-145The same tree contains both
...ModerateXGB...rootand...Moderate-XGB-BDT...rootnames;the analysis wrapper resolves
moderate2telXGBtoNTel2-PointSource-Moderate-XGB-BDT.Impact: Shell wrappers, cut files, and effective-area selection logic can disagree on which
product family is canonical.
F7 · MEDIUM · Maintainer README is stale and contradicts the transfer scripts
Location:
Eventdisplay_AnalysisFiles_VTS/README.md:29-30,77-89;transfer/pack_irfs_for_ucla.sh:11;transfer/put_irfs_to_ucla.sh:27The README documents a non-existent
EVNDISP.reconstruction.runparameter, refers genericallyto
tar_packages, and instructs maintainers to move packages toarchivebefore calling anupload script that only scans
tar_packages_${VERSION}/.Impact: Release maintenance instructions are no longer self-consistent.
F8 · MEDIUM ·
download_irfs.shcontinues into extract/remove after a failed transferLocation:
Eventdisplay_AnalysisFiles_VTS/download_irfs.sh:36-49tar --keep-newer-filesandrm -vexecute immediately after the copy/bbftpcall withoutchecking the transfer exit status.
Impact: Failed or partial downloads trigger extraction and cleanup, making recovery harder.
F9 · MEDIUM ·
copy_dispBDTs.shandcopy_dispXGB.shsilently overwriteSIMTYPEin a loopLocation:
Eventdisplay_AnalysisFiles_VTS/DispBDTs/copy_dispBDTs.sh:11-19;DispXGB/copy_dispXGB.sh:11-18Both scripts assign
SIMTYPErepeatedly, so only the last hard-coded value survives.copy_dispBDTs.shalso prints${IRVERSION}instead of${IRFVERSION}.Impact: IRF-maintenance copies silently pull the wrong model family with misleading
provenance information.
F10 · LOW · V4/V5 gamma/hadron copy script has dead atmosphere-selection code
Location:
Eventdisplay_AnalysisFiles_VTS/GammaHadronBDTs/copy_GammaHadron_V4_V5_BDTs.sh:21-31Loop iterates over
ATM21 ATM22but the branch condition checks forATM62; thesummer/winter split is never exercised.
Impact: Any future atmosphere-specific V4/V5 handling based on this template will be wrong.
F11 · LOW · Global DB fallback is present in text only, not in parsed configuration
Location:
Eventdisplay_AnalysisFiles_VTS/ParameterFiles/EVNDISP.global.runparameter:9-16Only the
romulus.ucsc.eduline is active (*-prefixed); theremus.ucsc.edufallback isunstarred and ignored.
Impact: No real failover at the parameter-file level if the primary DB endpoint is
unavailable.
F12 · LOW · Backup content mixed into the main instrument-file tree
Location:
auxiliary_files.txt:19298A
DispBDTs/AP.backsubtree sits alongside nominal production content.Impact: Ambiguity about which products are authoritative; risk of copying the wrong tree.
G — Release tests
G1 · HIGH ·
EventDisplay_Release_v492is still a skeleton, not an auditable release-test recordLocation:
EventDisplay_Release_v492/README.md:1-3The checkout contains only metadata plus
runparameter/V6.AP.runparameter.dat; none of theexpected result trees (
Crab/,mc_data_comparison/,irf_plotting/,SourceTests/,AP/radialAcceptances/) are present.Impact: The
v492artifact does not provide the evidence needed to review the release teststhe codebase expects to exist.
G2 · HIGH · Release-test output naming is inconsistent
Location:
EventDisplay_ReleaseTests_code/README.md:10-15;release_tests/montecarlo/energythresholds/plot.C:150-152Most scripts use
EventDisplay_Release_<version>, but the README points toEventDisplay_ReleaseTests_v492and the energy-threshold macro writes toEventDisplay_ReleaseTests_<version>/energythresholds/.Impact: Release products are scattered between differently named trees, making results look
missing and breaking documentation links.
G3 · HIGH · MC/data comparison applies the summer-atmosphere filter with an undefined variable
Location:
EventDisplay_ReleaseTests_code/release_tests/montecarlo/mc_data_comparison/compareDatawithMC.sh:138-146The winter branch checks
${atm}, but the summer branch checks${A}beforeAis assigned.Impact: Summer-epoch filtering silently uses the wrong atmosphere combinations, weakening one
of the main release-validation checks.
G4 · HIGH · Crab run-list generation:
WOBBBtypo leaves zero-wobble classification undefinedLocation:
EventDisplay_ReleaseTests_code/release_tests/sources/Crab/runlist_generator_from_anasum_log.sh:127-132,156-165When both wobble offsets are zero, the script assigns
WOBBB="0"instead ofWOBB="0", butlater decisions key on
WOBB.Impact: Zero-wobble runs inherit a stale wobble state and are grouped into the wrong run
lists, contaminating downstream source-test summaries.
G5 · HIGH · Source-test plotting macros are pinned to the
v491library and data layoutLocation:
release_tests/sources/Crab/plot_lightcurves.C:19;plot_energy_spectra.C:20;plot_skymaps.C:17;plot_sensitivity.C:11,27-40;plot_sensitivity_compare_cuts.C:16,44-62These macros hard-code
/.../EventDisplay_v491-al9/lib/libVAnaSum.soand for sensitivity plotshard-code
v491/v490result directories.Impact: Running on
v492products loads the wrong code and compares against fixed historicalpaths.
G6 · MEDIUM · IRF plotting is hard-coded to a single slice instead of broad parameter coverage
Location:
EventDisplay_ReleaseTests_code/release_tests/montecarlo/irf_plotting/irf_plotting.sh:16-25,57,75-89The driver hard-codes
COMPAREVERSION="v492", a fixed sim/cut family, and only runs the bodyfor
V6_2016_2017/ATM61.Impact: The nominal IRF plotting step no longer spans the stated parameter space and is weak
as a release-regression check.
G7 · MEDIUM · Shipped
v492run-parameter format encodes a fragile cut-name dualityLocation:
EventDisplay_Release_v492/runparameter/V6.AP.runparameter.dat:35-39;release_tests/sources/Crab/anasum_from_runlists.sh:34-40Each
* CUTline stores both a full IRF cut identifier and a short alias; consumers rely on thefourth field specifically.
Impact: Easy for future scripts to read the wrong cut token and silently select the wrong
inputs.
H — Performance & efficiency
H1 · HIGH ·
evndispkeeps time-sliced pedestal analysis enabled during normal data reconstructionLocation:
EventDisplay_v4/src/VEventLoop.cpp:490-495,1442-1461;EventDisplay_v4/src/VPedestalCalculator.cpp:30-199,280-381;logFiles/analysis/64080.log:71,598-599On the exercised run,
evndispprocesses ~205 events/s (543 481events in 44:04). The activepath still performs pedestal-variance bookkeeping for every telescope during standard
reconstruction:
VPedestalCalculator::doAnalysis()loops over hit channels, recomputes tracesums for every summation window, and fills per-pixel histograms.
Impact: A calibration-style side analysis remains in the hottest reconstruction loop and is a
plausible first-order contributor to the 16× throughput gap between
evndisp(~205 ev/s) andmscw_energy(~3 229 ev/s).H2 · MEDIUM ·
evndisprepeats invariant low-gain/sum-window validation in the per-event loopLocation:
EventDisplay_v4/src/VEventLoop.cpp:1204-1292The analysis path rechecks availability of low-gain multipliers for
sumwindow_1,sumwindow_2, andsumwindow_pass1on every event for every telescope, even though these arerun-level constants after initialisation.
Impact: Avoidable branch-heavy work repeated millions of times in long runs.
H3 · HIGH ·
anasumdouble-scansmscw.rootand reinitialises TMVA twice per runLocation:
EventDisplay_v4/src/VAnaSum.cpp:464-465;EventDisplay_v4/src/VStereoAnalysis.cpp:264-349,434-448;EventDisplay_v4/src/VTMVAEvaluator.cpp:366-511;logFiles/analysis/64080.anasum.log:97-103,237-243In the reflected-background path, ON and OFF both point to the same
mscw.root, yetVAnaSumruns separate passes. Each pass reads
174 937entries and rebuilds the gamma/hadron TMVAreaders from disk.
Impact: For every run,
anasumroughly doubles tree I/O and repeats expensive TMVA andeffective-area initialisation that could be shared.
H4 · MEDIUM ·
anasumuses a linear TMVA-bin search with recomputed mean energies per eventLocation:
EventDisplay_v4/src/VTMVAEvaluator.cpp:714-745,802-839Each gamma/hadron evaluation calls
getDataBin(), which scans all loaded bins and recomputesthe mean energy of each candidate before evaluating the selected reader.
Impact: Avoidable per-event classifier overhead in the inner
anasumloop, especially withmany energy/zenith bins.
H5 · MEDIUM · Multi-run
anasumjobs scale as O(runs × total entries in chain)Location:
EventDisplay_v4/src/VAnaSum.cpp:464-465;EventDisplay_v4/src/VStereoAnalysis.cpp:434-450fillHistograms()loops over all chain entries and then filters byrunNumber == irun; eachrun-specific pass re-reads entries from all other runs.
Impact: Runtime can grow toward O(runs × total_entries) instead of near-linear.
H6 · MEDIUM ·
evndispreads pedestal calibration inputs redundantly during initialisationLocation:
logFiles/analysis/64080.log:247-252,263-268,275-280,291-296The same telescope-level pedestal products are read twice in succession during calibration setup.
Impact: Duplicated calibration I/O in the slowest stage of the data pipeline.
H7 · MEDIUM ·
mscw_energyrepeatedly rebuilds TMVA datasets instead of amortising setupLocation:
logFiles/analysis/64080.mscw.log:156-169;logFiles/irfs/20deg_0.5wob_NOISE200.mscw.log:261-277Both data-side and MC-side logs contain repeated
Rebuilding Dataset Defaultmessages.Impact: Avoidable setup churn, expensive when multiplied across large IRF productions.
Cross-cutting issues (confirmed from logs)
X1 · HIGH ·
v2dl3silently sets quality flag to 0 when Eventdisplay logfile is missingLocation:
logFiles/analysis/64080.v2dl3.log:14v2dl3reports "Eventdisplay logfile not found in anasum root file. Quality flag set to 0."Impact: Downstream DL3 products lose quality metadata without a loud failure.
X2 · MEDIUM ·
anasumcontinues under missing TMVA-weight warningsLocation:
logFiles/analysis/64080.anasum.log:243-249,541-547;EventDisplay_v4/src/VTMVAEvaluator.cpp:190-279VTMVAEvaluatorreports "TMVA root file not found or incomplete file (0)" and continues;see also B4.
Impact: Incomplete classifier inputs survive initialisation silently.
X3 · MEDIUM ·
anasumreinitialises the same TMVA state twice, doubling warning noiseLocation:
logFiles/analysis/64080.anasum.log:237-247,535-545See also H3. The same missing/incomplete low-energy TMVA file warning is emitted once for ON
and once again for OFF.
Impact: Duplicated warnings make it harder to distinguish a new problem from a repeated one.
X4 · LOW · Data run contains large numbers of GPS-faulted events; pipeline continues without escalation
Location:
logFiles/analysis/64080.log:565-580,587-596Missing telescope events and GPS faults on telescopes 1 and 4 are logged and skipped.
Impact: Run-quality issues are not escalated; pipeline favours "continue with warnings" over
explicit run-quality gating.
Hyperparameter locations (reference)
Eventdisplay_AnalysisFiles_VTS/ParameterFiles/TMVA.BDT.runparameter:13-15Eventdisplay_AnalysisFiles_VTS/ParameterFiles/TMVA.BDTDisp.runparameter:1-2Eventdisplay-ML/src/eventdisplay_ml/hyper_parameters.py:9-45(hard-coded in Python)Eventdisplay_AnalysisFiles_VTS/ParameterFiles/XGB-stereo-parameter.json;XGB-classify-parameter.jsonPriority action list
Immediate (HIGH severity, silent wrong-physics or silent job failure)
ANALYSIS.anasum.shSUBC initialisation (anasum.sh:90-92) — jobs are silently never submitted (C1)get_disp_dir()to checkHVSETTINGS == "obsLowHV"instead of undefinedSIMTYPE(mscw_energy_sub.sh:136-152) — wrong DispBDT directory for all RedHV/UV runs at high zenith (C2)yimpvalidity check (VArrayAnalyzer.cpp:944-951) — NaN core-Y propagates into reconstruction (A6)VArrayAnalyzer.cpp:1241-1248) — wrong telescope pairs used in cut (A7)VCalibrationData.cpp:365-373;VCalibrator.cpp:899-903) — high-gain state corrupted (A4, A5)VInstrumentResponseFunctionData.cpp:472-485) — Xcore used twice, Ycore ignored (A12)trainTMVAforGammaHadronSeparation.cpp:255-266) — out-of-bounds vector access (A13)trainTMVAforGammaHadronSeparation.cpp:343-360) — classification factory used instead of regression (A14)IRF.generate_lookup_table_parts.sh:86-124;IRF.mscw_energy_MC.sh:97-139) — submitted path does not exist (D4)IRF.production.shdouble-loop and epoch variable (IRF.production.sh:337-387,353-366) — TMVA grid submitted multiple times with wrong epoch variable (D1, D2)IRF.mscw_energy_MC_sub.shquoted glob and stale XGB CLI (IRF.mscw_energy_MC_sub.sh:133-136,237-271) — files not copied; Python CLI mismatch (D3, E2)FORCEDATMOvia sed template (ANALYSIS.mscw_energy.sh:63) — ATM override is a no-op in batch (C3)mscw_energy_sub.shconsistent withprepare_atmo_string()— table lookup fails for V4/V5 GRISU data (C4)ANALYSIS.evndisp_sub.shDBTEXT file existence checks — missing files silently inject wrong metadata (C9)IRF.trainTMVAforGammaHadronSeparation.sh:107-109) — bin parsing fails (C10)VTraceHandler.cpp:783) —FADC[n]accessed on every event where the window reaches the rightmost search position; UB corrupts charge reconstruction (A17)VOnOff::doOnOffforParameterHistogramsbackground mutation (VOnOff.cpp:159-161) —hoff->ScaleafterAdddouble-scales background, corrupting reflected-region excess counts and significance inanasum(B11)VInstrumentResponseFunctionRunParametersilent key drop (VInstrumentResponseFunctionRunParameter.cpp:109-358) — add a terminalelse { cerr << "unknown key: " << temp << endl; }to catch parameter typos (B13)printRunParameterexit code on file-open failure (printRunParameter.cpp:330-335) — replaceexit(0)withexit(EXIT_FAILURE)so calling shell scripts detect errors (B14)Short-term (MEDIUM / HIGH, reproducibility and efficiency)
evndispanalysis loop — primary throughput bottleneck (H1)mscw.rootscan and double TMVA initialisation inanasum— doubles tree I/O (H3)tmva_style=Trueclassification (E5)config.py.get()on list —AttributeErroron valid parameter files (E6)ANALYSIS.dispXGB.shinto the defaultanasumworkflow, or add a guard that prevents*-XGB-BDTcut selection without XGB sidecars (E1)IRF.trainXGBforAngularReconstructionBinned.sh:48-56) (E3)CDatato the active reconstruction methods (E4)makeEffectiveArea"no problem" comment vs.exit(EXIT_FAILURE)mismatch (makeEffectiveArea.cpp:118;VInstrumentResponseFunctionRunParameter.cpp:437) — either remove the misleading comment or implement graceful fallback (B12)VTableLookupRunParametererror message (VTableLookupRunParameter.cpp:175) — change "2=read" to "0=read" (B15)VTraceHandler::getQuickMaxtmax initialisation (VTraceHandler.cpp:420-426) — initialisetmaxto 0 or the expected maximum before computingnMax(A18)