Missing intensity coefficients datasets cause NPE in SparkAffineFusion

[carshadi]

Steps to reproduce:

Run SparkIntensityMatching and IntensitySolver on a dataset where IllDefinedDataPointsException is thrown during the intensity matching stage for certain pairs of views (e.g., uniform/low‑variance regions, thresholds filtering most pixels).

SparkIntensityMatching parameters:

-x s3://aind-open-data/exaSPIM_794492_2026-01-09_16-50-40_processed_2026-01-17_14-44-16/tile_alignment/ip_split_affine_alignment/bigstitcher_affine.split.xml --numCoefficients 16,16,16 --renderScale 0.5 -o s3://aind-scratch-data/cameron.arshadi/794492-denoised-0.5_16-16-16_histogram2.n5/intensity --method HISTOGRAM

IntensitySolver parameters:

--xml "s3://aind-open-data/exaSPIM_794492_2026-01-09_16-50-40_processed_2026-01-17_14-44-16/tile_alignment/ip_split_affine_alignment/bigstitcher_affine.split.xml" --numCoefficients 16,16,16 --matchesPath "s3://aind-scratch-data/cameron.arshadi/794492-denoised-0.5_16-16-16_histogram2.n5/intensity" -o "s3://aind-scratch-data/cameron.arshadi/794492-denoised-0.5_16-16-16_histogram2.n5/coefficients.n5" --s3Region us-west-2

This yields 753 setups in the coefficients dataset s3://aind-scratch-data/cameron.arshadi/794492-denoised-0.5_16-16-16_histogram2.n5/coefficients.n5/ , whereas there are 800 setups in the xml.

After creating the fusion container, run SparkAffineFusion with --intensityN5Path ....

Observed behavior:

SparkAffineFusion throws a NullPointerException in CoefficientsIO.load because DatasetAttributes is null for missing coefficient datasets.

java.lang.NullPointerException: Cannot invoke "org.janelia.saalfeldlab.n5.DatasetAttributes.getNumDimensions()" because "attr" is null
  at net.preibisch.mvrecon.process.fusion.intensity.CoefficientsIO.load(CoefficientsIO.java:67)
  at net.preibisch.mvrecon.process.fusion.intensity.IntensityCorrection.readCoefficients(IntensityCorrection.java:259)
  at net.preibisch.bigstitcher.spark.SparkAffineFusion.lambda$... (SparkAffineFusion.java:554)

Why this happens:

IntensitySolver only writes coefficients for views present in the solved map. Views with no valid pairwise matches never enter the solver graph, so their coefficient datasets are never created.
Later, SparkAffineFusion assumes the dataset exists for every overlapping view and calls readCoefficients(...) unconditionally, which fails when the dataset is missing.

Expected behavior:

Fusion should proceed by treating missing coefficients as identity (no correction), rather than crashing.

Proposed fix:

In IntensitySolver, write identity coefficients for any view missing from the solved map (ensures all view setups are present in the output N5).
In SparkAffineFusion, guard coefficient loading by checking datasetExists, and fall back to identity when missing (avoid NPE for existing coefficient N5s).

I implemented this in main...carshadi:BigStitcher-Spark:fix-missing-intensity-coefficients and it seems to work.

The result is consistent with previous successful runs. Neuroglancer link

Let me know if I should open a PR.

Thanks!