Uploading the same Zarr assets to different embargoed Dandisets requires data transfer both times

[kabilar]

Hi @yarikoptic, I uploaded a set of Zarr assets to Dandiset 1602 and am now uploading the same assets to Dandiset 1411. The terminal output seems to suggest that data is being transferred again during the upload to the second Dandiset. My understanding is that if the assets already exist on the Archive and they are being uploaded again, then a link would just be created to the existing assets from the second Dandiset and actual data transfer would not occur. Thanks.

Terminal output:

03:04 $ DANDI_DEVEL=1 dandi upload --allow-any-path --validation ignore .
2025-09-19 03:05:16,249 [    INFO] Found 15 files to consider
PATH                                                   SIZE     ERRORS   PROGRESS STATUS                  MESSAGE                  
dandiset.yaml                                          1.3 kB                     skipped                 should be edited online  
...b-I74/micr/sub-I74_sample-IC_acq-overview_XPCT.json 77 Bytes   0               done                                             
...4/micr/sub-I74_sample-IC_acq-overview_XPCT.ome.zarr 89.9 GB    0           42% uploading                                        
.../micr/sub-I74_sample-IC_acq-zoom_chunk-01_XPCT.json 74 Bytes   0               done                                             
...r/sub-I74_sample-IC_acq-zoom_chunk-01_XPCT.ome.zarr 277.4 GB   0            1% uploading                                        
.../micr/sub-I74_sample-IC_acq-zoom_chunk-02_XPCT.json 74 Bytes   0               done                                             
...r/sub-I74_sample-IC_acq-zoom_chunk-02_XPCT.ome.zarr 341.1 GB   0            0% uploading                                        
.../micr/sub-I74_sample-IC_acq-zoom_chunk-03_XPCT.json 74 Bytes   0               done                                             
...r/sub-I74_sample-IC_acq-zoom_chunk-03_XPCT.ome.zarr 134.7 GB   0           17% uploading                                        
...I74/micr/sub-I74_sample-hemi_acq-overview_XPCT.json 77 Bytes   0               done                                             
...micr/sub-I74_sample-hemi_acq-overview_XPCT.ome.zarr 274.5 GB   0            4% uploading                                        
...icr/sub-I74_sample-hemi_acq-zoom_chunk-01_XPCT.json                                                                             
...sub-I74_sample-hemi_acq-zoom_chunk-01_XPCT.ome.zarr                                                                             
...icr/sub-I74_sample-hemi_acq-zoom_chunk-02_XPCT.json                                                                             
...sub-I74_sample-hemi_acq-zoom_chunk-02_XPCT.ome.zarr                                                                             
Summary:                                               1.1 TB          101.4 MB/s 1 skipped               1 should be edited online
                                                                                  5 done                                           
                                                                                  5 uploading 

cc @satra

[yarikoptic]

I was alerted to this issue by https://github.com/con/solidation reports as one of the "5 random open issues to fix" ;)
FWIW, it is "by design" ATM, since we do not support any kind of "branching" or "CoW" on Zarrs which are mutable. Our potential approach via

• Design doc for Zarr versioning/publishing support via Zarr Manifest Files dandi-archive#1892

I do not think would elevate that, unless we explicitly code for it as e.g. allowing explicit released/versioned zarr assets "upload" (would be quick since no data upload really) thus simply reusing existing manifests, but disallowing changes to zarrs. Something to indeed consider. Do you think it would suffice for your use case?