Central management system for the ASAP CRN Cloud data infrastructure. Python scripts maintain the source-of-truth archives for ASAP CRN Cloud entities: Datasets, Collections, Releases, and Common Data Elements (CDE).
| Repository | Purpose |
|---|---|
| ASAP-CRN/cloud-datasets | Source-of-truth archive for all team-contributed datasets |
| ASAP-CRN/cloud-collections | Curated collections of datasets, versioned for VWB Data Collections |
| ASAP-CRN/cloud-releases | Release records tying datasets and collections to versioned snapshots |
| ASAP-CRN/cloud-cde | Common Data Element definitions and versioning |
Datasets follow a versioned lifecycle tied to Zenodo DOIs:
- WIP — new dataset is staged in
WIP/with aversionfile and optional stubdataset.json - Acceptance —
dataset.jsonis created with a concept DOI (v0.1), buckets, and metadata - First release — dataset version bumped to
v1.0, DOI files written, archive snapshot created - Updates — major/minor version bumps produce new archive snapshots and updated DOI files
Regular releases publish newly accepted or updated datasets:
- Urgent releases — newly accepted, uncurated datasets
- Minor/Major releases — updated or newly curated datasets, typically organized into Collections
Each release record enumerates which datasets (and dataset versions) and collections belong to it, along with the CDE version in effect.
Release and dataset management scripts live in scripts/. Key scripts:
| Script | Purpose |
|---|---|
sync_cloud_datasets.py |
Validate/repair version files, ensure archive completeness, rebuild datasets.json |
make_v<X>_release.py |
Compose and write release artifacts for a given release version |
add_v<X>_release_DOIs.py |
Attach Zenodo DOIs after deposition for a given release |
The asap_orchestrator Python package (src/asap_orchestrator/) provides:
models.py— Pydantic models (Dataset,DatasetBuckets,Creator,ReleaseRecord) that define and validatedataset.jsonartifactsrelease.py— release composition helpersutil.py— shared utilities
from asap_orchestrator.models import Dataset
ds = Dataset.load("datasets/hafler-pmdbs-sn-rnaseq-pfc")
ds.save("datasets/hafler-pmdbs-sn-rnaseq-pfc")datasets.json # Master index of all datasets (rebuilt by sync_cloud_datasets.py)
WIP/ # Staging area for in-progress datasets
└── <dataset-name>/
├── dataset.json # Stub metadata (optional)
└── version # Target version string
datasets/
└── <dataset-name>/ # format: <team>-<tissue>-<modality>[-<qualifier>]
├── dataset.json # Canonical metadata: DOI, GCS buckets, releases, CDE version
├── version # Current version string (source of truth)
├── DOI/ # Zenodo deposition files for current version
│ ├── <dataset-name>.json
│ ├── dataset.doi # Concept DOI (all versions)
│ ├── version.doi # Version-specific DOI
│ └── deposition.json
├── refs/ # Reference files for current version
└── archive/ # Immutable snapshots of past versions
└── <version>/
├── dataset.json # Version snapshot
├── DOI/
└── refs/
collections.json # Master index of all collections
└── <collection-name>/
├── collection.json # Metadata: DOI versions, datasets per release version
└── archive/<version>/
└── collection.json # Version-specific metadata snapshot
releases.json # Master index of all releases
└── <release-version>/
├── release.json # Snapshot: all datasets, new_datasets, collections, CDE version
└── *README*.pdf # Release-specific README
cdes.json # Index of all CDE versions
└── <cde-version>/
├── cde.json # CDE date, version, list of tables
└── cde.csv # Snapshot CDE schema table
The bootstrap/ directory contains scripts, tools, and templates used to create the historical (pre v4.0.1) archive of Datasets, Collections, and Releases.