Add data integrity tools (checksum manifest builder and comparator), update README, and modernize package

CLAUDE.md

@@ -0,0 +1,67 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+
+`pds.pdc-cloud-tools` provides CLI tools and utilities for accessing and managing Planetary Data Cloud (PDC) resources on AWS S3, part of the NASA Planetary Data System (PDS) ecosystem. The package uses Python namespace packages under `pds.pdc`.
+
+## Development Setup
+
+```bash
+pip install --editable '.[dev]'
+pre-commit install
+pre-commit install -t pre-push
+pre-commit install -t prepare-commit-msg
+pre-commit install -t commit-msg
+```
+
+## Commands
+
+### Testing
+```bash
+pytest                          # run all tests
+pytest tests/path/to/test.py    # run single test file
+ptw                             # watch mode
+tox -e py39                     # run tests via tox (parallel with coverage)
+```
+
+### Linting
+```bash
+tox -e lint                     # run all linters via pre-commit
+pre-commit run --all            # run pre-commit hooks directly
+```
+
+### Build
+```bash
+python setup.py sdist bdist_wheel
+```
+
+## Code Style
+
+- Line length: 120 characters (Black + Flake8)
+- Docstrings: Google style (enforced by pydocstyle via flake8-docstrings)
+- Import order: enforced by `reorder-python-imports` pre-commit hook
+- Type annotations: mypy runs on `src/` at pre-commit time
+
+## Architecture
+
+The source lives under `src/pds/pdc/` using PDS namespace packaging. CLI entry points are registered in `setup.cfg`:
+
+- **`pdc-s3-download`** (`s3_download.py`) — Downloads S3 objects filtered by prefix, preserving directory structure. Accepts `--source-profile`, `--source-bucket`, `--source-prefix`, `--local-dest-dir` (also reads `AWS_PROFILE` and `AWS_BUCKET` env vars). Optional `--start-datetime` and `--end-datetime` (UTC, `YYYY-MM-DD` or `YYYY-MM-DDTHH:MM:SS`) filter by last-modified time.
+
+- **`pdc-inventory-summary`** (`inventory_summary.py`) — Summarizes S3 inventory files (gzipped CSV). Reads all `.gz` files in a directory tree and reports total object count and cumulative size. Assumes object size is in CSV column index 2.
+
+- **`data_integrity/`** — Data integrity verification for cloud migrations:
+  - `build_s3_checksum_manifest.py` — Generates a CSV manifest of S3 objects with checksums (prefers CRC64NVME, falls back to others). Supports `--resume-from` for incremental runs. Outputs: `bucket, key, size, checksum_algorithm, checksum_type, checksum_value, etag`.
+  - `compare_s3_manifests.py` — Compares two manifests by matching `(size, checksum_type, checksum_value)` tuples. Identifies objects in OLD manifest missing from NEW, and flags weak checksums. Returns exit code 1 if any OLD objects are unmatched.
+
+## CI/CD
+
+GitHub Actions workflows in `.github/workflows/`:
+- `branch-cicd.yaml` — runs on feature branches
+- `unstable-cicd.yaml` — runs on `main` pushes, publishes SNAPSHOT to Test PyPI via NASA-PDS Roundup Action
+- `stable-cicd.yaml` — runs on `release/*` branches for stable releases
+- `terraform_cicd.yaml` — infrastructure deployment
+
+Secrets required by CI: `ADMIN_GITHUB_TOKEN`, `TEST_PYPI_USERNAME`, `TEST_PYPI_PASSWORD`.

README.md

@@ -9,11 +9,208 @@ Install with:
 
     pip install pds.cloud-tools
 
-If possible, make it so that your program works out of the box without any additional configuration—but see the [Configuration](###configuration) section for details.
 
-To execute, run:
+## S3 Download (`pdc-s3-download`)
 
-    (put your run commands here)
+Downloads objects from an S3 bucket matching a given prefix to a local directory, preserving the key structure as a directory tree.
+
+### Basic usage
+
+```bash
+pdc-s3-download \
+  --source-profile my-aws-profile \
+  --source-bucket my-bucket \
+  --source-prefix path/to/objects/ \
+  --local-dest-dir ./downloads
+```
+
+`--source-profile` and `--source-bucket` can also be supplied via the `AWS_PROFILE` and `AWS_BUCKET` environment variables.
+
+### Filtering by last-modified time
+
+Use `--start-datetime` and/or `--end-datetime` to restrict downloads to objects last modified within a time range. All datetimes are UTC. Accepted formats: `YYYY-MM-DD` or `YYYY-MM-DDTHH:MM:SS`.
+
+| Arguments provided | Behavior |
+|--------------------|----------|
+| `--start-datetime` only | Download objects modified from start to now |
+| `--end-datetime` only | Download objects modified up to end |
+| Both | Download objects modified within [start, end] |
+| Neither | Download all matching objects |
+
+```bash
+# Download only objects modified in January 2025
+pdc-s3-download \
+  --source-profile my-aws-profile \
+  --source-bucket my-bucket \
+  --source-prefix path/to/objects/ \
+  --local-dest-dir ./downloads \
+  --start-datetime 2025-01-01 \
+  --end-datetime 2025-02-01
+
+# Download everything modified since a specific timestamp
+pdc-s3-download \
+  --source-profile my-aws-profile \
+  --source-bucket my-bucket \
+  --source-prefix path/to/objects/ \
+  --local-dest-dir ./downloads \
+  --start-datetime 2025-06-15T12:00:00
+```
+
+
+## Data Integrity Verification (S3 Bucket Migration)
+
+Use these two tools to verify that every object in an OLD S3 bucket has a matching copy in a NEW bucket — by content (size + checksum), not just by key name.
+
+### Overview
+
+| Command | Purpose |
+|---------|---------|
+| `pdc-build-checksum-manifest` | Generate a CSV manifest of every object in a bucket with its checksum, size, and ETag |
+| `pdc-compare-manifests` | Compare two manifests and report any OLD objects not represented in NEW |
+
+### AWS Credentials Setup
+
+The scripts use standard boto3 credential resolution. Options, in order of preference:
+
+**Named profile** (recommended):
+
+```ini
+# ~/.aws/credentials
+[old-account]
+aws_access_key_id = AKIA...
+aws_secret_access_key = ...
+
+[new-account]
+aws_access_key_id = AKIA...
+aws_secret_access_key = ...
+```
+
+**IAM role assumption** (for cross-account access without long-lived keys):
+
+```ini
+# ~/.aws/config
+[profile new-account]
+role_arn = arn:aws:iam::123456789012:role/DataMigrationRole
+source_profile = old-account
+region = us-west-2
+```
+
+**Environment variables** (for CI/automation):
+
+```bash
+export AWS_ACCESS_KEY_ID=...
+export AWS_SECRET_ACCESS_KEY=...
+export AWS_SESSION_TOKEN=...   # required when using temporary credentials
+```
+
+**Minimum required IAM permissions** for each bucket:
+
+```json
+{
+  "Effect": "Allow",
+  "Action": [
+    "s3:ListBucket",
+    "s3:GetObject",
+    "s3:GetObjectAttributes"
+  ],
+  "Resource": [
+    "arn:aws:s3:::your-bucket-name",
+    "arn:aws:s3:::your-bucket-name/*"
+  ]
+}
+```
+
+### Step-by-Step Workflow
+
+**Step 1 — Generate the OLD bucket manifest:**
+
+```bash
+pdc-build-checksum-manifest \
+  --bucket old-bucket-name \
+  --output old_manifest.csv \
+  --profile old-account \
+  --region us-west-2
+```
+
+**Step 2 — Generate the NEW bucket manifest:**
+
+```bash
+pdc-build-checksum-manifest \
+  --bucket new-bucket-name \
+  --output new_manifest.csv \
+  --profile new-account \
+  --region us-west-2
+```
+
+**Step 3 — Compare the manifests:**
+
+```bash
+pdc-compare-manifests \
+  --old old_manifest.csv \
+  --new new_manifest.csv \
+  --missing-output missing_in_new.csv \
+  --weak-output weak_checksums.csv
+```
+
+The comparison exits with code `0` (PASS) if every OLD object has a matching content signature in NEW, or `1` (FAIL) if any are missing.
+
+Sample output:
+
+```
+=== S3 Manifest Comparison Summary ===
+OLD objects total:               4823901
+OLD objects covered by NEW:      4823901
+OLD objects missing in NEW:      0
+OLD objects with weak checksum:  142
+Missing report:                  missing_in_new.csv
+Weak rows report:                weak_checksums.csv
+
+RESULT: PASS
+Every OLD object has at least one matching content signature in NEW.
+```
+
+### Handling Buckets with Millions of Objects
+
+Manifest generation uses the `list_objects_v2` paginator, which pages through arbitrarily large buckets automatically — there is no per-bucket size limit. Progress is printed to stderr every 1,000 objects.
+
+The comparison step loads both manifests into memory simultaneously. For very large buckets (tens of millions of objects), ensure the host has sufficient RAM.
+
+### Resuming After a Failure
+
+If manifest generation is interrupted (timeout, credential expiry, network error), resume by passing the partial output file as `--resume-from`. Both `--output` and `--resume-from` must point to the same file:
+
+```bash
+# Initial run (interrupted):
+pdc-build-checksum-manifest \
+  --bucket old-bucket-name \
+  --output old_manifest.csv \
+  --profile old-account
+
+# Resume run — pass the same file to both flags:
+pdc-build-checksum-manifest \
+  --bucket old-bucket-name \
+  --output old_manifest.csv \
+  --resume-from old_manifest.csv \
+  --profile old-account
+```
+
+The resume run appends new rows to the existing CSV and skips any key already recorded in it.
+
+### Understanding the Output Files
+
+**`missing_in_new.csv`** — OLD objects not found in NEW, with a `reason` column:
+
+| Reason | Meaning |
+|--------|---------|
+| `NO_MATCHING_SIGNATURE_IN_NEW` | Object content (size + checksum) not present in NEW |
+| `NO_USABLE_CHECKSUM` | Object has no stored checksum; cannot verify by content signature |
+
+**`weak_checksums.csv`** — rows from either manifest lacking a usable checksum. This occurs when objects were uploaded without `--checksum-algorithm`. These objects cannot be verified by content signature alone.
+
+### Limitations
+
+- Checksums are only present if they were stored at upload time. Objects copied with `aws s3 cp` without `--checksum-algorithm` will have no checksum and appear in the weak-checksum report.
+- Matching is by content signature `(size, checksum_type, checksum_value)`, not by key. An object that was renamed or moved but has identical content will be counted as covered.
 
 
 ## Code of Conduct

scripts/csv-converter.py

@@ -0,0 +1,87 @@
+#!/usr/bin/env python3
+
+import csv
+import gzip
+import glob
+import os
+from pathlib import Path
+
+# Input: one or more raw S3 Inventory CSV.gz files
+INPUT_GLOBS = [
+    "/Users/jpadams/proj/pds/pdsen/workspace/cloud-tools/data/inventories/new_bucket_raw/**/*.csv.gz",
+]
+
+# Output: one normalized gzipped CSV
+OUTPUT_FILE = "/Users/jpadams/proj/pds/pdsen/workspace/cloud-tools/data/inventories/new_bucket_processed/new_row_level_normalized.csv.gz"
+
+HEADER = [
+    "bucket",
+    "key",
+    "size",
+    "checksum_algorithm",
+    "checksum_type",
+    "checksum_value",
+    "etag",
+]
+
+def iter_input_files(patterns):
+    for pattern in patterns:
+        for path in glob.glob(pattern, recursive=True):
+            yield path
+
+def main():
+    input_files = list(iter_input_files(INPUT_GLOBS))
+    if not input_files:
+        raise SystemExit("No input files found.")
+
+    Path(os.path.dirname(OUTPUT_FILE) or ".").mkdir(parents=True, exist_ok=True)
+
+    rows_written = 0
+
+    with gzip.open(OUTPUT_FILE, "wt", newline="", encoding="utf-8") as out_f:
+        writer = csv.writer(out_f, quoting=csv.QUOTE_MINIMAL)
+        writer.writerow(HEADER)
+
+        for input_file in input_files:
+            with gzip.open(input_file, "rt", newline="", encoding="utf-8") as in_f:
+                reader = csv.reader(in_f)
+
+                for row in reader:
+                    if not row:
+                        continue
+
+                    # Skip header rows if present
+                    if row[0] == "Bucket" or row[0] == "bucket":
+                        continue
+
+                    # We only need these source positions from the raw inventory row:
+                    # 0 = bucket
+                    # 1 = key
+                    # 2 = size
+                    # 4 = e_tag
+                    try:
+                        bucket = row[0]
+                        key = row[1]
+                        size = row[2]
+                        etag = row[4]
+                    except IndexError:
+                        raise ValueError(f"Unexpected row shape in {input_file}: {row}")
+
+                    if not etag:
+                        continue
+
+                    writer.writerow([
+                        bucket,
+                        key,
+                        size,
+                        "etag",
+                        "etag",
+                        etag,
+                        etag,
+                    ])
+                    rows_written += 1
+
+    print(f"Wrote {rows_written} rows to {OUTPUT_FILE}")
+
+if __name__ == "__main__":
+    main()