Skip to content

Layer-Based Metadata Tracking & Documentation Consolidation #3

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
nittygritty-zzy merged 3 commits into from
Oct 21, 2025
Merged

Layer-Based Metadata Tracking & Documentation Consolidation #3

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
8487dcf
Implement layer-based metadata tracking across Medallion Architecture
Oct 21, 2025
86a50f0
Clean up redundant documentation
Oct 21, 2025
e299d7b
Consolidate documentation into single comprehensive guide
Oct 21, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
706 changes: 706 additions & 0 deletions docs/PIPELINE_OPERATIONS_GUIDE.md
View file
Open in desktop

Large diffs are not rendered by default.

10 changes: 5 additions & 5 deletions docs/getting-started/DATA_CONFIGURATION.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ Edit `config/system_profile.yaml` (gitignored - safe for personal paths):
cp config/system_profile.yaml.example config/system_profile.yaml

# Edit system_profile.yaml
data_root: /Volumes/ExternalSSD/quantmini-data/data
data_root: /Volumes/ExternalSSD/quantlake/data
```

**Pros**:
Expand Down Expand Up @@ -148,13 +148,13 @@ Store data on a fast external drive:

```bash
# macOS
DATA_ROOT=/Volumes/ExternalSSD/quantmini-data/data
DATA_ROOT=/Volumes/ExternalSSD/quantlake/data

# Linux
DATA_ROOT=/mnt/storage/quantmini-data/data
DATA_ROOT=/mnt/storage/quantlake/data

# Windows (WSL)
DATA_ROOT=/mnt/d/quantmini-data/data
DATA_ROOT=/mnt/d/quantlake/data
```

**Pros**: More storage capacity, doesn't fill system drive
Expand All @@ -169,7 +169,7 @@ Store data on NAS or cloud storage:
DATA_ROOT=/mnt/nas/quantmini/data

# Cloud (mounted via rclone, etc.)
DATA_ROOT=/mnt/s3/quantmini-data/data
DATA_ROOT=/mnt/s3/quantlake/data
```

**Pros**: Accessible from multiple machines, backup built-in
Expand Down
4 changes: 2 additions & 2 deletions docs/guides/data-ingestion-strategies.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -509,15 +509,15 @@ uv run python scripts/validation/validate_duckdb_access.py
**Solution**:
```bash
# Check disk usage
df -h /Volumes/sandisk/quantmini-data
df -h /Volumes/sandisk/quantlake

# Clean old data
uv run python scripts/maintenance/cleanup_old_data.py \
--keep-days 365

# Move to external drive
rsync -av --progress \
/Volumes/sandisk/quantmini-data/ \
/Volumes/sandisk/quantlake/ \
/Volumes/backup/quantmini-archive/
```

Expand Down
Loading
Loading