Add stream tile export pipeline (export_tiles.sh)

00_setup.sh

@@ -4,11 +4,17 @@
 
 set -euo pipefail
 
-echo "==> Installing osmium-tool …"
-arch -arm64 brew install osmium-tool
+echo "==> Installing brew packages …"
+for pkg in osmium-tool tippecanoe; do
+    if arch -arm64 brew list "$pkg" &>/dev/null; then
+        echo "    $pkg already installed, skipping"
+    else
+        arch -arm64 brew install "$pkg"
+    fi
+done
 
 echo "==> Installing Python geo packages …"
-pip3 install geopandas rasterio shapely pandas numpy pyproj
+pip3 install --quiet geopandas rasterio shapely pandas numpy pyproj boto3
 
 echo "==> Checking ogr2ogr …"
 ogr2ogr --version | head -1

01_grass_hydro.sh

@@ -275,7 +275,7 @@ echo "--- Exporting accum.tif ---"
 r.out.gdal input=accum output="${OUTPUT_DIR}/accum.tif" \
     format=GTiff createopt="COMPRESS=LZW,TILED=YES" --overwrite
 
-echo "--- Exporting streams.gpkg ---"
+echo "--- Exporting streams.gpkg (UTM) ---"
 v.out.ogr input=streams_lines output="${OUTPUT_DIR}/streams.gpkg" \
     format=GPKG --overwrite
 
@@ -289,6 +289,11 @@ echo "    modeled_by_cell.csv  → \$(wc -l < "${OUTPUT_DIR}/modeled_by_cell.csv
 echo "    osm_by_cell.csv      → \$(wc -l < "${OUTPUT_DIR}/osm_by_cell.csv") rows"
 GRASS_SCRIPT
 
+echo "--- Reprojecting streams to WGS84 ---"
+ogr2ogr -f GPKG -t_srs EPSG:4326 \
+    "${OUTPUT_DIR}/streams_wgs84.gpkg" \
+    "${OUTPUT_DIR}/streams.gpkg" -overwrite
+
 echo ""
 echo "==> GRASS done. Next step:"
 echo "    python3 02_grid_analysis.py"

README.md

@@ -96,6 +96,67 @@ Output is written to `output/<name>/gap_analysis.geojson`.
 
 Cells with `max_strahler ≥ 3` and `delta_m > 0` are promoted to **high** regardless of density — significant tributaries are never buried in low/medium.
 
+## Tile export (Mapbox)
+
+After one or more pipeline runs complete, use `export_tiles.sh` to merge all area stream vectors into a single MBTiles file and optionally publish it to a Mapbox tileset.
+
+### Setup (one-time, included in `00_setup.sh`)
+
+```bash
+arch -arm64 brew install tippecanoe
+pip3 install mapbox-tilesets
+```
+
+### Register completed areas
+
+Edit `areas_registry.json` to add each completed run:
+
+```json
+{
+  "siquijor":     { "dir": "siquijor",           "bbox": [123.4, 9.0, 123.8, 9.4] },
+  "catanduanes":  { "dir": "catanduanes",         "bbox": [123.9807, 13.4791, 124.4971, 14.1459] }
+}
+```
+
+- **`dir`** — subfolder under `output/` containing `streams_wgs84.gpkg`
+- **`bbox`** — `[W, S, E, N]` used to clip streams before merging; prevents overlap at area boundaries
+
+Each pipeline run also writes `output/<name>/metadata.json`. Pass `--auto` to discover these automatically without editing the registry.
+
+### Run
+
+```bash
+# Dry run — produces output/tiles/streams_ph.mbtiles
+bash export_tiles.sh
+
+# Upload to Mapbox
+MAPBOX_ACCESS_TOKEN=pk.xxx MAPBOX_USERNAME=myuser \
+  bash export_tiles.sh --upload --tileset myuser.streams-ph
+```
+
+| Option | Description |
+|---|---|
+| `--upload` | Upload to Mapbox after generating MBTiles |
+| `--tileset` | Mapbox tileset ID (required with `--upload`) |
+| `--registry` | Path to registry JSON (default: `areas_registry.json`) |
+| `--auto` | Auto-discover completed runs via `output/*/metadata.json` |
+
+Tiles are generated at zoom 8–14. Each feature carries `strahler` and `source_area` attributes for use in Mapbox style rules. To include additional attributes (e.g. `length`, `gradient`), edit the `SQL=` line in `export_tiles.sh`.
+
+### Folder structure (updated)
+
+```
+├── export_tiles.sh        tile export + Mapbox upload
+├── areas_registry.json    area slug → output dir + bbox
+└── output/
+    ├── <name>/
+    │   ├── streams_wgs84.gpkg   WGS84 stream lines (written by pipeline)
+    │   ├── metadata.json        run metadata for auto-discovery
+    │   └── gap_analysis.geojson
+    └── tiles/
+        └── streams_ph.mbtiles  merged tile output
+```
+
 ## OSM waterway types included
 
 `river`, `stream`, `canal`, `drain`, `ditch`, `tidal_channel`
@@ -109,12 +170,19 @@ Cells with `max_strahler ≥ 3` and `delta_m > 0` are promoted to **high** regar
 ├── 00_download.sh         download DEM tiles + OSM data for any bbox
 ├── 01_grass_hydro.sh      GRASS GIS spatial pipeline
 ├── 02_grid_analysis.py    gap scoring and GeoJSON export
+├── export_tiles.sh        merge streams → MBTiles → Mapbox upload
+├── areas_registry.json    area slug → output dir + bbox for tile export
 ├── data/
 │   ├── srtm/              Copernicus DEM 30m tiles (shared cache, gitignored)
-│   ├── osm/               OSM waterway files per area (gitignored)
+│   ├── osm/               OSM waterway + lake files per area (gitignored)
 │   └── boundary/          Admin boundaries (gitignored)
 └── output/
-    └── <name>/            Per-area outputs including gap_analysis.geojson
+    ├── <name>/            Per-area outputs
+    │   ├── gap_analysis.geojson
+    │   ├── streams_wgs84.gpkg
+    │   └── metadata.json
+    └── tiles/
+        └── streams_ph.mbtiles
 ```
 
 The `grass/` GRASS database is created at runtime and is gitignored.
@@ -126,6 +194,6 @@ The `grass/` GRASS database is created at runtime and is gitignored.
 - Stream threshold of 200 cells ≈ 0.18 km² contributing area. Increase to 500–1000 for noisier/flatter terrain.
 - Copernicus DEM 30m is available from a public AWS S3 bucket — no authentication required.
 - For large areas (e.g. whole Philippines), Overpass times out. Download `philippines-latest.osm.pbf` from [Geofabrik](https://download.geofabrik.de/asia/philippines.html) and pass it via `--osm-file`; the pipeline filters and clips it automatically.
-- Named lake polygons (`lakes_<name>.gpkg`) are cached in `data/osm/` and reused on re-runs. If no named lakes exist in the bbox the step is silently skipped.
+- Named lake polygons (`lakes_<name>.gpkg`) are re-extracted from the PBF on every run when `--osm-file` is provided. If no named lakes exist in the bbox the step is silently skipped.
 - `--carve` burns OSM waterways into the DEM before watershed analysis using `r.carve`. This improves main-stem alignment significantly on flat/coastal terrain. Width must be ≥ 30 m (one DEM pixel) to have any effect; the default 90 m (3 pixels) gives reliable results.
 - `--carve` is off by default so existing single-island runs reproduce unchanged results.

areas_registry.json

@@ -0,0 +1,6 @@
+{
+  "siquijor":    { "dir": "siquijor",           "bbox": [123.4,    9.0,   123.8,   9.4]  },
+  "catanduanes": { "dir": "catanduanes",         "bbox": [123.9807,13.4791,124.4971,14.1459] },
+  "ilocos-norte":{ "dir": "ilocos-norte", "bbox": [120.339, 17.675, 120.987,18.680] },
+  "mindoro":     { "dir": "mindoro",             "bbox": [120.284, 12.120, 121.597,13.545] }
+}