Merge pull request #242 from paroki/feat/sync

sync packages

Author: kilip

.env.example

@@ -33,3 +33,9 @@ R2_ACCESS_KEY_ID=
 R2_SECRET_ACCESS_KEY=
 R2_BUCKET_NAME=
 R2_PUBLIC_URL=
+
+# DUK Sync
+SYNC_TARGET_URL=https://umat.kasri.id
+SYNC_USERNAME=
+SYNC_PASSWORD=
+SYNC_COOKIE_TTL=3600

.github/workflows/ci.yml

@@ -35,6 +35,10 @@ env:
   CF_ACCOUNT_ID: ${{secrets.CF_ACCOUNT_ID}}
   WHATSAPP_TOKEN: ${{secrets.WHATSAPP_TOKEN}}
   SENTRY_AUTH_TOKEN: ${{secrets.SENTRY_AUTH_TOKEN}}
+  SYNC_TARGET_URL: ${{ secrets.SYNC_TARGET_URL }}
+  SYNC_USERNAME: ${{ secrets.SYNC_USERNAME }}
+  SYNC_PASSWORD: ${{ secrets.SYNC_PASSWORD }}
+  SYNC_COOKIE_TTL: 1500
 
 jobs:
   check:

AGENTS.md

@@ -1,13 +1,23 @@
 # AGENTS.md
 
-This guide is intended for AI agents (Copilot, Cursor, Claude, etc.) working in the **Domus** repository.
+This guide is intended for AI agents (Copilot, Cursor, Claude, Gemini etc.) working in the **Domus** repository.
+
+---
+
+# 🛡️ Anti-Hallucination Protocol
+1. **Strict Scope:** Code *only* what is explicitly requested. No extra files, no unsolicited features.
+2. **Stop & Ask:** If anything is unclear, stop and ask. Zero assumptions.
+3. **Brief:** Keep explanations concise and focused.
+4. **Read First:** State which `AGENTS.md` section and quote the relevant issue instruction before coding.
+5. **Verify Imports:** Only import from paths that explicitly exist in the codebase or issue.
+6. **ALWAYS** follow `AGENTS.md` and focus on the current issue only.
+7. **Wait for Go:** Never start coding until explicitly told **"gas ngoding"**.
 
 ---
 
 ## Agent Self-Verification Checklist
 
 **MANDATORY**: Before providing any code response, verify your implementation against this checklist.
-
 - [ ] **No `any`**: Ensure no `explicit any` is used. Use `unknown`, generics, or search `@domus/core` for the correct type.
 - [ ] **No `shadcn` edits**: NEVER touch files in `apps/dash/src/shared/ui/shadcn/`. Perform customizations by creating wrapper components or extending styles in other directories.
 - [ ] **Result Pattern**: Confirm all services return `Result<T>` via `ok()`/`fail()` and never throw.
@@ -18,6 +28,7 @@ This guide is intended for AI agents (Copilot, Cursor, Claude, etc.) working in
 - [ ] **Auth Responsibility**: Check authentication in Proxy/Action and authorization in Service.
 - [ ] **Thin Exports**: Ensure route components are logic-less exports.
 - [ ] **Naming**: Use entity-only names for services/repositories (no `.service.ts`).
+- [ ] **Zod Best Practices**: Use top-level Zod functions (`z.uuid()`, `z.uuidv7()`, `z.email()`, `z.url()`) instead of chained string methods (`z.string().uuid()`, etc.) to support better tree-shaking and follow Zod 4 conventions.
 
 ---
 

README.md

@@ -36,7 +36,8 @@ This project is managed using Turborepo with a **Just-in-Time (JIT) Packages** s
 ```text
 apps/
 ├── dash/           # Main Web Application (Next.js 16 - Feature-Sliced Design)
-└── cron/           # Scheduled Tasks (Cloudflare Workers)
+├── cron/           # Scheduled Tasks (Cloudflare Workers)
+└── crawler/        # Data Synchronization CLI (Dockerized for Mini PC)
 
 packages/
 ├── core/           # Business Logic, Entities, Result Pattern (Framework Agnostic)
@@ -97,6 +98,13 @@ Available commands from the root folder:
 - `pnpm typecheck` — Run TypeScript compiler checks.
 - `pnpm test` — Run unit tests.
 
+## ⚙️ Crawler Deployment
+
+The **Domus Crawler** is a dedicated CLI tool designed to run on a local mini PC at the parish office. It syncs data from the central DUK system to a local database.
+
+For detailed deployment and usage instructions, see:
+👉 [apps/crawler/README.md](./apps/crawler/README.md)
+
 ## 📄 License
 
 **domus** is released under the **MIT** license. All code and documentation are open source and can be freely used, modified, and distributed in accordance with the license terms.

apps/crawler/.gitignore

@@ -0,0 +1 @@
+data

apps/crawler/Dockerfile

@@ -0,0 +1,39 @@
+# Base image
+FROM node:24-alpine AS base
+
+# Install pnpm and essentials
+RUN npm install -g pnpm && apk add --no-cache libc6-compat
+
+# Set working directory
+WORKDIR /app
+
+# Copy monorepo root config files
+COPY package.json pnpm-workspace.yaml pnpm-lock.yaml ./
+
+# Copy internal packages and the crawler app
+COPY packages ./packages
+COPY apps/crawler ./apps/crawler
+
+# Install dependencies
+RUN pnpm install --frozen-lockfile
+
+# Build the crawler
+RUN pnpm --filter @domus/crawler build
+
+# Deployment Stage
+FROM node:24-alpine AS runner
+WORKDIR /app
+
+# Ensure pnpm is available in runner if needed, but we run with node
+# For simplicity, we can copy the whole node_modules from base
+# or use pnpm deploy in the build stage. 
+# The instructions say: ENTRYPOINT ["node", "apps/crawler/dist/index.js"]
+# This implies we need the dependencies.
+
+# Copy everything from base (including node_modules and source)
+COPY --from=base /app /app
+
+# Set entry point
+# We use tsx via pnpm start because raw node has issues with ESM extensions in a monorepo
+ENTRYPOINT ["pnpm", "--filter", "@domus/crawler", "start"]
+CMD ["crawl"]

apps/crawler/README.md

@@ -0,0 +1,71 @@
+# Domus Crawler
+
+A dedicated CLI tool driven by `@domus/sync` to orchestrate data synchronization between the central DUK system and a local parish database. Designed to be deployed on a mini PC within the parish network.
+
+## 🚀 Overview
+
+The crawler performs four main tasks:
+1.  **Seed**: Populates local regional data (Provinces, Regencies, Districts, Villages) from BPS sources.
+2.  **Scrape**: Fetches reference data and parishioner records from DUK into a staging area.
+3.  **Transform**: Processes staged data into production-ready tables in the local database.
+4.  **Sync-Back**: Pushes local updates (made in Domus) back to the DUK system.
+
+## ⚙️ Configuration
+
+The crawler requires the following environment variables. Create a `.env` file in this directory or pass them to Docker.
+
+| Variable | Description | Example |
+| :--- | :--- | :--- |
+| `DATABASE_URL` | Local PostgreSQL connection string. | `postgresql://user:pass@localhost:5432/domus` |
+| `SYNC_TARGET_URL` | Base URL of the DUK system. | `https://duk-target.com` |
+| `SYNC_USERNAME` | Username for DUK system login. | `parish_admin` |
+| `SYNC_PASSWORD` | Password for DUK system login. | `********` |
+
+## 🛠️ Usage
+
+### Local Development
+
+Running from the monorepo root:
+
+```bash
+# Show help
+pnpm --filter @domus/crawler start help
+
+# Run full pipeline
+pnpm --filter @domus/crawler start crawl
+
+# Individual commands
+pnpm --filter @domus/crawler start seed
+pnpm --filter @domus/crawler start scrape
+pnpm --filter @domus/crawler start transform
+pnpm --filter @domus/crawler start syncback
+```
+
+### Docker Deployment
+
+For production deployment on a mini PC:
+
+1.  **Build the image** (run from monorepo root):
+    ```bash
+    docker build -t domus-crawler -f apps/crawler/Dockerfile .
+    ```
+
+2.  **Run the container**:
+    ```bash
+    docker run --rm --env-file .env domus-crawler [command]
+    ```
+
+    *Example: Run full crawl*
+    ```bash
+    docker run --rm --env-file .env domus-crawler crawl
+    ```
+
+## 🏗️ Docker Details
+
+The Dockerfile uses a multi-stage build:
+- **Base**: Installs all monorepo dependencies.
+- **Run**: Uses `tsx` to execute the TypeScript source directly, ensuring full compatibility with the monorepo's ESM structure without complex build-time transpilation issues.
+
+---
+
+Built with ❤️ for Kristus Raja Barong Tongkok Parish.

apps/crawler/package.json

@@ -0,0 +1,24 @@
+{
+  "name": "@domus/crawler",
+  "version": "0.0.0",
+  "private": true,
+  "type": "module",
+  "scripts": {
+    "start": "tsx src/index.ts",
+    "build": "tsc",
+    "start:prod": "node dist/index.js",
+    "typecheck": "tsc --noEmit"
+  },
+  "dependencies": {
+    "@domus/sync": "workspace:*",
+    "@domus/db": "workspace:*",
+    "@domus/core": "workspace:*",
+    "drizzle-orm": "^0.45.1"
+  },
+  "devDependencies": {
+    "@domus/tsconfig": "workspace:*",
+    "@types/node": "24.x",
+    "tsx": "^4.19.3",
+    "typescript": "^5.7.3"
+  }
+}