Implement Google Cloud terraform & CLI tooling

.claude/agents/test-writer.md

@@ -25,7 +25,7 @@ You are a test engineer for OpenContext. You write pytest tests that follow the
 
 ## Conventions to follow
 
-**asyncio:** `asyncio_mode = "auto"` is in `pyproject.toml`. Do NOT add `@pytest.mark.asyncio` decorators.
+**asyncio:** `asyncio_mode = "auto"` is in `pyproject.toml`. `@pytest.mark.asyncio` is present on existing tests (redundant but harmless); omit it in new tests.
 
 **Test structure:**
 ```python
@@ -40,18 +40,20 @@ class TestFeatureName:
             "timeout": 120,
         }
 
+    @pytest.mark.asyncio
     async def test_verb_noun_condition(self, plugin_config):
         ...
 ```
 
 **Mock pattern for httpx:**
 ```python
-from unittest.mock import AsyncMock, MagicMock, patch
+from unittest.mock import AsyncMock, Mock, patch
 
-with patch("plugins.{name}.plugin.httpx.AsyncClient") as mock_cls:
+with patch("httpx.AsyncClient") as mock_cls:
     mock_client = AsyncMock()
-    mock_response = MagicMock()
+    mock_response = Mock()
     mock_response.json.return_value = {"success": True, "result": []}
+    mock_response.raise_for_status = Mock()
     mock_client.get = AsyncMock(return_value=mock_response)
     mock_cls.return_value = mock_client
 ```

.claude/rules/code-style.md

@@ -22,7 +22,7 @@ from core.plugin_manager import PluginManager
 ```
 
 ## Type Hints
-Required on all public method signatures. Prefer `X | None` for optional types, consistent with the Python >=3.11 baseline and existing codebase usage.
+Required on all public method signatures. Prefer `X | None` for optional types in new or updated code. Existing `Optional[X]` annotations are also acceptable until touched for other changes.
 
 ## Docstrings
 Google-style, triple-quoted. Class docstring before `__init__`. Methods get a one-liner minimum.

.claude/rules/testing.md

@@ -6,8 +6,21 @@ alwaysApply: false
 
 # Testing Conventions
 
+## Where tests live
+
+See [`tests/README.md`](../../tests/README.md). Summary:
+
+| Path | Purpose |
+|------|---------|
+| `tests/unit/` | Fast tests with mocks (`pytest.mark.unit`) |
+| `tests/integration/` | Hermetic cross-boundary flows (`pytest.mark.integration`) |
+| `tests/security/` | SSRF / SQL / SoQL guards (`pytest.mark.security`) |
+| `tests/smoke/` | Minimal CLI/protocol smoke (`pytest.mark.smoke`) |
+
+Markers are registered in `pyproject.toml` under `[tool.pytest.ini_options]`.
+
 ## asyncio
-`asyncio_mode = "auto"` is set in `pyproject.toml`. Do NOT add `@pytest.mark.asyncio` to individual async test methods — it's redundant.
+`asyncio_mode = "auto"` is set in `pyproject.toml`. `@pytest.mark.asyncio` is present on many existing tests (redundant but harmless); omit it in new tests.
 
 ## Test Structure
 ```python
@@ -23,21 +36,22 @@ class TestPluginInitialization:
             "timeout": 120,
         }
 
+    @pytest.mark.asyncio
     async def test_initialize_succeeds_on_valid_config(self, plugin_config):
         """test_verb_noun_condition naming."""
 ```
 
 ## Mock Pattern
 ```python
-from unittest.mock import AsyncMock, MagicMock, patch
+from unittest.mock import AsyncMock, Mock, patch
 
-# Patch at the import location, not the definition location
-with patch("plugins.ckan.plugin.httpx.AsyncClient") as mock_client_class:
+with patch("httpx.AsyncClient") as mock_client_class:
     mock_client = AsyncMock()
-    mock_response = MagicMock()
+    mock_response = Mock()
     mock_response.json.return_value = {"success": True, "result": []}
+    mock_response.raise_for_status = Mock()
     mock_client.get = AsyncMock(return_value=mock_response)
-    mock_client_class.return_value.__aenter__ = AsyncMock(return_value=mock_client)
+    mock_client_class.return_value = mock_client
 ```
 
 ## AWS / boto3

.claude/settings.json

@@ -1,5 +1,5 @@
 {
-  "$schema": "https://schemas.anthropic.com/claude-code/settings.json",
+  "$schema": "https://json.schemastore.org/claude-code-settings.json",
   "permissions": {
     "allow": [
       "Bash(uv run *)",
@@ -63,7 +63,7 @@
         "hooks": [
           {
             "type": "command",
-            "command": "python3 -c \"import sys,json,subprocess; d=json.load(sys.stdin); fp=d.get('tool_input',{}).get('file_path',''); fp.endswith('.py') and subprocess.run(['uv','run','ruff','format',fp],capture_output=True)\""
+            "command": "uv run python3 -c \"import sys,json,subprocess; d=json.load(sys.stdin); fp=d.get('tool_input',{}).get('file_path',''); fp.endswith('.py') and subprocess.run(['uv','run','ruff','format',fp],capture_output=True)\""
           }
         ]
       }

.claude/skills/add-plugin/SKILL.md

@@ -67,18 +67,18 @@ uv run opencontext test --url http://localhost:8000/mcp
 ```
 
 ### 5. Write tests
-Create `tests/test_{name}_plugin.py` following the structure in `tests/test_ckan_plugin.py`:
+Create `tests/unit/plugins/{name}/test_{name}_plugin.py` following the structure in `tests/unit/plugins/ckan/test_ckan_plugin.py`:
 - Group by `TestXxx` classes
 - Use `AsyncMock` for httpx calls (see mock pattern in `.claude/rules/testing.md`)
 - Fixtures return dicts, not Pydantic models
 
 ```bash
-uv run pytest tests/test_{name}_plugin.py -v --cov=custom_plugins --cov-report=term-missing
+uv run pytest tests/unit/plugins/{name}/test_{name}_plugin.py -v --cov=custom_plugins --cov-report=term-missing
 ```
 
 ### 6. Verify coverage gate still passes
 ```bash
-uv run pytest tests/ -n auto --cov=core --cov=plugins --cov-fail-under=80
+uv run pytest tests/ -n auto --cov=core --cov=plugins --cov=server --cov-fail-under=80
 ```
 
 ## Common Mistakes

.claude/skills/fix-coverage/SKILL.md

@@ -12,7 +12,7 @@ command: /fix-coverage
 ## 1. Find the gaps
 ```bash
 uv run pytest tests/ -n auto \
-  --cov=core --cov=plugins \
+  --cov=core --cov=plugins --cov=server \
   --cov-report=term-missing \
   --cov-report=html
 ```
@@ -33,7 +33,7 @@ Don't add tests for these unless you're actually fixing a bug there.
 4. New code you just wrote
 
 ## 4. Write targeted tests
-Follow patterns in `tests/test_ckan_plugin.py`:
+Follow patterns in `tests/unit/plugins/ckan/test_ckan_plugin.py`:
 - Group by `TestXxx` class
 - `AsyncMock` for httpx, `MagicMock` for sync
 - Patch at the import location: `patch("plugins.arcgis.plugin.httpx.AsyncClient")`
@@ -44,7 +44,7 @@ Focus on uncovered branches (the `term-missing` output shows which lines). One t
 ## 5. Verify
 ```bash
 uv run pytest tests/ -n auto \
-  --cov=core --cov=plugins \
+  --cov=core --cov=plugins --cov=server \
   --cov-fail-under=80
 ```
 CI gate: must pass with `--cov-fail-under=80`.

.cursor/rules/project.mdc

@@ -0,0 +1,22 @@
+---
+description: Binds Cursor agents to repo-specific context via AI.md.
+alwaysApply: true
+---
+
+@../../AI.md
+
+## Cursor specifics
+
+- Repo-wide facts live in `@../../AI.md`. Claude Code permissions, hooks, and bundled prompts live in `CLAUDE.md` and `.claude/`; do not load `CLAUDE.md` as the primary context unless the user opens it.
+- The repo ships `.vscode/tasks.json` (Cursor-compatible). On folder open it runs `git pull` then `uv sync --all-extras`. The `Test: run all` and `Lint: ruff fix + format` tasks mirror the CI commands in `AI.md` if you prefer the task UI.
+- The `@../../AI.md` include resolves relative to this file; if this rule is moved, update the path.
+
+## Context Hygiene
+
+- As the final step of any agentic task, after tests pass and before closing the task, scan for anything non-obvious you discovered: architectural decisions made, constraints encountered, non-obvious file relationships, gotchas debugged, commands that only work a certain way.
+- If any such discovery would have saved you time had it been in `AI.md` at the start of the session, add it there.
+- Add to `AI.md` only — never bloat this file with project facts.
+- Write additions as pointers or single-line facts, not paragraphs.
+- Do not add things that are obvious from the code, already documented in `docs/`, generic best practices, or specific to just the current task.
+- If nothing non-obvious was discovered, make no changes — do not add placeholder or "session notes" content.
+- Never restructure or reformat existing `AI.md` content during an update — append only, to the most relevant existing section.

.github/workflows/ci.yml

@@ -107,6 +107,7 @@ jobs:
             -n auto \
             --cov=core \
             --cov=plugins \
+            --cov=server \
             --cov-report=term-missing \
             --cov-fail-under=80
 

.github/workflows/infra.yml

@@ -37,6 +37,7 @@ jobs:
       - name: Create placeholder artifacts for validation
         run: |
           python3 -c "import zipfile; zipfile.ZipFile('terraform/aws/lambda-deployment.zip', 'w').close()"
+          python3 -c "import zipfile; zipfile.ZipFile('terraform/gcp/gcf-deployment.zip', 'w').close()"
 
       - name: Validate all Terraform directories
         run: |

.github/workflows/release.yml

@@ -51,6 +51,7 @@ jobs:
             -n auto \
             --cov=core \
             --cov=plugins \
+            --cov=server \
             --cov-report=term-missing \
             --cov-fail-under=80
 

.gitignore

@@ -213,6 +213,7 @@ lambda-deployment.zip
 terraform/*/.terraform/
 terraform/*/.terraform.lock.hcl
 terraform/*/lambda-deployment.zip
+terraform/*/gcf-deployment.zip
 
 # Terraform state (contains secrets, ignore across all folders)
 **/*.tfstate

.vscode/tasks.json

@@ -49,7 +49,7 @@
     {
       "label": "Test: run all",
       "type": "shell",
-      "command": "uv run pytest tests/ -n auto --cov=core --cov=plugins --cov-fail-under=80",
+      "command": "uv run pytest tests/ -n auto --cov=core --cov=plugins --cov=server --cov-fail-under=80",
       "presentation": {
         "reveal": "always",
         "panel": "shared"

AI.md

@@ -0,0 +1,68 @@
+# AI.md
+
+OpenContext is an extensible MCP (Model Context Protocol) framework that exposes
+public open-data portals (CKAN, ArcGIS Hub, Socrata) to AI assistants. Stack:
+Python 3.11+ managed with `uv`, Typer CLI, aiohttp dev server, deployed to AWS
+(Lambda + API Gateway) or GCP (Cloud Functions gen2) via Terraform (`--cloud`).
+
+## Directory map (non-obvious only)
+
+- `custom_plugins/` — drop-in user plugins; auto-discovered at startup and copied into the Lambda bundle by `cli/commands/deploy.py`
+- `server/adapters/` — runtime entry points: `aws_lambda.py` for production, local aiohttp server for dev
+- `examples/` — per-city example `config.yaml` files
+- `client/` — optional Go stdio↔HTTP MCP bridge for stdio-only clients
+- `.claude/` — Claude Code project config (`settings.json`, glob-scoped `rules/`, `skills/`, `agents/`); see `CLAUDE.md`
+- `.cursor/` — Cursor rules; see `.cursor/rules/project.mdc`
+
+Self-evident dirs (`core/`, `plugins/`, `cli/`, `tests/`, `docs/`, `terraform/`) are intentionally omitted.
+
+## Dev commands
+
+- Install full dev deps (matches CI): `uv sync --all-extras`
+- CLI-only install: `uv sync --extra cli`
+- Bootstrap config: `cp config-example.yaml config.yaml` then edit; optional `uv run pre-commit install`
+- Local MCP server (http://localhost:8000/mcp): `uv run opencontext serve`
+- Smoke test running server: `uv run opencontext test --url http://localhost:8000/mcp`
+- Tests with CI coverage gate: `uv run pytest tests/ -n auto --cov=core --cov=plugins --cov=server --cov-report=term-missing --cov-fail-under=80`
+- Targeted suites by marker: `uv run pytest tests/unit -m unit -v` (also `integration`, `security`, `smoke`)
+- Go client tests: `cd client && go test ./...`
+- Lint as CI runs it (no autofix): `uv run ruff check core/ plugins/ server/ tests/`
+- Local autofix + format: `uv run ruff check core/ plugins/ server/ tests/ --fix --unsafe-fixes && uv run ruff format core/ plugins/ server/ tests/`
+- CVE audit (CI parity): `uv run pip-audit -r requirements.txt`
+- Validate before deploy: `uv run opencontext validate --env staging`
+- Deploy: `uv run opencontext deploy --env staging` (requires TTY; prompts for confirmation)
+
+`--env` defaults to `staging` on every command that accepts it.
+`--cloud` defaults to `aws` on every command that accepts it.
+
+## Hard constraints
+
+- **Exactly one `plugins.*.enabled: true`** in `config.yaml`. Enforced at server startup (`core/validators.py`, `core/plugin_manager.py`) and pre-deploy (`cli/commands/deploy.py::_validate_single_plugin`). Multiple data sources require forking the repo, not toggling more flags.
+- **`config.yaml` is gitignored.** Never commit it. Template-level changes go in `config-example.yaml`.
+- **Branch off `develop`; PRs target `develop`, not `main`.** Prefixes: `feature/`, `bugfix/`, `docs/`, `chore/`.
+- **CI fails below 80% coverage** on `core`, `plugins`, `server`. New code needs tests. `pyproject.toml` `[tool.coverage.run].omit` documents which modules are excluded — adding logic to those without un-omitting will not earn coverage.
+- **MCP tool names are namespaced `plugin__tool_name`** (double underscore, prepended automatically). Plugins must NOT include the prefix in their own `get_tools()` names.
+- **Lambda packaging targets `x86_64-manylinux2014` + Python 3.11.** `cli/commands/deploy.py::_package_lambda` runs `uv pip install -r requirements.txt --python-platform x86_64-manylinux2014 --python-version 3.11`. New deps must be wheel-compatible with that target — local `uv sync` success is not sufficient proof.
+- **Lambda bundle uses `requirements.txt`, not the `uv` lockfile.** Update both when adding deps; `pip-audit` in CI scans `requirements.txt`.
+- **`asyncio_mode = "auto"`** is set in `pyproject.toml`. `@pytest.mark.asyncio` is present on existing tests (redundant but harmless); omit it in new tests.
+- **Never modify `config.yaml`, `.env*`, `terraform/**/*.tfvars`, or `terraform/**/*.tfstate*`** from agent tools; these are denied in `.claude/settings.json` and contain secrets / state.
+- **Plugins inherit `MCPPlugin`** (`core/interfaces.py`) and must implement `initialize`, `shutdown`, `get_tools`, `execute_tool`, `health_check`. Data-source plugins inherit `DataPlugin` and add `search_datasets`, `get_dataset`, `query_data`.
+- **`PluginType` enum values** (`core/interfaces.py`): `OPEN_DATA`, `CUSTOM_API`, `DATABASE`, `ANALYTICS`.
+
+## Pointers
+
+- `@docs/GETTING_STARTED.md`
+- `@docs/QUICKSTART.md`
+- `@docs/CLI.md`
+- `@docs/ARCHITECTURE.md`
+- `@docs/BUILT_IN_PLUGINS.md`
+- `@docs/CUSTOM_PLUGINS.md`
+- `@docs/DEPLOYMENT.md`
+- `@docs/TESTING.md`
+- `@docs/FAQ.md`
+- `@tests/README.md`
+- `@CONTRIBUTING.md`
+- `@.claude/settings.json`
+- `@.claude/rules/`
+- `@.claude/skills/`
+- `@.claude/agents/`

CLAUDE.md

@@ -11,11 +11,9 @@ Each repo fork runs **exactly one** plugin. `core/validators.py` and `core/plugi
 ```bash
 uv sync --all-extras        # install all deps including cli + dev
 cp config-example.yaml config.yaml   # then edit for your data source
-uv run pre-commit install          # set up git hooks
+pre-commit install          # set up git hooks
 ```
 
-`requirements.txt` pins dependencies for Lambda bundles and `pip-audit` in CI; local development uses `uv sync`, not `pip install -r requirements.txt`.
-
 ## Common Commands
 
 ```bash
@@ -26,11 +24,11 @@ opencontext serve
 opencontext test --url http://localhost:8000/mcp
 
 # Tests
-uv run pytest tests/ -n auto --cov=core --cov=plugins --cov-fail-under=80
+pytest tests/ -n auto --cov=core --cov=plugins --cov-fail-under=80
 
 # Lint + format (matches CI)
-uv run ruff check core/ plugins/ server/ tests/ --fix --unsafe-fixes
-uv run ruff format core/ plugins/ server/ tests/
+ruff check core/ plugins/ server/ tests/ --fix --unsafe-fixes
+ruff format core/ plugins/ server/ tests/
 
 # CLI
 opencontext validate --env staging    # validate config + Terraform before deploy
@@ -49,9 +47,11 @@ core/           # Framework kernel — interfaces, MCP server, plugin manager, v
 plugins/        # Built-in plugins: ckan/, arcgis/, socrata/
 custom_plugins/ # Drop user plugins here — auto-discovered at startup
 cli/            # Typer CLI (opencontext command)
+server/         # HTTP adapters: local aiohttp + AWS Lambda entry point
 server/adapters/ # local aiohttp dev server + AWS Lambda entry point
 tests/          # pytest suite (80% coverage required)
 terraform/aws/  # Lambda + API Gateway + IAM IaC
+terraform/gcp/  # Cloud Functions gen2 + GCS + IAM IaC (see terraform/gcp/README.md)
 docs/           # Architecture, deployment, plugin authoring guides
 ```
 
@@ -72,7 +72,7 @@ To add a custom plugin:
 2. Implement: `initialize`, `shutdown`, `get_tools`, `execute_tool`, `health_check`
 3. Enable in `config.yaml` under `plugins.my_plugin.enabled: true`
 
-See `docs/CUSTOM_PLUGINS.md` for the interface contract and `.claude/skills/add-plugin/SKILL.md` for the step-by-step workflow.
+See `docs/CUSTOM_PLUGINS.md` for full interface contract.
 
 ## Config (`config.yaml`)
 
@@ -112,15 +112,6 @@ uv run pip-audit -r requirements.txt
 uv run pytest tests/ -n auto --cov=core --cov=plugins --cov-fail-under=80
 ```
 
-## Branching
-
-Branch off `develop`: `feature/`, `bugfix/`, `docs/`, `chore/`. PRs target `develop`, not `main`.
-Terraform workspace naming: `{city}-{env}` (e.g., `chicago-staging`) — created by `opencontext configure`.
-
-## Conventions
-
-Coding style, test patterns, plugin authoring, and infrastructure rules live in `.claude/rules/` and load automatically for relevant file types.
-
 ## Gotchas
 
 - **Multiple plugins enabled** → hard crash at startup. Only one `enabled: true` allowed.

CONTRIBUTING.md

@@ -21,7 +21,7 @@ uv run pre-commit install            # set up lint/format hooks
 **PR checklist:**
 
 - [ ] `uv run ruff check core/ plugins/ server/ tests/` passes
-- [ ] `uv run pytest tests/ -n auto --cov=core --cov=plugins --cov-fail-under=80` passes
+- [ ] `uv run pytest tests/ -n auto --cov=core --cov=plugins --cov=server --cov-fail-under=80` passes
 - [ ] New code has tests; coverage gate is enforced in CI
 
 ## Code Style
@@ -40,16 +40,21 @@ Hooks: Ruff (Python), yamllint, gofmt. Type hints are expected on all public met
 ## Testing
 
 ```bash
-uv run pytest tests/ -n auto --cov=core --cov=plugins --cov-fail-under=80
+uv run pytest tests/ -n auto --cov=core --cov=plugins --cov=server --cov-fail-under=80
+
+# Targeted suites (markers defined in pyproject.toml)
+uv run pytest tests/integration -m integration -v
+uv run pytest tests/unit -m unit -v
+uv run pytest tests/security -m security -v
 
 # HTML coverage report (find gaps)
-uv run pytest tests/ --cov=core --cov=plugins --cov-report=html
+uv run pytest tests/ --cov=core --cov=plugins --cov=server --cov-report=html
 open htmlcov/index.html
 ```
 
 - Coverage gate is 80% — enforced in CI. New code needs tests.
-- Built-in plugin tests: `tests/plugins/{plugin_name}/`
-- Custom plugin tests: `tests/custom_plugins/{plugin_name}/`
+- See [`tests/README.md`](tests/README.md) for the layout: `tests/unit/`, `tests/integration/` (hermetic cross-boundary), `tests/security/`, `tests/smoke/`.
+- Built-in plugin unit tests: `tests/unit/plugins/{ckan,arcgis,socrata}/`
 - Mock external HTTP (use `httpx`'s mock transport or `pytest-httpx`) — do not hit live APIs in unit tests.
 
 ## Plugin Development