Security hardening, wallet fast-path, and doc updates

Security:
- Add URL validation (core/url_validation.py) to block SSRF: private IPs,
  cloud metadata, non-HTTP schemes return 400 url_blocked
- Sanitize error responses: generic messages to clients, details only in logs
- Wallet ledger: thread-safe balance operations, crypto-random tokens,
  test wallets only when FAIRFETCH_TEST_MODE=true
- CORS: allow * only in test mode; production restricted to publisher domain
- Validate wallet inputs: positive amounts, owner length, balance caps

Wallet fast-path:
- In-memory WalletLedger with charge/top_up/transactions
- X-WALLET-TOKEN skips 402 when balance sufficient
- Endpoints: POST /wallet/register, GET /wallet/balance, POST /wallet/topup,
  GET /wallet/transactions

Docs:
- README: Security section, config notes, project structure
- DEVELOPMENT: URL validation and test_mode behavior
- CONCEPTS: Allowed URLs and SSRF protection
- AI_AGENT_GUIDE: url_blocked, test wallets only in test mode
- PUBLISHER_GUIDE: Production CORS and wallet behavior
- openapi: 400 UrlBlocked, 502/503, X-WALLET-TOKEN in payment description

Author: arun-fairfetch

DEVELOPMENT.md

@@ -166,6 +166,31 @@ make lint       # ruff check + format
 make typecheck  # mypy strict mode
 ```
 
+## Security
+
+### URL validation (SSRF protection)
+
+The `url` query parameter is validated before any outbound fetch. Blocked targets include:
+
+- Non-HTTP(S) schemes (`file://`, `ftp://`, etc.)
+- Loopback and private IPs (`127.x`, `10.x`, `172.16–31.x`, `192.168.x`)
+- Link-local and cloud metadata (e.g. `169.254.169.254`, `metadata.google.internal`)
+
+Requests with a disallowed URL receive `400` with `{"error": "url_blocked", "detail": "The requested URL is not allowed."}`.
+
+To test:
+
+```bash
+curl -s "http://localhost:8402/content/fetch?url=http://127.0.0.1/admin" \
+  -H "X-PAYMENT: test_paid_fairfetch"
+# → 400, url_blocked
+```
+
+### Test mode vs production
+
+- **`FAIRFETCH_TEST_MODE=true`** (default): CORS allows all origins (`*`); wallet ledger pre-seeds `wallet_test_agent_alpha` and `wallet_test_agent_beta`; mock payment tokens accepted.
+- **`FAIRFETCH_TEST_MODE=false`**: CORS is restricted to `https://{FAIRFETCH_PUBLISHER_DOMAIN}`; no pre-seeded wallets; use real payment integration.
+
 ## Architecture Decisions
 
 ### Open Core: interfaces/ vs implementations

README.md

@@ -289,6 +289,65 @@ X-Content-Hash: sha256:2c449548...        # Fingerprint of the content
 
 <br />
 
+## 👛 Wallet-Based Payment (Fast Path)
+
+The 402 round-trip makes sense for discovery, but once an AI company is onboarded it's inefficient to negotiate payment on every request. Fairfetch supports **pre-funded wallets** that skip the 402 entirely:
+
+```bash
+# Register a wallet (in production, this happens through the Fairfetch marketplace)
+curl -X POST "http://localhost:8402/wallet/register?owner=AcmeAI&initial_balance=100000"
+# → {"wallet_token": "wallet_a1b2c3d4...", "balance": 100000, ...}
+
+# Now fetch content instantly — no 402, no X-PAYMENT negotiation
+curl -H "X-WALLET-TOKEN: wallet_test_agent_alpha" \
+     -H "Accept: text/markdown" \
+     "http://localhost:8402/content/fetch?url=https://example.com"
+```
+
+The response includes your remaining balance and a transaction receipt:
+
+```http
+X-FairFetch-Payment-Method: wallet        # Paid via wallet (not x402)
+X-FairFetch-Wallet-Balance: 99000         # Remaining balance after this charge
+X-PAYMENT-RECEIPT: ff_3a7c9e2b...         # Transaction ID in the ledger
+X-FairFetch-License-ID: 47db4290...:k2+w  # Usage Grant (same as x402 flow)
+```
+
+**How it works in practice:**
+
+| | x402 (One-Time Payment) | Wallet (Pre-Funded) |
+|---|---|---|
+| **First request** | 402 → pay → retry → content | Content immediately |
+| **Round-trips** | 2 | 1 |
+| **Best for** | Occasional access, discovery | High-volume production use |
+| **Billing** | Per-request settlement | Balance deducted, settled monthly (Premium) |
+
+<details>
+<summary><strong>Wallet management endpoints</strong></summary>
+
+```bash
+# Check balance
+curl "http://localhost:8402/wallet/balance?token=wallet_test_agent_alpha"
+# → {"owner": "TestAgentAlpha", "balance": 99000, ...}
+
+# Add funds
+curl -X POST "http://localhost:8402/wallet/topup?token=wallet_test_agent_alpha&amount=50000"
+# → {"amount_added": 50000, "new_balance": 149000}
+
+# Transaction history
+curl "http://localhost:8402/wallet/transactions?token=wallet_test_agent_alpha"
+# → {"transactions": [{"tx_id": "ff_...", "amount": 1000, ...}, ...]}
+```
+
+</details>
+
+> [!TIP]
+> Two test wallets are pre-loaded for local development:
+> - `wallet_test_agent_alpha` — balance 100,000 ($0.10)
+> - `wallet_test_agent_beta` — balance 500,000 ($0.50)
+
+<br />
+
 ## 📊 Usage Categories & Tiered Pricing
 
 Not all content usage is equal. Fairfetch defines **usage categories** that control what an AI agent is permitted to do with the content, with escalating compliance requirements and pricing:
@@ -480,7 +539,9 @@ Every successful response includes these headers. Think of them as a receipt and
 | `X-FairFetch-Origin-Signature` | A digital fingerprint proving the publisher's server produced this exact content. Like a notary stamp — tamper-proof. | `GllQLb/V4Vd+Su...` (base64) |
 | `X-FairFetch-License-ID` | Your Usage Grant reference. Store this — it's your proof of legal access if questions arise later. Format: `grant_id:signature_prefix`. | `47db4290...:k2+wXE3x...` |
 | `X-Content-Hash` | A fingerprint of the content body itself, so you can verify nothing was altered in transit. | `sha256:2c449548...` |
-| `X-PAYMENT-RECEIPT` | Proof that payment was settled. In test mode this is a simulated transaction hash. In production, a real on-chain transaction ID. | `0x6d8ce1bf...` |
+| `X-PAYMENT-RECEIPT` | Proof that payment was settled. For x402: a transaction hash. For wallets: a ledger transaction ID (`ff_...`). | `0x6d8ce1bf...` or `ff_3a7c9e...` |
+| `X-FairFetch-Payment-Method` | How the agent paid: `wallet` (pre-funded account) or `x402` (one-time payment). | `wallet` |
+| `X-FairFetch-Wallet-Balance` | Remaining wallet balance after this charge (only present for wallet payments). | `99000` |
 | `X-Fairfetch-Version` | Protocol version, so clients know which Fairfetch spec they're talking to. | `0.2` |
 
 > [!TIP]
@@ -510,6 +571,7 @@ Every successful response includes these headers. Think of them as a receipt and
 ```
 fairfetch/
 ├── docs/                    # Guides for publishers & AI agents
+│   ├── CONCEPTS.md          # Plain-language concepts & headers
 │   ├── PUBLISHER_GUIDE.md   # CDN deployment & onboarding
 │   └── AI_AGENT_GUIDE.md    # MCP/REST integration for agents
 ├── interfaces/              # Open Standard (abstract bases)
@@ -520,7 +582,8 @@ fairfetch/
 │   ├── converter.py         # HTML → Markdown (trafilatura)
 │   ├── summarizer.py        # LiteLLM implementation
 │   ├── knowledge_packet.py  # JSON-LD builder
-│   └── signatures.py        # Ed25519 signing
+│   ├── signatures.py        # Ed25519 signing
+│   └── url_validation.py    # SSRF protection (block private/metadata URLs)
 ├── mcp_server/              # Direct Pipeline (MCP)
 │   └── server.py            # FastMCP tools + resources
 ├── api/                     # Direct Pipeline (REST)
@@ -529,7 +592,8 @@ fairfetch/
 │   ├── negotiation.py       # Content negotiation + bot steering
 │   └── dependencies.py      # FairFetchConfig + DI
 ├── payments/                # x402 micro-payments
-│   ├── x402.py              # Middleware with grant issuance
+│   ├── x402.py              # Middleware (wallet + x402)
+│   ├── wallet_ledger.py     # In-memory wallet ledger (test_mode seeds)
 │   ├── mock_facilitator.py  # Local test facilitator
 │   └── mock_license_facilitator.py
 ├── compliance/              # EU AI Act 2026
@@ -545,7 +609,7 @@ fairfetch/
 │   └── akamai/              # EdgeWorkers (JS)
 ├── scripts/                 # Dev scripts
 │   └── dev_server.py        # Local launcher (make dev)
-├── tests/                   # 106 tests · 98% coverage
+├── tests/                   # 127 tests · 98% coverage
 ├── .github/workflows/       # CI pipeline
 ├── openapi.yaml             # REST API spec
 ├── mcp.json                 # MCP Inspector config
@@ -560,9 +624,9 @@ fairfetch/
 
 | Variable | Default | Description |
 |----------|---------|-------------|
-| `FAIRFETCH_TEST_MODE` | `true` | Enable mock facilitator + grants |
+| `FAIRFETCH_TEST_MODE` | `true` | Enable mock facilitator + grants; when `false`, CORS is restricted to your domain and no test wallets are pre-seeded |
 | `FAIRFETCH_PUBLISHER_WALLET` | `0x000...` | EVM wallet for payments |
-| `FAIRFETCH_PUBLISHER_DOMAIN` | `localhost` | Publisher domain |
+| `FAIRFETCH_PUBLISHER_DOMAIN` | `localhost` | Publisher domain (also used as CORS origin when test mode is off) |
 | `FAIRFETCH_CONTENT_PRICE` | `1000` | Price in smallest USDC unit |
 | `FAIRFETCH_SIGNING_KEY` | *(generated)* | Ed25519 private key (b64) |
 | `FAIRFETCH_LICENSE_TYPE` | `publisher-terms` | Default license |
@@ -573,6 +637,14 @@ fairfetch/
 
 <br />
 
+## 🔒 Security
+
+- **URL validation:** The `url` parameter is validated before any outbound request. Private IPs (e.g. `127.0.0.1`, `10.x`, `192.168.x`), cloud metadata endpoints (e.g. `169.254.169.254`), and non-HTTP(S) schemes are rejected with `400` and `error: "url_blocked"`. This prevents SSRF (server-side request forgery).
+- **Test mode:** With `FAIRFETCH_TEST_MODE=false`, CORS allows only `https://{FAIRFETCH_PUBLISHER_DOMAIN}` and the ledger does not pre-seed test wallets. Use test mode only for local development.
+- **Error responses:** Upstream fetch and summarization errors return generic messages to clients; details are logged server-side only.
+
+<br />
+
 ## 📖 Detailed Guides
 
 | Guide | What's Inside |

api/main.py

@@ -19,6 +19,7 @@
     get_config,
 )
 from api.routes import router
+from payments.wallet_ledger import WalletLedger
 from payments.x402 import X402Middleware
 
 logging.basicConfig(level=logging.INFO, format="%(asctime)s %(name)s %(levelname)s %(message)s")
@@ -41,14 +42,17 @@ def create_app() -> FastAPI:
         redoc_url="/redoc",
     )
 
+    cors_origins = ["*"] if config.test_mode else [f"https://{config.publisher_domain}"]
+
     application.add_middleware(
         CORSMiddleware,
-        allow_origins=["*"],
+        allow_origins=cors_origins,
         allow_methods=["*"],
         allow_headers=[
             "*",
             "X-PAYMENT",
             "X-PAYMENT-RECEIPT",
+            "X-WALLET-TOKEN",
             "X-USAGE-CATEGORY",
             "X-FairFetch-License-ID",
             "X-FairFetch-Origin-Signature",
@@ -61,6 +65,8 @@ def create_app() -> FastAPI:
             "X-FairFetch-Compliance-Level",
             "X-FairFetch-License-ID",
             "X-FairFetch-Origin-Signature",
+            "X-FairFetch-Payment-Method",
+            "X-FairFetch-Wallet-Balance",
             "X-FairFetch-Preferred-Access",
             "X-FairFetch-LLMS-Txt",
             "X-FairFetch-MCP-Endpoint",
@@ -73,14 +79,23 @@ def create_app() -> FastAPI:
     license_provider = (
         build_license_provider(config, signer) if config.enable_usage_grants else None
     )
+    wallet_ledger = WalletLedger(test_mode=config.test_mode)
 
     application.add_middleware(
         X402Middleware,
         facilitator=facilitator,
         requirement=requirement,
         license_provider=license_provider,
+        wallet_ledger=wallet_ledger,
         paid_path_prefixes=["/content/"],
-        exempt_paths=["/health", "/openapi.json", "/docs", "/redoc", "/compliance/"],
+        exempt_paths=[
+            "/health",
+            "/openapi.json",
+            "/docs",
+            "/redoc",
+            "/compliance/",
+            "/wallet/",
+        ],
     )
 
     application.state.config = config
@@ -89,6 +104,7 @@ def create_app() -> FastAPI:
     application.state.summarizer = build_summarizer(config)
     application.state.packet_builder = build_packet_builder(signer)
     application.state.license_provider = license_provider
+    application.state.wallet_ledger = wallet_ledger
     application.state.scraper_intercept_count = 0
 
     application.include_router(router)