Introduce Unified Integration Framework (UIF), Supply-Chain RFQ system, and provider adapters

[LokiMetaSmith]

Summary

Add a Unified Integration Framework (UIF) to centrally manage external integrations (VCS, LLMs, supply-chain, etc.), implement a Request-for-Quote (RFQ) feature with secure inbound webhooks, and provide provider adapters for GitHub, GitLab, and a generic supply-chain endpoint. Includes API routes, documentation, tests, and wiring into existing OKH extraction and LLM provider registration.

Motivation

• Centralize integration lifecycle, configuration, and health checks.
• Enable outbound RFQs and inbound Quotes/status updates from external suppliers (e.g., WeFlourish).
• Make LLM providers and VCS extractors discoverable/usable through a shared integration API.
• Improve observability and make adding future providers straightforward.

What changed (high level)

Major additions and modifications:

• Configuration

  • config/integration_config.json — default integration configuration and examples.

• Documentation

  • docs/api/integration.md — UIF API and examples.
  • docs/features/rfq.md — RFQ feature spec (outbound RFQs, inbound quotes/status, webhook signing).
  • docs/index.md, docs/api/index.md — index updates.

• Integration framework

  • src/core/integration/manager.py — IntegrationManager singleton: config loading, provider registration, initialization, request routing, shutdown.
  • src/core/integration/models/base.py — IntegrationCategory, IntegrationRequest, IntegrationResponse, ProviderStatus.
  • src/core/integration/config/config.py — IntegrationConfig (pydantic).
  • src/core/integration/providers/base.py — BaseIntegrationProvider abstract interface.
  • src/core/integration/providers/github.py — GitHub provider adapter using GitHubExtractor.
  • src/core/integration/providers/gitlab.py — GitLab provider adapter using GitLabExtractor.
  • src/core/integration/providers/supply_chain.py — GenericSupplyChainProvider (send RFQ, HMAC signing).

• RFQ system

  • src/core/models/rfq.py — RFQ and Quote models (Pydantic), enums, to_dict helpers.
  • src/core/services/rfq_service.py — RFQService: create_rfq (persistence + broadcast), create_quote, get_rfq, get_quotes_for_rfq, update_rfq_status.
  • src/core/api/routes/rfq.py — FastAPI routes for RFQ create/retrieve, quotes list, and webhooks (/webhooks/quotes, /webhooks/status) with HMAC verification.

• Integration API endpoints

  • src/core/api/routes/integration.py — /providers and /status endpoints.

• Wiring & compatibility changes

  • src/core/llm/providers/base.py — BaseLLMProvider now inherits BaseIntegrationProvider and adapts to IntegrationRequest/Response.
  • src/core/llm/service.py — LLM providers are registered with IntegrationManager during initialization.
  • src/core/services/okh_service.py — OKH extraction uses IntegrationManager to request VCS extraction (github/gitlab) rather than instantiating extractors directly.
  • src/core/api/decorators.py — external_integration decorator and llm_endpoint compatibility adjustments to ensure IntegrationManager initialization where needed.
  • src/core/main.py — include new routers: /v1/api/integration and /v1/api/rfq.

• Storage/discovery

  • src/core/storage/smart_discovery.py — added rfq/quote discovery rules and validators.

• Tests

  • tests/* — new tests validating IntegrationManager, providers, RFQService, API endpoints, webhook security, and OKH integration paths (see tests added in compare).

API (endpoints added)

• GET /v1/api/integration/providers

  • List configured and registered providers (name, type, category, connected, status)

• GET /v1/api/integration/status

  • Health & connection status for all providers

• POST /v1/api/rfq/

  • Create a new RFQ and broadcast to configured supply-chain providers.
  • Example body:

    {
      "project_name": "Project Alpha",
      "description": "Prototype chassis",
      "capabilities": ["cnc-milling"],
      "callback_url": "https://example.com/v1/api/rfq/webhooks"
    }

• GET /v1/api/rfq/{rfq_id}

  • Retrieve RFQ details

• GET /v1/api/rfq/{rfq_id}/quotes

  • List quotes received for an RFQ

• POST /v1/api/rfq/webhooks/quotes

  • Receive a new Quote from external providers (requires X-OHM-Signature HMAC-SHA256 header)

• POST /v1/api/rfq/webhooks/status

  • Receive status updates for RFQs (requires X-OHM-Signature)

Webhook signing contract

• HMAC-SHA256 over the raw request body (JSON) using the provider's webhook_secret.
• Header: X-OHM-Signature:
• verify_signature dependency checks incoming signature against configured supply_chain providers' webhook_secret values.

Configuration & environment variables

• Add or customize config/integration_config.json. Example keys used:

  • provider entry fields:
    • provider_type (e.g., "github", "gitlab", "supply_chain")
    • use_secrets (boolean)
    • secret_key_env (env var name to fetch secret via secrets manager)
    • api_url, api_key, webhook_secret (for supply-chain)
  • defaults: ai_model, vcs_platform

• Common env vars (depending on provider config):

  • GITHUB_TOKEN
  • GITLAB_TOKEN
  • OPENAI_API_KEY
  • ANTHROPIC_API_KEY
  • Provider-specific keys such as WEFLOURISH_API_KEY

Security notes

• Webhooks must be signed with HMAC-SHA256 and validated by the application.
• Secrets are injected via get_secrets_manager() and not stored in code.
• Current verification compares signature against all configured supply_chain webhook_secrets. Consider tightening to identify provider explicitly in webhook requests (e.g., provider-id header or provider-specific webhook URLs) in follow-ups.

How to run and test locally

1. Ensure required environment variables are set for any providers used (or mock them for tests).
2. Install dev/test dependencies and run tests:
   • pip install -r requirements-dev.txt
   • pytest -q
3. Run the app for manual testing:
   • export necessary env vars (GITHUB_TOKEN, etc.)
   • uvicorn src.core.main:app --reload
4. Example: create RFQ via curl

   curl -X POST http://localhost:8000/v1/api/rfq/ \
     -H "Content-Type: application/json" \
     -d '{"project_name":"Test Project","capabilities":["cnc-milling"]}'

5. Example: compute and send webhook signature (Python)

   import hmac, hashlib, json
   secret = b"provider-secret"
   payload = {"rfq_id":"...","provider_id":"...","amount":1500}
   body = json.dumps(payload, separators=(",", ":"), sort_keys=True).encode("utf-8")
   sig = hmac.new(secret, body, hashlib.sha256).hexdigest()
   # then curl with: -H "X-OHM-Signature: <sig>"

Testing & CI

• New unit/async tests added under tests/:
  • tests/core/integration/test_manager.py
  • tests/core/integration/test_providers.py
  • tests/core/services/test_okh_integration.py
  • tests/core/services/test_rfq_service.py
  • tests/core/api/routes/test_rfq_routes.py
  • tests/core/api/routes/test_security.py
  • tests/core/api/test_integration_api.py
• Ensure CI supports asyncio tests (pytest-asyncio) and any mocking of external services.

Backward compatibility & potential breaking changes

• Decorators and endpoint initialization now lazy-initialize IntegrationManager. This is intended to be backward compatible but can expose startup ordering issues in rare setups — please review.
• BaseLLMProvider now subclasses BaseIntegrationProvider (backward compatible in practice but check any custom LLM provider implementations).
• Provider naming & config: IntegrationManager expects provider_type strings to match registered provider classes; misconfigured provider_type will log warnings and skip provider initialization.

Risks, trade-offs, and mitigations

• Initialization ordering: IntegrationManager.initialize is async; it’s guarded by locks and lazy initialization, but long init can delay request handling. Future improvement: initialize at app startup to fail-fast if needed.
• Webhook provider mapping: current approach compares incoming signature against all supply_chain provider secrets. Future work: explicit provider-identifying headers or separate webhook endpoints per provider to avoid ambiguity.
• Network reliability: direct synchronous broadcast to providers currently lacks retries/backoff/circuit-breaker. Future work: queue RFQs, add retries and monitoring.

Migration & deploy notes

• Deploy config/integration_config.json or populate equivalent config via your configuration pipeline.
• Ensure secrets are available to the secrets manager or environment variables referenced by config.
• Ensure storage credentials allow writing to rfq/requests and rfq/quotes paths.
• Run test suite, watch logs for IntegrationManager provider initialization.

Checklist for reviewers

• Validate integration_config.json schema and example values.
• Review IntegrationManager lifecycle, locking, and provider registration for race conditions and error reporting.
• Verify webhook verification logic and header naming matches expectations with external partners.
• Check RFQ API contract in docs/features/rfq.md against partner expectations.
• Run tests locally and in CI (pytest); confirm async tests pass.
• Confirm no sensitive secrets are checked in.
• Ensure OKH extraction still yields expected manifests after integration wiring.
• Consider whether IntegrationManager should be eagerly initialized in app startup.

Remaining TODOs / follow-ups

• Add provider selection strategies (round-robin, priority, region, health-based).
• Add retry/backoff and circuit breakers for external API calls (send_rfq).
• Improve provider health checks (actual API pings, rate limit checks).
• Map inbound webhooks to providers explicitly (provider-id header or dedicated webhook endpoints).
• Add more metrics/tracing (per-provider latency, error rates).
• Add additional provider adapters (more LLMs, S3/Blob storage, ERP connectors).

Files changed (representative)

Key new/modified files (see the commit for the full list):

• config/integration_config.json
• docs/api/integration.md
• docs/features/rfq.md
• src/core/integration/manager.py
• src/core/integration/models/base.py
• src/core/integration/providers/{base,github,gitlab,supply_chain}.py
• src/core/models/rfq.py
• src/core/services/rfq_service.py
• src/core/api/routes/{integration.py,rfq.py}
• src/core/llm/providers/base.py
• src/core/llm/service.py
• src/core/services/okh_service.py
• src/core/storage/smart_discovery.py
• tests/* (new tests for integration, rfq, providers, APIs)

PR description suggestion (copy-paste)

This PR message is already formatted above — feel free to paste the entire contents of this PR body into GitHub as the PR description.

If you want, I can:

• Shorten this into a 3–5 paragraph summary for the PR description and attach a separate design doc in the repo, or
• Expand any section (e.g., give more concrete examples of config/integration_config.json, or add sample webhook verification code). Which would you prefer?