Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,8 @@ an ADR, run `make adr-check`.
a directory.
- Keep changes focused. Do not refactor unrelated code while completing a task.
- Record a new ADR when changing a project-wide architectural default.
- Treat MCP as an opt-in primary adapter. Expose selected use cases, not every HTTP route.
- Publish external events through a transactional outbox. Assume at-least-once delivery and require idempotent consumers.

## Python Style

Expand Down Expand Up @@ -59,6 +61,18 @@ When implementing a feature:
9. Wire dependencies in the composition root.
10. Add the smallest useful tests at each affected boundary.

When adding MCP tools or resources, call application handlers and query
services through the composition root. Do not call HTTP routes or SQLAlchemy
sessions from an MCP adapter. Define authentication, authorization, audit
logging, and a threat review before enabling remote MCP access.

When publishing to Kafka or another broker, depend on the service-layer
`IntegrationMessageBus` port. Translate domain events into
versioned integration events and persist them to the transactional outbox with
the aggregate change. Relay them outside the write transaction. Document
consumer idempotency, retries, dead-letter handling, retention, and
observability before production deployment.

Use domain events for facts that already happened. Use commands for requests to
perform work. Do not treat API response models as domain events. Introduce
versioned integration events before publishing public broker contracts.
Expand Down
28 changes: 26 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -45,9 +45,33 @@ adr-context: ## Prints active architecture guidance for agents
migrate: ## Applies relational database migrations
uv run alembic upgrade head

.PHONY: mcp
mcp: ## Runs the optional MCP addon over Streamable HTTP
uv run --extra mcp python -m template.addons.mcp.server

.PHONY: kafka-up
kafka-up: ## Starts the local Kafka broker
docker compose up -d kafka

.PHONY: kafka-relay
kafka-relay: ## Relays transactional outbox events to Kafka
uv run --extra kafka python -m template.addons.kafka.worker

.PHONY: kafka-consume
kafka-consume: ## Prints local user integration events from Kafka
docker compose exec kafka kafka-console-consumer --bootstrap-server kafka:29092 --topic users.events --from-beginning

.PHONY: kafka-ui-up
kafka-ui-up: ## Starts local Kafka with the optional Kafbat UI
docker compose --profile observability up -d kafbat-ui

.PHONY: kafka-ui-down
kafka-ui-down: ## Stops the optional Kafbat UI
docker compose --profile observability stop kafbat-ui

.PHONY: cover
cover: ## Executes tests cases with coverage reports
uv run pytest --cov src/template --cov-fail-under=100 --junitxml reports/xunit.xml \
uv run --extra mcp --extra kafka pytest --cov src/template --cov-fail-under=100 --junitxml reports/xunit.xml \
--cov-report xml:reports/coverage.xml --cov-report term-missing

.PHONY: format
Expand All @@ -62,7 +86,7 @@ pre-commit: ## Runs pre-commit hooks on all files
lint: ## Applies static analysis and type checks
uv run ruff check ./src ./tests ./scripts ./migrations
uv run ruff format --check ./src ./tests ./scripts ./migrations
uv run pyrefly check
uv run --extra mcp --extra kafka pyrefly check

.PHONY: fix
fix: ## Fix lint errors
Expand Down
90 changes: 90 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,8 @@ the [Cosmic Python](https://www.cosmicpython.com/) guidelines.
* [About](#about)
* [Features](#features)
* [Architecture Decision Records](#architecture-decision-records)
* [Optional MCP Addon](#optional-mcp-addon)
* [Optional Kafka Addon](#optional-kafka-addon)
* [Project Structure](#project-structure)
* [Environment Variables](#environment-variables)
* [Recommended Directory Structure](#recommended-directory-structure)
Expand Down Expand Up @@ -64,6 +66,72 @@ Repository-level [agent instructions](AGENTS.md) turn those decisions into an im
The [Cosmic Python coverage matrix](docs/cosmic-python-coverage.md) distinguishes included patterns from conditional
extensions such as optimistic locking, broker adapters, and the transactional outbox.

### Optional MCP Addon

Projects with agent clients can install an opt-in [Model Context Protocol](https://modelcontextprotocol.io/) adapter:

```bash
uv sync --extra mcp
make mcp
```

The standalone Streamable HTTP server is available at `http://127.0.0.1:8000/mcp`.
It demonstrates an intentional primary adapter rather than automatic route exposure:

- `onboard_user` is an MCP tool that dispatches the existing registration command.
- `users://{user_id}` is an MCP resource backed by the existing CQRS reader.
- The tool returns the resource URI so an agent can read persisted preferences before later user-specific actions.

The addon is local-only scaffolding until a project defines authentication, authorization, audit logging, and a threat
review. See [ADR 0015](docs/adr/0015-mcp-is-an-opt-in-primary-adapter.md).

### Optional Kafka Addon

Projects that publish public integration events can install the Kafka addon:

```bash
uv sync --extra kafka
make migrate
make kafka-up
make kafka-relay
```

In another terminal, inspect the published events:

```bash
make kafka-consume
```

For a local browser-based view of topics, messages, partitions, keys, and
consumer groups, start the optional [Kafbat UI](https://github.com/kafbat/kafka-ui):

```bash
make kafka-ui-up
```

Open `http://localhost:8080` and select the `local` cluster. Stop the UI with:

```bash
make kafka-ui-down
```

User registration demonstrates the transactional outbox pattern:

1. The aggregate raises the internal `UserRegistered` domain event.
2. The SQLAlchemy unit of work stores the user and a versioned `UserRegisteredV1` outbox row in one transaction.
3. The broker-neutral relay publishes pending rows through the service-layer `IntegrationMessageBus` port.
4. The opt-in Kafka adapter sends them to the local `users.events` topic.
5. Kafka messages use `userId` as the key and a camel-case JSON envelope containing `eventId`, `occurredAt`,
`eventType`, `schemaVersion`, and `payload`.

Delivery is at least once. Consumers must deduplicate with `eventId`. Docker Compose runs a single-node KRaft broker for
local development only. Production projects must define security, retention, observability, retry, and dead-letter
policies. See [ADR 0016](docs/adr/0016-external-message-buses-use-a-transactional-outbox.md).

Kafbat UI is also a local-development convenience. It is pinned to a release
image, kept outside the Python application, and excluded from the default
Compose profile.

### Project Structure

#### Environment Variables
Expand Down Expand Up @@ -99,6 +167,16 @@ Variables prefixed with `DATABASE_` configure relational persistence.
Use Alembic migrations for normal schema management. `DATABASE_AUTO_CREATE_SCHEMA`
exists for isolated tests and local demonstrations only.

Variables prefixed with `KAFKA_` configure the optional outbox relay.

| Name | Description | Default Value |
|-------------------------|-------------------------------------|-----------------|
| KAFKA_BOOTSTRAP_SERVERS | Comma-separated Kafka broker list | localhost:9092 |
| KAFKA_USER_EVENTS_TOPIC | Topic for public user events | users.events |
| KAFKA_PUBLISH_TIMEOUT | Publisher acknowledgement timeout | 10.0 |
| KAFKA_RELAY_INTERVAL | Delay between relay polling passes | 1.0 |
| KAFKA_BATCH_SIZE | Maximum events per relay pass | 100 |

### Recommended Directory Structure

As the application grows, keep the dependency direction visible in the directory structure. The domain remains plain
Expand All @@ -110,6 +188,8 @@ Python. Framework validation belongs at the entrypoint boundary, and persistence
|-- src/template
| |-- adapters # (3)
| | |-- models
| | | `-- outbox.py
| | |-- outbox.py
| | |-- queries.py
| | |-- repository.py
| | `-- unit_of_work.py
Expand All @@ -121,6 +201,11 @@ Python. Framework validation belongs at the entrypoint boundary, and persistence
| | |-- monitor.py
| | |-- schemas.py
| | `-- users.py
| |-- integration_events
| | `-- user.py
| |-- addons # (5)
| | |-- kafka
| | `-- mcp
| |-- service_layer # (2)
| | |-- handlers.py
| | |-- messagebus.py
Expand Down Expand Up @@ -155,6 +240,10 @@ Python. Framework validation belongs at the entrypoint boundary, and persistence
inward-facing adapters.
- **(4)**. Entrypoints are the places we drive our application from. FastAPI routes and Pydantic request or response
schemas live here. In ports and adapters terminology, these are primary or driving adapters.
- **(5)**. Addons are optional primary adapters for selected projects. The MCP addon exposes agent-oriented tools and
resources through the same application ports without becoming a default runtime dependency. The Kafka addon is an
optional secondary adapter implementing the broker-neutral `IntegrationMessageBus` port. The service layer relays
public events from the transactional outbox without depending on Kafka.

The root `bootstrap.py` module is the composition root. It wires concrete adapters to application ports without leaking
framework concerns into the domain.
Expand Down Expand Up @@ -381,6 +470,7 @@ make cover
- [FastAPI official Documentation](https://fastapi.tiangolo.com/)
- [Pydantic official Documentation](https://pydantic-docs.helpmanual.io/)
- [UV official Documentation](https://docs.astral.sh/uv/)
- [MCP Python SDK](https://github.com/modelcontextprotocol/python-sdk)
- [Cosmic Python](https://cosmicpython.com/)

## Licence
Expand Down
31 changes: 31 additions & 0 deletions docker-compose.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,35 @@
services:
kafka:
image: confluentinc/confluent-local:8.2.1
container_name: cosmic-fastapi-kafka
ports:
- "9092:9092"
environment:
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT,CONTROLLER:PLAINTEXT
KAFKA_LISTENERS: PLAINTEXT://:29092,PLAINTEXT_HOST://:9092,CONTROLLER://:29093
KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
KAFKA_PROCESS_ROLES: broker,controller
KAFKA_NODE_ID: 1
KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:29093
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
CLUSTER_ID: MkU3OEVBNTcwNTJENDM2Qk

kafbat-ui:
image: ghcr.io/kafbat/kafka-ui:v1.5.0
container_name: cosmic-fastapi-kafbat-ui
profiles:
- observability
depends_on:
- kafka
ports:
- "8080:8080"
environment:
KAFKA_CLUSTERS_0_NAME: local
KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka:29092

app:
# This tells Docker Compose to build the image from the Dockerfile
# in the current directory.
Expand Down
53 changes: 53 additions & 0 deletions docs/adr/0015-mcp-is-an-opt-in-primary-adapter.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
# ADR 0015: MCP Is an Opt-In Primary Adapter

- Status: Accepted
- Date: 2026-05-31

## Context

Coding agents increasingly act as application clients. Model Context Protocol
(MCP) gives them a standard way to discover tools and read context. Exposing
every HTTP endpoint automatically would blur use-case intent, enlarge the
attack surface, and couple an agent contract to transport details.

## Decision

Offer MCP as an optional primary adapter backed by the official Python SDK.

- Install it explicitly with the `mcp` project extra.
- Expose selected mutations as MCP tools and selected read-only context as MCP
resources.
- Dispatch tools through the same command bus and query services as HTTP
entrypoints.
- Keep MCP server imports isolated from the default runtime.
- Run the example as a standalone Streamable HTTP server. Mounting it into an
existing ASGI process remains a project-level deployment choice.
- Require a project to define authentication, authorization, and audit policy
before exposing an MCP server beyond a trusted local environment.

The sample addon exposes `onboard_user` as a tool and `users://{user_id}` as a
resource. The tool returns the durable resource URI so an agent can read
persisted preferences before taking later user-specific actions.

## Consequences

Agent-oriented projects receive a concrete MCP path without making MCP a
dependency or network surface for every generated service. MCP contracts are
intentional and can evolve independently from HTTP routes. Projects must make
security and deployment decisions before production exposure.

## Agent Guidance

- Do not auto-generate MCP tools from FastAPI routes.
- Keep mutations explicit and narrow. Prefer one aggregate boundary per tool.
- Expose read-only context as resources when it gives an agent durable,
addressable information.
- Call the application layer, never HTTP routes or SQLAlchemy sessions.
- Do not expose internal domain events as MCP tools.
- Add authentication, authorization, audit logging, and a threat review before
enabling remote access.

## References

- [Official MCP Python SDK](https://github.com/modelcontextprotocol/python-sdk)
- [MCP Python SDK server documentation](https://modelcontextprotocol.github.io/python-sdk/server/)
75 changes: 75 additions & 0 deletions docs/adr/0016-external-message-buses-use-a-transactional-outbox.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
# ADR 0016: External Message Buses Use a Transactional Outbox

- Status: Accepted
- Date: 2026-05-31

## Context

Publishing directly to Kafka after a database commit can lose an event when
the process stops between the two writes. Publishing before commit can emit an
event for state that later rolls back. A distributed transaction between the
application database and Kafka adds complexity that most projects do not need.

## Decision

Define a broker-neutral `IntegrationMessageBus` port in the service layer and
provide Kafka as one opt-in secondary adapter. Back external publication with
a transactional outbox.

- Translate selected domain events into versioned public integration events.
- Store each integration event in `outbox_events` in the same database
transaction as aggregate changes.
- Relay unpublished rows through `IntegrationMessageBus` in a separate worker
process.
- Mark a row as published only after the configured broker acknowledges the
event.
- In the Kafka adapter, publish with the aggregate identity as the key so events for one
aggregate receive a stable partitioning default.
- Treat delivery as at least once. Consumers must be idempotent using
`eventId`.
- Keep the in-process message bus for local reactions. The external
`IntegrationMessageBus` is an integration boundary, not a replacement for
domain events.

The sample translates `UserRegistered` into `UserRegisteredV1`. Its Kafka
adapter publishes to the `users.events` topic. Local Docker Compose provides a
single-node KRaft broker for development only. A project can replace Kafka with
RabbitMQ, SQS, or another adapter without changing the relay use case.

The optional `observability` Docker Compose profile adds Kafbat UI for local
inspection of topics, partitions, messages, keys, and consumer groups. It is a
developer tool, not an application dependency or production default.

## Consequences

A broker outage does not fail user registration after the database is
available. Events remain durable until the relay can publish them. A relay may
publish the same event more than once if it stops after Kafka acknowledges a
message but before the database records completion.

The example intentionally does not include a broker consumer. A project should
add one when it has a concrete downstream use case, together with idempotency
storage and retry policy.

## Agent Guidance

- Do not publish to an external bus inside an HTTP route, aggregate, or database
transaction.
- Add versioned integration schemas before external consumers depend on an
event.
- Keep domain events and public integration events distinct.
- Depend on `IntegrationMessageBus` in the service layer. Keep Kafka, RabbitMQ,
SQS, and SDK-specific types inside secondary adapters.
- Use `eventId` for consumer deduplication and aggregate identity as the Kafka
key unless ordering requirements justify another choice.
- Document dead-letter handling, retention, observability, and consumer retry
policy before production deployment.
- Do not treat local Docker Compose settings as production Kafka guidance.
- Keep Kafbat UI and similar dashboards outside application code. Require
production-specific access controls before deploying an operational UI.

## References

- [Cosmic Python: Events and the Message Bus](https://www.cosmicpython.com/book/chapter_08_events_and_message_bus.html)
- [Confluent Kafka Python client](https://docs.confluent.io/kafka-clients/python/current/overview.html)
- [Confluent KRaft overview](https://docs.confluent.io/platform/current/kafka-metadata/kraft.html)
4 changes: 4 additions & 0 deletions docs/adr/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,8 @@ treated as migration work, not as a precedent to repeat.
| [0012](0012-camel-case-json-message-contracts.md) | Camel-Case JSON Message Contracts | Accepted |
| [0013](0013-aggregates-define-consistency-boundaries.md) | Aggregates Define Consistency Boundaries | Accepted |
| [0014](0014-cqrs-read-models-are-purpose-built.md) | CQRS Read Models Are Purpose Built | Accepted |
| [0015](0015-mcp-is-an-opt-in-primary-adapter.md) | MCP Is an Opt-In Primary Adapter | Accepted |
| [0016](0016-external-message-buses-use-a-transactional-outbox.md) | External Message Buses Use a Transactional Outbox | Accepted |

## Agent Checklist

Expand All @@ -51,6 +53,8 @@ Before writing code:
11. Add integration tests where adapters meet real infrastructure.
12. Record a new ADR when changing an accepted default.
13. Use Conventional Commits when creating Git history.
14. Add MCP as an opt-in primary adapter only when agents are application clients.
15. Publish Kafka events through a transactional outbox and require idempotent consumers.

## Decision Pruner

Expand Down
Loading