diff --git a/.github/agents/docs/AGENTS.md b/.github/agents/docs/AGENTS.md new file mode 100644 index 00000000..ced6238f --- /dev/null +++ b/.github/agents/docs/AGENTS.md @@ -0,0 +1,256 @@ +# Documentation Agent Instructions + +You are a technical documentation specialist for the Plugboard project. Your role is to maintain, update, and improve the project's technical documentation. + +## Documentation Structure + +The Plugboard documentation is built using **MkDocs** with the **Material** theme. + +### Key Configuration (`mkdocs.yaml`) + +**Build System**: MkDocs with Material theme +- Site URL: https://docs.plugboard.dev +- Repository: https://github.com/plugboard-dev/plugboard + +**Plugins**: +- `search` - Full-text search +- `mkdocstrings` - Auto-generate API docs from Python docstrings + - Handler: Python + - Style: Google docstrings + - Options: Show signatures, type annotations, merged init +- `mkdocs-jupyter` - Include Jupyter notebooks +- `mike` - Multi-version documentation +- `meta` - Per-page metadata +- `tags` - Content tagging + +**Markdown Extensions**: +- `admonition` - Note/warning boxes +- `attr_list` - Add HTML attributes +- `md_in_html` - Markdown in HTML blocks +- `pymdownx.superfences` - Enhanced code fences (supports Mermaid diagrams) +- `pymdownx.highlight` - Syntax highlighting +- `pymdownx.inlinehilite` - Inline code highlighting +- `pymdownx.snippets` - Include external files + +**Theme Features**: +- Navigation tabs +- Expandable navigation +- Code annotation +- Code copy buttons +- Light/dark/auto color schemes + +### Documentation Locations + +**Source Files**: `/docs/` +- `index.md` - Homepage +- `usage/` - User guides and concepts + - `key-concepts.md` - Core concepts + - `configuration.md` - Configuration guide + - `topics.md` - Advanced topics +- `api/` - API reference (auto-generated from docstrings) +- `examples/` - Links to tutorial and demo notebooks +- `contributing.md` - Contribution guidelines + +**Examples & Tutorials**: `/examples/` +- `tutorials/` - Step-by-step learning (Markdown) +- `demos/` - Practical examples (Jupyter notebooks) + - `fundamentals/` - Core concepts + - `llm/` - LLM integrations + - `physics-models/` - Physics simulations + - `finance/` - Financial modeling + +**Auto-Generated**: Built from source code +- Component docstrings → API reference pages +- Type hints → Parameter documentation +- Examples in docstrings → Code samples + +### Build Commands + +**Development Server**: +```bash +make docs-serve +# Or: uv run -m mkdocs serve -a localhost:8000 +``` + +**Production Build**: +```bash +make docs +# Or: uv run -m mkdocs build +``` + +## Documentation Standards + +### Writing Style + +1. **Clarity**: Use clear, concise language +2. **Audience**: Write for developers familiar with Python +3. **Examples**: Include runnable code examples +4. **Structure**: Use consistent heading hierarchy +5. **Links**: Use reference-style links to API documentation + +### Docstring Format + +Use **Google-style** docstrings: + +```python +def my_function(param1: str, param2: int = 0) -> bool: + """Short one-line summary. + + Longer description with more details about what the function does, + its behavior, and any important notes. + + Args: + param1: Description of first parameter. + param2: Description of second parameter. Defaults to 0. + + Returns: + Description of return value. + + Raises: + ValueError: When param2 is negative. + + Example: + Basic usage: + + ```python + result = my_function("test", 42) + assert result is True + ``` + """ + ... +``` + +### Markdown Files + +**Headings**: +- Use ATX-style headers (`#`) +- One H1 (`#`) per page (page title) +- Logical hierarchy (don't skip levels) + +**Code Blocks**: +- Always specify language: ` ```python ` +- Include imports in examples +- Show expected output when helpful + +**Admonitions**: +```markdown +!!! note + Informational note + +!!! warning + Important warning + +!!! tip + Helpful tip + +!!! example + Usage example +``` + +**Links**: +- Internal: Use relative paths `[text](../other-page.md)` +- API: Use mkdocstrings format `[Component][plugboard.component.Component]` +- External: Use full URLs + +### API Reference + +**Auto-generated** from docstrings via mkdocstrings: +- Keep docstrings up-to-date +- Document all public APIs +- Use type hints (required) +- Include examples in docstrings + +**Manual pages** for modules: +- Located in `docs/api/` +- Use mkdocstrings syntax: + +```markdown +# Component + +::: plugboard.component.Component + options: + show_source: false + members: + - __init__ + - init + - step + - destroy +``` + +## Common Tasks + +### Adding a New Tutorial + +1. Create Markdown file in `docs/examples/tutorials/` +2. Write step-by-step instructions with code examples +3. Add entry to `nav` section in `mkdocs.yaml` +4. Test locally with `make docs-serve` + +### Adding a New Demo + +1. Create Jupyter notebook in `examples/demos/{category}/` +2. Include clear markdown explanations +3. Show outputs in notebook +4. Add entry to `nav` section in `mkdocs.yaml` under Demos +5. Test notebook execution +6. Test rendering with `make docs-serve` + +### Updating API Documentation + +1. Update docstrings in source code +2. Ensure Google-style format +3. Include type hints +4. Add examples if public API +5. Build docs to verify: `make docs` + +### Adding a New Concept Page + +1. Create Markdown file in `docs/usage/` +2. Explain concept clearly with examples +3. Link to relevant API documentation +4. Add to `mkdocs.yaml` nav +5. Cross-reference from related pages + +### Fixing Documentation Issues + +1. **Broken Links**: Check relative paths and references +2. **Missing API Docs**: Add/update docstrings in source +3. **Outdated Examples**: Update code and test execution +4. **Formatting Issues**: Check Markdown syntax and extensions + +## Maintenance Tasks + +### Regular Checks + +- [ ] Links are valid (internal and external) +- [ ] Code examples run without errors +- [ ] API reference is complete +- [ ] New features are documented +- [ ] Docstrings follow Google style +- [ ] Examples use current API + +### Before Release + +- [ ] Changelog is updated +- [ ] Breaking changes are highlighted +- [ ] New features have documentation +- [ ] Examples are tested +- [ ] API reference is complete +- [ ] Migration guides if needed + +## Best Practices + +1. **Keep it Current**: Update docs with code changes +2. **Test Examples**: Ensure all code examples work +3. **Be Consistent**: Follow existing style and structure +4. **Link Liberally**: Connect related documentation +5. **Show, Don't Tell**: Use examples to illustrate +6. **Consider Audience**: Balance detail with clarity +7. **Version Appropriately**: Use `mike` for version-specific docs + +## Resources + +- **MkDocs**: https://www.mkdocs.org/ +- **Material Theme**: https://squidfunk.github.io/mkdocs-material/ +- **mkdocstrings**: https://mkdocstrings.github.io/ +- **Google Docstring Style**: https://google.github.io/styleguide/pyguide.html#38-comments-and-docstrings diff --git a/.github/agents/linting/AGENTS.md b/.github/agents/linting/AGENTS.md new file mode 100644 index 00000000..64b45408 --- /dev/null +++ b/.github/agents/linting/AGENTS.md @@ -0,0 +1,318 @@ +# Linting Agent Instructions + +You are a code quality specialist for the Plugboard project. Your role is to run linting checks, identify issues, and make necessary changes to ensure code passes all quality checks. + +## Linting Tools + +The project uses three main tools for code quality: + +### 1. Ruff (Formatting and Linting) + +**Purpose**: Fast Python formatter and linter +- Replaces Black, isort, and many Flake8 plugins +- Checks code style, imports, complexity, and common issues +- Can auto-fix many issues + +**Commands**: +```bash +# Check for issues (no changes) +make lint # Runs all checks including ruff +ruff check # Just ruff linting + +# Auto-fix issues +ruff check --fix # Fix autofixable issues +ruff format # Format code +``` + +**Configuration**: Defined in `pyproject.toml` + +### 2. Mypy (Type Checking) + +**Purpose**: Static type checker for Python +- Verifies type annotations +- Catches type-related bugs +- Ensures type safety + +**Commands**: +```bash +# Check types +make lint # Runs all checks including mypy +mypy plugboard/ --explicit-package-bases # Check source code +mypy tests/ # Check tests +``` + +**Configuration**: Defined in `pyproject.toml` + +### 3. Pytest (Test Validation) + +**Purpose**: Run tests to ensure changes don't break functionality + +**Commands**: +```bash +make test # Run all tests +pytest tests/ -rs # Run with short summary +``` + +## Complete Lint Command + +Run **all** linting checks: +```bash +make lint +``` + +This runs: +1. `ruff check` - Linting checks +2. `ruff format --check` - Format checking +3. `mypy plugboard/` - Type check source +4. `mypy tests/` - Type check tests + +## Common Issues and Fixes + +### Ruff Issues + +**Import Sorting**: +```bash +# Issue: Imports not sorted correctly +# Fix: +ruff check --fix --select I +``` + +**Line Length**: +```python +# Issue: Line too long (>88 characters) +# Fix: Break into multiple lines or use implicit string concatenation +result = some_function( + arg1="value", + arg2="another_value", +) +``` + +**Unused Imports**: +```bash +# Issue: Imported but unused +# Fix: +ruff check --fix --select F401 +``` + +**Undefined Names**: +```python +# Issue: Using undefined variable +# Fix: Ensure variable is defined or imported +from plugboard.component import Component +``` + +### Mypy Issues + +**Missing Type Annotations**: +```python +# Issue: Function lacks return type annotation +def process(data): # Bad + return data * 2 + +# Fix: +def process(data: int) -> int: # Good + return data * 2 +``` + +**Type Mismatches**: +```python +# Issue: Assigned value doesn't match type +value: int = "string" # Bad + +# Fix: +value: str = "string" # Good +# Or: +value: int = 42 # Good +``` + +**Optional Types**: +```python +# Issue: Value might be None +def get_name(user: User) -> str: + return user.name # mypy error if name can be None + +# Fix: +from typing import Optional + +def get_name(user: User) -> Optional[str]: + return user.name +``` + +**Generic Types**: +```python +# Issue: Missing generic type parameters +def process(items: list) -> None: # Bad + pass + +# Fix: +def process(items: list[str]) -> None: # Good + pass +``` + +### Format Issues + +**Code Not Formatted**: +```bash +# Issue: Code doesn't match ruff format style +# Fix: +ruff format . +``` + +## Workflow for Fixing Lint Issues + +### Step 1: Identify Issues +```bash +make lint +``` +Review output to understand what needs fixing. + +### Step 2: Auto-fix What You Can +```bash +# Fix ruff issues +ruff check --fix + +# Format code +ruff format . +``` + +### Step 3: Manual Fixes +Address remaining issues that require manual intervention: +- Type annotation problems +- Complex refactoring needs +- Logic errors flagged by linters + +### Step 4: Verify Fixes +```bash +make lint +``` +Ensure all checks pass. + +### Step 5: Test +```bash +make test +``` +Verify fixes didn't break functionality. + +## Code Quality Standards + +### Type Annotations + +**Required**: All code must be fully type-annotated + +```python +# Functions +def calculate(value: float, multiplier: float) -> float: + return value * multiplier + +# Methods +class MyComponent(Component): + def __init__(self, param: str, **kwargs: _t.Unpack[ComponentArgsDict]) -> None: + super().__init__(**kwargs) + self._param: str = param + +# Variables (when not obvious) +items: list[str] = [] +config: dict[str, Any] = {} +``` + +### Import Organization + +Order (enforced by ruff): +1. Standard library imports +2. Third-party imports +3. Local imports + +```python +# Standard library +import asyncio +from typing import Any + +# Third-party +from pydantic import BaseModel +import msgspec + +# Local +from plugboard.component import Component +from plugboard.schemas import ComponentArgsDict +``` + +### Code Style + +Follow Ruff's default style (similar to Black): +- Line length: 88 characters (configurable) +- 4-space indentation +- Trailing commas in multi-line structures +- Double quotes for strings + +## Common Patterns + +### Async Functions + +```python +async def process_data(self) -> None: + """Process data asynchronously.""" + result = await self.fetch_data() + await self.save_result(result) +``` + +### Type Unions + +```python +from typing import Union + +def handle_value(value: str | int) -> str: # Python 3.10+ + return str(value) + +# Or for older syntax: +def handle_value(value: Union[str, int]) -> str: + return str(value) +``` + +### Generic Collections + +```python +from typing import Any + +# Specific types preferred +items: list[str] = [] +mapping: dict[str, int] = {} + +# Use Any when truly needed +config: dict[str, Any] = {} +``` + +## Handling Edge Cases + +### Generated Code +If code is generated and shouldn't be linted: +- Add `# noqa` comments for specific lines +- Add file to exclusions in `pyproject.toml` + +### Type Checking Limitations +If mypy has issues with third-party libraries: +- Use `# type: ignore` comments sparingly +- Check if library has type stubs +- Consider using `cast()` for clarity + +### Legacy Code +When working with legacy code: +- Fix lint issues in files you modify +- Don't need to fix entire codebase +- Focus on changed lines and related code + +## Pre-commit Checks + +The project uses pre-commit hooks (`.pre-commit-config.yaml`). + +Running manually: +```bash +pre-commit run --all-files +``` + +## Resources + +- **Ruff**: https://docs.astral.sh/ruff/ +- **Mypy**: https://mypy.readthedocs.io/ +- **Type Hints**: https://docs.python.org/3/library/typing.html +- **PEP 484**: https://peps.python.org/pep-0484/ (Type Hints) +- **PEP 8**: https://peps.python.org/pep-0008/ (Style Guide) diff --git a/.github/instructions/models.instructions.md b/.github/instructions/models.instructions.md deleted file mode 100644 index a2a3cf8f..00000000 --- a/.github/instructions/models.instructions.md +++ /dev/null @@ -1,179 +0,0 @@ ---- -applyTo: "examples/**/*.py,examples/**/*.ipynb" ---- - -# Project Overview - -Plugboard is an event-driven modelling and orchestration framework in Python for simulating and driving complex processes with many interconnected stateful components. - -## Planning a model - -Help users to plan their models from a high-level overview to a detailed design. This should include: - -* The inputs and outputs of the model; -* The components that will be needed to implement each part of the model, and any inputs, outputs and parameters they will need; -* The data flow between components. - -For example, a model of a hot-water tank might have components for the water tank, the heater and the thermostat. Additional components might be needed to load data from a file or database, and similarly to save simulation results. - -## Implementing components - -Help users set up the components they need to implement their model. Custom components can be implemented by subclassing the [`Component`][plugboard.component.Component]. Common components for tasks like loading data can be imported from [`plugboard.library`][plugboard.library]. - -An empty component looks like this: - -```python -import typing as _t - -from plugboard.component import Component, IOController as IO -from plugboard.schemas import ComponentArgsDict - -class Offset(Component): - """Implements `x = a + offset`.""" - io = IO(inputs=["a"], outputs=["x"]) - - def __init__(self, offset: float = 0, **kwargs: _t.Unpack[ComponentArgsDict]) -> None: - super().__init__(**kwargs) - self._offset = offset - - async def step(self) -> None: - # TODO: Implement business logic here - # Example `self.x = self.a + self._offset` - pass -``` - -## Connecting components into a process - -You can help users to connect their components together. For initial development and testing use a [LocalProcess][plugboard.process.LocalProcess] to run the model in a single process. - -Example code to connect components together and create a process: - -```python -from plugboard.connector import AsyncioConnector -from plugboard.process import LocalProcess -from plugboard.schemas import ConnectorSpec - -connect = lambda in_, out_: AsyncioConnector( - spec=ConnectorSpec(source=in_, target=out_) -) -process = LocalProcess( - components=[ - Random(name="random", iters=5, low=0, high=10), - Offset(name="offset", offset=10), - Scale(name="scale", scale=2), - Sum(name="sum"), - Save(name="save-input", path="input.txt"), - Save(name="save-output", path="output.txt"), - ], - connectors=[ - # Connect x output of the component named "random" to the value_to_save input of the component named "save-input", etc. - connect("random.x", "save-input.value_to_save"), - connect("random.x", "offset.a"), - connect("random.x", "scale.a"), - connect("offset.x", "sum.a"), - connect("scale.x", "sum.b"), - connect("sum.x", "save-output.value_to_save"), - ], -) -``` - -If you need a diagram of the process you can import `plugboard.diagram.markdown_diagram` and use it to create a markdown representation of the process: - -```python -from plugboard.diagram import markdown_diagram -diagram = markdown_diagram(process) -print(diagram) -``` - -## Running the model - -You can help users to run their model. For example, to run the model defined above: - -```python - -import asyncio - -async with process: - await process.run() -``` - -## Event-driven models - -You can help users to implement event-driven models using Plugboard's event system. Components can emit and handle events to communicate with each other. - -Examples of where you might want to use events include: -* A component that monitors a data stream and emits an event when a threshold is crossed; -* A component that listens for events and triggers actions in response, e.g. sending an alert; -* A trading algorithm that uses events to signal buy/sell decisions. - -Events must be defined by inheriting from the `plugboard.events.Event` class. Each event class should define the data it carries using a Pydantic `BaseModel`. For example: - -```python -from pydantic import BaseModel -from plugboard.events import Event - -class MyEventData(BaseModel): - some_value: int - another_value: str - -class MyEvent(Event): - data: MyEventData -``` - -Components can emit events using the `self.io.queue_event()` method or by returning them from an event handler. Event handlers are defined using methods decorated with `@EventClass.handler`. For example: - -```python -from plugboard.component import Component, IOController as IO - -class MyEventPublisher(Component): - io = IO(inputs=["some_input"], output_events=[MyEvent]) - - async def step(self) -> None: - # Emit an event - event_data = MyEventData(some_value=42, another_value=f"received {self.some_input}") - self.io.queue_event(MyEvent(source=self.name, data=event_data)) - -class MyEventSubscriber(Component): - io = IO(input_events=[MyEvent], output_events=[MyEvent]) - - @MyEvent.handler - async def handle_my_event(self, event: MyEvent) -> MyEvent: - # Handle the event - print(f"Received event: {event.data}") - output_event_data = MyEventData(some_value=event.data.some_value + 1, another_value="handled") - return MyEvent(source=self.name, data=output_event_data) -``` - -To assemble a process with event-driven components, you can use the same approach as for non-event-driven components. You will need to create connectors for event-driven components using `plugboard.events.event_connector_builder.EventConnectorBuilder`. For example: - -```python -from plugboard.connector import AsyncioConnector, ConnectorBuilder -from plugboard.events.event_connector_builder import EventConnectorBuilder -from plugboard.process import LocalProcess - -# Define components.... -component_1 = ... -component_2 = ... - -# Define connectors for non-event components as before -connect = lambda in_, out_: AsyncioConnector(spec=ConnectorSpec(source=in_, target=out_)) -connectors = [ - connect("component_1.output", "component_2.input"), - ... -] - -connector_builder = ConnectorBuilder(connector_cls=AsyncioConnector) -event_connector_builder = EventConnectorBuilder(connector_builder=connector_builder) -event_connectors = list(event_connector_builder.build(components).values()) - -process = LocalProcess( - components=[ - component_1, component_2, ... - ], - connectors=connectors + event_connectors, -) -``` - -## Exporting models - -If the user wants to export their model you use in the CLI, you can do this by calling `process.dump("path/to/file.yaml")`. diff --git a/.github/instructions/source.instructions.md b/.github/instructions/source.instructions.md deleted file mode 100644 index 050a5602..00000000 --- a/.github/instructions/source.instructions.md +++ /dev/null @@ -1,51 +0,0 @@ ---- -applyTo: "plugboard/**/*.py" ---- -# GitHub Copilot Instructions for the Plugboard Repository - -This document provides guidelines for using AI coding agents to contribute to the Plugboard project. Following these instructions will help ensure that contributions are consistent with the project's architecture, conventions, and style. - -## Project Overview & Architecture - -Plugboard is an event-driven framework in Python for simulating and orchestrating complex processes. The core architectural concepts are `Component`, `Process`, and `Connector`. - -- **`Component`**: The fundamental building block for modeling logic. Found in `plugboard/component/`. - - Components have a defined lifecycle: `__init__`, `init`, `step`, `run`, and `destroy`. - - I/O (inputs, outputs, events) is declared via a class-level `io: IOController` attribute. - - The `step` method contains the primary logic and is executed repeatedly. - - The framework is asynchronous (`asyncio`), so all lifecycle methods (`init`, `step`, `destroy`) must be `async`. - -- **`Process`**: Manages a collection of `Component`s and their interconnections. Found in `plugboard/process/`. - - A `Process` orchestrates the execution of components. - - `LocalProcess` runs all components in a single process. Other process types may support distributed execution. - -- **`Connector`**: Defines the communication channels between component outputs and inputs. Found in `plugboard/connector/`. - - Connectors link a source (`component_name.output_name`) to a target (`component_name.input_name`). - -- **State Management**: The `StateBackend` (see `plugboard/state/`) tracks the status of all components and the overall process. This is crucial for monitoring and for distributed execution. - -- **Configuration**: Processes can be defined in Python or declared in YAML files for execution via the CLI (`plugboard process run ...`). - -## Developer Workflow - -- **Setup**: The project uses `uv` for dependency management. Set up your environment and install dependencies from `pyproject.toml`. -- **Testing**: Tests are written with `pytest` and are located in the `tests/` directory. - - Run all tests with `make test`. - - Run integration tests with `make test-integration`. - - When adding a new feature, please include corresponding unit and/or integration tests. -- **Linting & Formatting**: The project uses `ruff` for formatting and linting, and `mypy` for static type checking. - - Run `make lint` to check for issues. - - Run `make format` to automatically format the code. - - All code must be fully type-annotated. -- **CLI**: The command-line interface is defined using `typer` in `plugboard/cli/`. Use `plugboard --help` to see available commands. - -## Code Conventions & Patterns - -- **Asynchronous Everywhere**: The entire framework is built on `asyncio`. All I/O operations and component lifecycle methods should be `async`. -- **Dependency Injection**: The project uses `that-depends` for dependency injection. See `plugboard/utils/DI.py` for the container setup. -- **Immutability**: Use `msgspec.Struct(frozen=True)` for data structures that should be immutable. -- **Extending Components**: When creating a new component, inherit from `plugboard.component.Component` and implement the required `async` methods. Remember to call `super().__init__()`. -- **Events**: Components can communicate via an event system. Define custom events by inheriting from `plugboard.events.Event` and add handlers to your component using the `@Event.handler` decorator. -- **Logging**: Use the `structlog` logger available through dependency injection: `self._logger = DI.logger.resolve_sync().bind(...)`. - -By adhering to these guidelines, you can help maintain the quality and consistency of the Plugboard codebase. diff --git a/.vscode/settings.json b/.vscode/settings.json new file mode 100644 index 00000000..14e1340e --- /dev/null +++ b/.vscode/settings.json @@ -0,0 +1,16 @@ +{ + "github.copilot.chat.codeGeneration.instructions": [ + { + "file": "AGENTS.md" + }, + { + "file": "examples/AGENTS.md" + }, + { + "file": ".github/agents/docs/AGENTS.md" + }, + { + "file": ".github/agents/linting/AGENTS.md" + } + ] +} diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 00000000..1abc29a2 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,154 @@ +# AI Agent Instructions for Plugboard + +This document provides guidelines for AI coding agents working on the Plugboard project. Following these instructions ensures contributions are consistent with the project's architecture, conventions, and style. + +## Project Overview + +Plugboard is an event-driven framework in Python for simulating and orchestrating complex processes with interconnected stateful components. + +### Core Architecture + +**Component**: The fundamental building block for modeling logic (see `plugboard/component/`) +- Lifecycle: `__init__` → `init` → `step` (repeated) → `destroy` +- I/O declaration: Use class-level `io: IOController` attribute +- Business logic: Implement in the `step` method +- Asynchronous: All lifecycle methods (`init`, `step`, `destroy`) must be `async` + +**Process**: Orchestrates execution of components (see `plugboard/process/`) +- Manages collections of components and their interconnections +- `LocalProcess`: Runs all components in a single process +- Supports distributed execution via other process types + +**Connector**: Defines communication channels between components (see `plugboard/connector/`) +- Links outputs to inputs: `component_name.output_name` → `component_name.input_name` +- Various connector types available for different execution contexts + +**State Management**: Tracks component and process status (see `plugboard/state/`) +- Critical for monitoring and distributed execution +- Uses `StateBackend` abstraction + +**Configuration**: Flexible process definition +- Python-based: Direct component instantiation +- YAML-based: Declarative configuration for CLI execution (`plugboard process run ...`) + +## Development Environment + +### Setup +- **Package Manager**: Uses `uv` for dependency management +- **Dependencies**: Defined in `pyproject.toml` +- **Python Version**: Requires Python ≥3.12 + +### Testing +- **Framework**: `pytest` +- **Location**: `tests/` directory +- **Commands**: + - `make test` - Run all tests + - `make test-integration` - Run integration tests +- **Best Practice**: Always include tests with new features + +### Linting & Formatting +- **Tools**: + - `ruff` - Formatting and linting + - `mypy` - Static type checking +- **Commands**: + - `make lint` - Check for issues + - `make format` - Auto-format code +- **Requirement**: All code must be fully type-annotated + +### CLI +- **Framework**: Built with `typer` +- **Location**: `plugboard/cli/` +- **Usage**: `plugboard --help` + +## Code Standards + +### Async Pattern +- Entire framework built on `asyncio` +- All I/O operations must be async +- All component lifecycle methods must be async + +### Dependency Injection +- Uses `that-depends` for DI +- Container setup: `plugboard/utils/DI.py` +- Access logger: `self._logger = DI.logger.resolve_sync().bind(...)` + +### Data Structures +- Prefer immutable structures: `msgspec.Struct(frozen=True)` +- Use Pydantic models for validation where needed + +### Components +When creating components: +1. Inherit from `plugboard.component.Component` +2. Always call `super().__init__()` in `__init__` +3. Declare I/O via class-level `io` attribute +4. Implement required async methods +5. Use type hints throughout + +Example: +```python +import typing as _t +from plugboard.component import Component, IOController as IO +from plugboard.schemas import ComponentArgsDict + +class MyComponent(Component): + io = IO(inputs=["input_a"], outputs=["output_x"]) + + def __init__( + self, + param: float = 1.0, + **kwargs: _t.Unpack[ComponentArgsDict] + ) -> None: + super().__init__(**kwargs) + self._param = param + + async def step(self) -> None: + # Business logic here + self.output_x = self.input_a * self._param +``` + +### Events +- Event system for component communication +- Define events by inheriting from `plugboard.events.Event` +- Add handlers with `@Event.handler` decorator +- Emit events via `self.io.queue_event()` or return from handlers + +## Best Practices + +1. **Minimal Changes**: Make surgical, focused changes +2. **Type Safety**: Maintain full type annotations +3. **Testing**: Add tests for new functionality +4. **Documentation**: Update docstrings and docs for public APIs +5. **Async Discipline**: Never use blocking I/O operations +6. **Immutability**: Prefer immutable data structures +7. **Logging**: Use structured logging via `structlog` +8. **Error Handling**: Use appropriate exception types from `plugboard.exceptions` + +## Common Tasks + +### Adding a New Component +1. Create class inheriting from `Component` +2. Define `io` with inputs/outputs +3. Implement `__init__` with proper signature +4. Implement async `step` method +5. Add tests in `tests/` +6. Update documentation if public API + +### Modifying Core Framework +1. Understand impact on existing components +2. Ensure backward compatibility where possible +3. Update type stubs if needed +4. Run full test suite +5. Update relevant documentation + +### Working with Events +1. Define event class with data model +2. Declare in component's `io` (input_events/output_events) +3. Implement handlers with decorators +4. Use `EventConnectorBuilder` for wiring + +## Resources + +- **Repository**: https://github.com/plugboard-dev/plugboard +- **Documentation**: https://docs.plugboard.dev +- **Issue Tracker**: GitHub Issues +- **Discussions**: GitHub Discussions diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 2c60c9ad..d38121ae 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -40,6 +40,13 @@ The package documentation uses [Material for MkDocs](https://squidfunk.github.io uv run mkdocs serve ``` -### Building example models +### AI-assisted development -This repo includes a [custom LLM prompt](.github/instructions/models.instructions.md) for the [examples](examples/) folder. If you use GitHub Copilot, this can help you build a Plugboard model from a description of the process and/or the components that you would like to implement. We recommend using Copilot in agent mode and allowing it to implement the boilerplate code from your input prompt. +This repo includes custom AI agent prompts to assist with development: + +- [AGENTS.md](AGENTS.md) - General guidelines for working with the Plugboard codebase +- [examples/AGENTS.md](examples/AGENTS.md) - Specific guidance for building example models and demos +- [.github/agents/docs/AGENTS.md](.github/agents/docs/AGENTS.md) - Documentation maintenance agent +- [.github/agents/linting/AGENTS.md](.github/agents/linting/AGENTS.md) - Linting and code quality agent + +If you use GitHub Copilot or other AI coding assistants that support the AGENTS.md convention, these prompts can help you build Plugboard models from a description of the process and/or the components that you would like to implement. We recommend using Copilot in agent mode and allowing it to implement the boilerplate code from your input prompt. diff --git a/examples/AGENTS.md b/examples/AGENTS.md new file mode 100644 index 00000000..e7e69fce --- /dev/null +++ b/examples/AGENTS.md @@ -0,0 +1,358 @@ +# AI Agent Instructions for Plugboard Examples + +This document provides guidelines for AI agents working with Plugboard example code, demonstrations, and tutorials. + +## Purpose + +These examples demonstrate how to use Plugboard to model and simulate complex processes. Help users build intuitive, well-documented examples that showcase Plugboard's capabilities. + +## Example Categories + +### Tutorials (`tutorials/`) +Step-by-step learning materials for new users. Focus on: +- Clear explanations of concepts +- Progressive complexity +- Runnable code with expected outputs +- Markdown documentation alongside code + +### Demos (`demos/`) +Practical applications organized by domain: +- `fundamentals/`: Core Plugboard concepts +- `llm/`: LLM and AI integrations +- `physics-models/`: Physics-based simulations +- `finance/`: Financial modeling examples + +## Creating Examples + +### Planning a Model + +Help users structure their model from concept to implementation: + +1. **Define Scope** + - Clear problem statement + - Expected inputs and outputs + - Success criteria + +2. **Component Design** + - Break problem into logical components + - Define each component's inputs, outputs, and parameters + - Identify data dependencies and flow + +3. **Data Flow** + - Map connections between components + - Identify feedback loops + - Plan event-driven interactions if needed + +Example breakdown for a hot-water tank model: +- **Components**: WaterTank, Heater, Thermostat, DataLoader, ResultSaver +- **Flow**: Temperature sensor → Thermostat → Heater → Tank → Temperature sensor +- **Data**: Load initial conditions, save temperature over time + +### Implementing Components + +Use appropriate components from the library or create custom ones: + +**Using Built-in Components** +```python +from plugboard.library import Load, Save, Random + +# Load data from file +data_loader = Load(name="load_data", path="input.csv") + +# Save results +results_saver = Save(name="save_results", path="output.csv") +``` + +**Creating Custom Components** +```python +import typing as _t +from plugboard.component import Component, IOController as IO +from plugboard.schemas import ComponentArgsDict + +class Offset(Component): + """Adds a constant offset to input value.""" + io = IO(inputs=["a"], outputs=["x"]) + + def __init__( + self, + offset: float = 0, + **kwargs: _t.Unpack[ComponentArgsDict] + ) -> None: + super().__init__(**kwargs) + self._offset = offset + + async def step(self) -> None: + self.x = self.a + self._offset +``` + +### Assembling a Process + +Connect components and create a runnable process: + +```python +from plugboard.connector import AsyncioConnector +from plugboard.process import LocalProcess +from plugboard.schemas import ConnectorSpec + +# Helper for creating connectors +connect = lambda src, tgt: AsyncioConnector( + spec=ConnectorSpec(source=src, target=tgt) +) + +# Create process with components and connectors +process = LocalProcess( + components=[ + Random(name="random", iters=5, low=0, high=10), + Offset(name="offset", offset=10), + Scale(name="scale", scale=2), + Sum(name="sum"), + Save(name="save-input", path="input.txt"), + Save(name="save-output", path="output.txt"), + ], + connectors=[ + connect("random.x", "save-input.value_to_save"), + connect("random.x", "offset.a"), + connect("random.x", "scale.a"), + connect("offset.x", "sum.a"), + connect("scale.x", "sum.b"), + connect("sum.x", "save-output.value_to_save"), + ], +) +``` + +### Visualizing Process Flow + +Generate diagrams to help understand the model: + +```python +from plugboard.diagram import markdown_diagram + +diagram = markdown_diagram(process) +print(diagram) +``` + +### Running the Model + +Execute the process asynchronously: + +```python +import asyncio + +async def main(): + async with process: + await process.run() + +if __name__ == "__main__": + asyncio.run(main()) +``` + +## Event-Driven Models + +For models requiring dynamic interactions and responses: + +### When to Use Events +- Monitoring thresholds and triggering alerts +- Conditional workflows (if X happens, do Y) +- Multi-agent systems with communication +- Real-time data processing with decision points + +### Defining Events + +```python +from pydantic import BaseModel +from plugboard.events import Event + +class ThresholdCrossedData(BaseModel): + component: str + value: float + threshold: float + timestamp: float + +class ThresholdCrossed(Event): + data: ThresholdCrossedData +``` + +### Publishing Events + +```python +from plugboard.component import Component, IOController as IO + +class Monitor(Component): + io = IO( + inputs=["value"], + outputs=[], + output_events=[ThresholdCrossed] + ) + + def __init__(self, threshold: float, **kwargs) -> None: + super().__init__(**kwargs) + self._threshold = threshold + + async def step(self) -> None: + if self.value > self._threshold: + event_data = ThresholdCrossedData( + component=self.name, + value=self.value, + threshold=self._threshold, + timestamp=time.time() + ) + self.io.queue_event( + ThresholdCrossed(source=self.name, data=event_data) + ) +``` + +### Subscribing to Events + +```python +class AlertHandler(Component): + io = IO( + inputs=[], + outputs=[], + input_events=[ThresholdCrossed], + output_events=[AlertSent] + ) + + @ThresholdCrossed.handler + async def handle_threshold(self, event: ThresholdCrossed) -> AlertSent: + print(f"ALERT: {event.data.component} exceeded threshold!") + + # Return new event + alert_data = AlertSentData( + original_event=event.data, + alert_method="console" + ) + return AlertSent(source=self.name, data=alert_data) +``` + +### Wiring Event-Driven Components + +```python +from plugboard.connector import ConnectorBuilder +from plugboard.events.event_connector_builder import EventConnectorBuilder + +# Regular connectors +connect = lambda src, tgt: AsyncioConnector( + spec=ConnectorSpec(source=src, target=tgt) +) +connectors = [ + connect("data_source.value", "monitor.value"), + # ... other data connectors +] + +# Event connectors (automatically wired) +connector_builder = ConnectorBuilder(connector_cls=AsyncioConnector) +event_connector_builder = EventConnectorBuilder( + connector_builder=connector_builder +) +components = [monitor, alert_handler, ...] +event_connectors = list( + event_connector_builder.build(components).values() +) + +# Create process with both types +process = LocalProcess( + components=components, + connectors=connectors + event_connectors, +) +``` + +## Exporting Models + +Save process configuration for reuse: + +```python +# Export to YAML +process.dump("my_model.yaml") + +# Later, load and run via CLI +# $ plugboard process run my_model.yaml +``` + +## Jupyter Notebooks + +For interactive demonstrations: + +1. **Structure** + - Clear markdown sections + - Code cells with explanations + - Visualizations of results + - Summary of findings + +2. **Best Practices** + - Keep cells focused and small + - Add docstrings to helper functions + - Show intermediate results + - Include error handling + - Clean up resources properly + +3. **Output** + - Include sample outputs in notebook + - Generate plots where helpful + - Provide interpretation of results + +## Documentation Standards + +### Code Comments +- Explain *why*, not *what* +- Document assumptions +- Note limitations or edge cases + +### Docstrings +- Use Google-style docstrings +- Document all parameters and return values +- Include usage examples for complex functions + +### Markdown Files +- Clear headings and structure +- Code blocks with syntax highlighting +- Link to relevant API documentation +- Include expected outputs + +## Testing Examples + +While examples are primarily educational: +- Ensure all code actually runs +- Verify outputs are as expected +- Check for common error cases +- Test with different parameter values + +## Common Patterns + +### Data Processing Pipeline +```python +components = [ + Load(name="load", path="data.csv"), + Transform(name="transform", ...), + Filter(name="filter", ...), + Save(name="save", path="output.csv"), +] +``` + +### Simulation Loop +```python +components = [ + Clock(name="clock", steps=100), + Model(name="model", ...), + Observer(name="observer", ...), + Save(name="save", ...), +] +``` + +### Multi-Agent System +```python +components = [ + Agent(name="agent_1", ...), + Agent(name="agent_2", ...), + Environment(name="env", ...), + Coordinator(name="coord", ...), +] +# Use events for agent communication +``` + +## Resources + +- **Library Components**: `plugboard.library` +- **Component Base Class**: `plugboard.component.Component` +- **Process Types**: `plugboard.process` +- **Event System**: `plugboard.events` +- **API Documentation**: https://docs.plugboard.dev