|
| 1 | +# Feature specification: Artifact content composition via Jinja2 filters |
| 2 | + |
| 3 | +**Feature Branch**: `infp-504-artifact-composition` |
| 4 | +**Created**: 2026-02-18 |
| 5 | +**Status**: Draft |
| 6 | +**Jira**: INFP-504 (part of INFP-304 Artifact of Artifacts initiative) |
| 7 | + |
| 8 | +## Overview |
| 9 | + |
| 10 | +Enable customers building modular configuration pipelines to compose larger artifacts from smaller sub-artifacts by referencing and inlining rendered artifact content directly inside a Jinja2 transform, without duplicating template logic or GraphQL query fields. |
| 11 | + |
| 12 | +## User scenarios & testing *(mandatory)* |
| 13 | + |
| 14 | +### User story 1 - inline artifact content in a composite template (Priority: P1) |
| 15 | + |
| 16 | +A network engineer maintains separate section-level artifacts for routing policy, interfaces, and base config. They want a composite "startup config" artifact whose Jinja2 template pulls in each section's rendered content via a `storage_id` already present in the GraphQL query result — without copy-pasting template logic. |
| 17 | + |
| 18 | +The template uses `artifact.node.storage_id.value | artifact_content` and the rendered output assembles all sections automatically. |
| 19 | + |
| 20 | +**Why this priority**: This is the primary use case that delivers the modular pipeline capability. Everything else in this feature supports or extends it. |
| 21 | + |
| 22 | +**Independent Test**: A Jinja2 template calling `artifact_content` with a valid storage_id can be rendered against a real or mocked Infrahub instance and the output matches the expected concatenated artifact contents. |
| 23 | + |
| 24 | +**Acceptance Scenarios**: |
| 25 | + |
| 26 | +1. **Given** a `Jinja2Template` constructed with a valid `InfrahubClient` and a template calling `storage_id | artifact_content`, **When** the template is rendered with a data dict containing a valid storage_id string, **Then** the output contains the raw string content fetched from the object store. |
| 27 | +2. **Given** the same setup but the storage_id is null or the object store cannot retrieve the content, **When** rendered, **Then** the filter raises a descriptive error indicating the retrieval failure. |
| 28 | +3. **Given** a `Jinja2Template` constructed *without* an `InfrahubClient` and a template calling `artifact_content`, **When** rendered, **Then** an error is raised with a message clearly stating that an `InfrahubClient` is required for this filter. |
| 29 | +4. **Given** a template using `artifact_content` and `validate(restricted=True)` is called, **Then** a `JinjaTemplateOperationViolationError` is raised, confirming the filter is blocked in local restricted mode. |
| 30 | + |
| 31 | +--- |
| 32 | + |
| 33 | +### User story 2 - inline file object content in a composite template (Priority: P2) |
| 34 | + |
| 35 | +A template author needs to embed the content of a stored file object (as distinct from an artifact) into a Jinja2 template. They use `storage_id | file_object_content` and the same injection and error-handling behaviour applies. |
| 36 | + |
| 37 | +**Why this priority**: Mirrors `artifact_content` for the file-object use case; same implementation pattern, lower novelty. |
| 38 | + |
| 39 | +**Independent Test**: A template calling `file_object_content` renders correctly with a valid storage_id, and raises a descriptive error for null or unresolvable storage_ids. |
| 40 | + |
| 41 | +**Acceptance Scenarios**: |
| 42 | + |
| 43 | +1. **Given** a `Jinja2Template` with a client and a valid file-object storage_id, **When** rendered, **Then** the raw file content string is returned. |
| 44 | +2. **Given** a null or missing storage_id value, **When** the filter is invoked, **Then** an error is raised with a descriptive message about the retrieval failure. |
| 45 | +3. **Given** no client provided to `Jinja2Template`, **When** the filter is invoked, **Then** an error is raised. |
| 46 | + |
| 47 | +--- |
| 48 | + |
| 49 | +### User story 3 - parse structured artifact content in a template (Priority: P3) |
| 50 | + |
| 51 | +A template author retrieves a JSON-formatted artifact and needs to traverse its structure as a dict within the template. They chain `storage_id | artifact_content | from_json` to obtain a parsed object, then access fields normally. |
| 52 | + |
| 53 | +**Why this priority**: Unlocks structured composition use cases; depends on `artifact_content` (P1) being in place. `from_json`/`from_yaml` are useful in isolation too. |
| 54 | + |
| 55 | +**Independent Test**: A template chaining `artifact_content | from_json` renders correctly and the output reflects values from parsed JSON fields. |
| 56 | + |
| 57 | +**Acceptance Scenarios**: |
| 58 | + |
| 59 | +1. **Given** a template using `storage_id | artifact_content | from_json`, **When** rendered with a storage_id pointing to valid JSON content, **Then** the template can access keys of the parsed object. |
| 60 | +2. **Given** `storage_id | artifact_content | from_yaml`, **When** rendered with YAML content, **Then** the template can access keys of the parsed mapping. |
| 61 | +3. **Given** `from_json` or `from_yaml` applied to an empty string (for example, a template variable that is explicitly empty), **When** rendered, **Then** the filter returns an empty dict or appropriate empty value without raising. |
| 62 | + |
| 63 | +--- |
| 64 | + |
| 65 | +### User story 4 - security gate blocks filters in computed attributes context (Priority: P1) |
| 66 | + |
| 67 | +The Infrahub API server executes computed attributes locally and must block `artifact_content` and `file_object_content` because no network calls should be made within that context. Prefect workers run inside Infrahub with a client and must be able to use these filters. Other currently-untrusted Jinja2 filters (for example, `safe`, `attr`) must remain subject to their existing restriction rules — this feature must not inadvertently widen their permissions. |
| 68 | + |
| 69 | +The existing single `restricted: bool` parameter on `validate()` is insufficient: flipping it to `False` to permit Infrahub filters would also permit all other untrusted filters. The validation mechanism must be extended to express at least three distinct execution contexts. |
| 70 | + |
| 71 | +**Why this priority**: Preventing these filters from running in the computed attributes context is a hard requirement. Shares P1 priority with User Story 1. |
| 72 | + |
| 73 | +**Independent Test**: Validation in the computed-attributes context raises `JinjaTemplateOperationViolationError` for templates using `artifact_content` or `file_object_content`. Validation in the Prefect-worker context passes for the same templates. Neither context changes the restriction behaviour of other currently-untrusted filters. |
| 74 | + |
| 75 | +**Acceptance Scenarios**: |
| 76 | + |
| 77 | +1. **Given** a template referencing `artifact_content`, **When** validated in the computed-attributes context, **Then** `JinjaTemplateOperationViolationError` is raised. |
| 78 | +2. **Given** the same template, **When** validated in the Prefect-worker context with a client-initialised `Jinja2Template`, **Then** validation passes. |
| 79 | +3. **Given** a template using an existing untrusted filter (for example, `safe`), **When** validated in the Prefect-worker context, **Then** `JinjaTemplateOperationViolationError` is still raised — the Prefect-worker context does not unlock other untrusted filters. |
| 80 | + |
| 81 | +--- |
| 82 | + |
| 83 | +### Edge cases |
| 84 | + |
| 85 | +- What happens if a storage_id value is `None` (Python None) rather than a missing string? Both cases must raise a descriptive error. |
| 86 | +- What if the object store raises a network or authentication error mid-render? All error conditions (null storage_id, not-found, auth failure, network failure) raise exceptions — there is no silent fallback. |
| 87 | +- What if `from_json` or `from_yaml` already exists in the netutils filter set? De-duplicate rather than shadow. |
| 88 | +- What happens when `from_json` or `from_yaml` receives malformed content (invalid JSON/YAML syntax)? `JinjaFilterError` is raised — no silent fallback. |
| 89 | +- What if the same filter name is registered twice (for example, a user-supplied filter that shadows `artifact_content`)? Existing override behaviour should be preserved. |
| 90 | +- File-based templates use a regular `Environment` (not sandboxed); the new filters must be injected correctly in both cases. |
| 91 | + |
| 92 | +## Requirements *(mandatory)* |
| 93 | + |
| 94 | +### Functional requirements |
| 95 | + |
| 96 | +- **FR-001**: `Jinja2Template.__init__` MUST accept an optional `client` parameter of type `InfrahubClient | None` (default `None`). `InfrahubClientSync` is not supported. |
| 97 | +- **FR-002**: A dedicated class (for example, `InfrahubFilters`) MUST be introduced to hold the client reference and expose the Infrahub-specific filter callable methods. `Jinja2Template` instantiates this class when a client is provided and registers its filters into the Jinja2 environment. |
| 98 | +- **FR-003**: The system MUST provide an `artifact_content` Jinja2 filter that accepts a `storage_id` string and returns the raw string content of the referenced artifact, using the artifact-specific API path. |
| 99 | +- **FR-004**: The system MUST provide a `file_object_content` Jinja2 filter that accepts a `storage_id` string and returns the raw string content of the referenced file object, using the file-object-specific API path or metadata handling — this implementation is distinct from `artifact_content`. |
| 100 | +- **FR-005**: Both `artifact_content` and `file_object_content` MUST raise `JinjaFilterError` when the input `storage_id` is null or empty, or when the object store cannot retrieve the content for any reason (not found, network failure, auth failure). |
| 101 | +- **FR-006**: Both `artifact_content` and `file_object_content` MUST raise `JinjaFilterError` when invoked and no `InfrahubClient` was supplied to `Jinja2Template` at construction time. The error message MUST name the filter and explain that an `InfrahubClient` is required. |
| 102 | +- **FR-007**: Both `artifact_content` and `file_object_content` MUST be registered with `trusted=False` in the `FilterDefinition` registry so that `validate(restricted=True)` blocks them in the computed attributes execution context (Infrahub API server). They are only permitted to execute on Prefect workers, where an `InfrahubClient` is available. |
| 103 | +- **FR-008**: The system MUST provide `from_json` and `from_yaml` Jinja2 filters (adding them only if not already present in the environment) that parse a string into a Python dict/list. Applying them to an empty string MUST return an empty dict without raising. Applying them to malformed content MUST raise `JinjaFilterError`. |
| 104 | +- **FR-009**: `from_json` and `from_yaml` MUST be registered as trusted filters (`trusted=True`) since they perform no external I/O. |
| 105 | +- **FR-010**: All new filters MUST work correctly with `InfrahubClient` (async). `InfrahubClientSync` is not a supported client type for `Jinja2Template`. |
| 106 | +- **FR-011**: All `JinjaFilterError` instances MUST carry an actionable error message that identifies the filter name, the cause of failure, and any remediation hint (for example: "artifact_content requires an InfrahubClient — pass one via Jinja2Template(client=...)"). |
| 107 | +- **FR-012**: A new `JinjaFilterError` exception class MUST be added to `infrahub_sdk/template/exceptions.py` as a subclass of `JinjaTemplateError`. |
| 108 | +- **FR-013**: Documentation MUST include a Python transform example demonstrating artifact content retrieval via `client.object_store.get(identifier=storage_id)`. No new SDK convenience method will be added. |
| 109 | + |
| 110 | +### Key entities |
| 111 | + |
| 112 | +- **`Jinja2Template`**: Gains an optional `client` constructor parameter; delegates client-bound filter registration to `InfrahubFilters`. |
| 113 | +- **`InfrahubFilters`**: New class that holds an `InfrahubClient` reference and exposes `artifact_content`, `file_object_content`, and any other client-dependent filter methods. Registered into the Jinja2 filter map when a client is provided. |
| 114 | +- **`FilterDefinition`**: Existing dataclass used to declare filter `name`, `trusted` flag, and `source`. New entries are added here for all new filters. |
| 115 | +- **`ObjectStore` / `ObjectStoreSync`**: Existing async/sync storage clients used by `InfrahubFilters` to perform `get(identifier=storage_id)` calls. |
| 116 | +- **`JinjaFilterError`**: New exception class, subclass of `JinjaTemplateError`, raised by `InfrahubFilters` methods on all filter-level failures (no client, null/empty storage_id, retrieval error). |
| 117 | + |
| 118 | +## Success criteria *(mandatory)* |
| 119 | + |
| 120 | +### Measurable outcomes |
| 121 | + |
| 122 | +- **SC-001**: A composite Jinja2 artifact template using `artifact_content` renders successfully end-to-end (integration test), with output containing all expected sub-artifact content. |
| 123 | +- **SC-002**: `validate(restricted=True)` on any template referencing `artifact_content` or `file_object_content` always raises a security violation — zero false negatives across the test suite. |
| 124 | +- **SC-003**: All filter error conditions (no client, null/empty storage_id, retrieval failure) produce a descriptive, actionable error message — no silent failures, no raw tracebacks as the primary user-facing message. |
| 125 | +- **SC-004**: The async execution path (`InfrahubClient`) is covered by unit tests with no regressions to existing filter behaviour. |
| 126 | +- **SC-005**: The full unit test suite (`uv run pytest tests/unit/`) passes without modification after the feature is added. |
| 127 | +- **SC-006**: A template chaining `artifact_content | from_json` or `artifact_content | from_yaml` can access parsed fields from a structured artifact in a rendered output. |
| 128 | + |
| 129 | +## Assumptions |
| 130 | + |
| 131 | +- The `artifact_content` and `file_object_content` filters receive a `storage_id` string directly from the template variable context — extracted from the GraphQL query result by the template author. The filter does not resolve artifact names — it operates on storage IDs only. |
| 132 | +- Ordering of artifact generation is a known limitation: artifacts may be generated in parallel. This is a documented constraint, not something this feature enforces. Future event-driven pipeline work (INFP-227) will address ordering. |
| 133 | +- `from_json` and `from_yaml` are not currently present in the builtin or netutils filter sets; they will be added as part of this feature. If they already exist, the implementation de-duplicates rather than overrides. |
| 134 | +- All failure modes from the filters (null storage_id, empty storage_id, object not found, network error, auth error) raise exceptions. There is no silent fallback to an empty string. |
| 135 | +- The permitted execution context for `artifact_content` and `file_object_content` is Prefect workers only. The computed attributes path in the Infrahub API server always runs `validate(restricted=True)`, which blocks these filters before rendering begins. |
| 136 | +- The `InfrahubFilters` class provides synchronous callables to Jinja2's filter map; the underlying client is always `InfrahubClient` (async). Async I/O calls are handled consistently with the SDK's existing pattern. |
| 137 | + |
| 138 | +## Dependencies & constraints |
| 139 | + |
| 140 | +- Depends on `ObjectStore.get(identifier)` in `infrahub_sdk/object_store.py`. |
| 141 | +- Depends on the existing `FilterDefinition` dataclass and `trusted` flag mechanism in `infrahub_sdk/template/filters.py`. |
| 142 | +- Depends on the existing `validate(restricted=True)` security mechanism in `Jinja2Template`. |
| 143 | +- Must not break any existing filter behaviour or the `validate()` contract. |
| 144 | +- No new external Python dependencies may be introduced without approval. |
| 145 | +- Related: INFP-304 (Artifact of Artifacts), INFP-496 (Modular GraphQL queries), INFP-227 (Modular generators / event-driven pipeline). |
| 146 | + |
| 147 | +## Open questions |
| 148 | + |
| 149 | +- **Filter naming**: `artifact_content` is the working name. Alternatives are open. |
| 150 | +- **Sandboxed environment injection**: The `render_jinja2_template` method in `integrator.py` has access to `self.sdk`; the exact threading path to pass the client into `Jinja2Template` needs investigation during planning. |
| 151 | +- **Validation level model**: The current `validate(restricted: bool)` parameter is too coarse to express the three distinct execution contexts this feature requires. A natural evolution would be to replace the boolean with an enum (for example: `core` for the Infrahub API server, `worker` for Prefect background workers, `untrusted` for fully restricted local execution). Filters tagged as `worker`-only would be blocked in the `core` context but permitted in the `worker` context, while `trusted` filters remain available in all contexts. The exact enum design and migration of existing call sites is a technical decision for the implementation plan, but the interface change should be considered up front to avoid needing to revisit `validate()` again later. |
| 152 | + |
| 153 | +## Clarifications |
| 154 | + |
| 155 | +### Session 2026-02-18 |
| 156 | + |
| 157 | +- Q: Are `artifact_content` and `file_object_content` identical at the storage API level, or do they use different API paths / metadata handling? → A: Different implementations — `file_object_content` uses a different API path or carries different metadata handling than `artifact_content`. |
| 158 | +- Q: Where are these filters permitted to execute, and what mechanism enforces the boundary? → A: Blocked in computed attributes (executed locally in the Infrahub API server, which uses `validate(restricted=True)`); permitted on Prefect workers, which have access to an `InfrahubClient`. The `trusted=False` registration enforces this boundary via the existing restricted-mode validation. |
| 159 | +- Q: What exception class should filter-level errors (no client, retrieval failure) raise? → A: A new `JinjaFilterError` class that is a child of the existing `JinjaTemplateError` base class. |
| 160 | +- Q: Should the SDK expose a convenience method for artifact content retrieval in Python transforms? → A: No new method — document `client.object_store.get(identifier=storage_id)` directly. |
| 161 | +- Q: What should `from_json`/`from_yaml` do on malformed input? → A: Raise `JinjaFilterError` on malformed JSON or YAML input. |
0 commit comments