Skip to content

LOG-9344: add context files for agentic SDLC #269

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
vparfonov wants to merge 1 commit into
base: v0.54.0-rh
Choose a base branch
from
Open

LOG-9344: add context files for agentic SDLC #269

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
137 changes: 103 additions & 34 deletions AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,51 +1,120 @@
# AGENTS.md
# AGENTS.md — Red Hat Vector Fork Context

This file provides guidance to AI models and human developers working with code in this repository.
This file provides context for AI agents and developers working in this repository. It covers only what is specific to this Red Hat fork. For generic Vector development commands, project structure, and Rust conventions, see the [upstream AGENTS.md](https://github.com/vectordotdev/vector/blob/master/AGENTS.md).

## Project Overview
## What This Repo Is

This repository is a Red Hat fork of the upstream Vector project, specifically maintained for OpenShift Logging. It serves as a log collector and forwarder that runs as a daemon set on each OpenShift cluster node to gather application and node logs.
A fork of [vectordotdev/vector](https://github.com/vectordotdev/vector) maintained by Red Hat for OpenShift Logging. Vector runs as a DaemonSet on every OpenShift node, collecting application and infrastructure logs. It is deployed and configured by the [cluster-logging-operator](https://github.com/openshift/cluster-logging-operator) — Vector is never run standalone in production.

**Important**: The `rh-main` branch is intentionally minimal and contains only documentation. All active development occurs in version-specific branches.
**Upstream remote:** `vectordev` (git@github.com:vectordotdev/vector.git)
**Fork origin:** `origin` (git@github.com:ViaQ/vector.git)

## Branch Structure
## Critical: Branch Selection

The actual implementation code resides in version-specific branches, not in `rh-main`. Current active branches include:
**Never commit code to `rh-main`.** It contains only documentation.

- `v0.47.0-rh` - OpenShift 6.4 (Vector v0.47.0)
- `v0.37.1-rh` - OpenShift 6.0-6.3 (Vector v0.37.1)
- `release-5.9` - OpenShift 5.9 (Vector v0.34.1)
- `release-5.8` - OpenShift 5.8 (Vector v0.28.1)
| Branch | Vector Version | Rust | Use |
|--------------|----------------|------|----------------------------------|
| `v0.47.0-rh` | v0.47.0 | 1.85 | Stable — OpenShift 6.2–6.5 |
| `v0.54.0-rh` | v0.54.0 | 1.92 | Next — future OpenShift releases |

## Development Workflow
Always check out the appropriate version branch before making changes:

When working on this project:
```bash
git checkout v0.47.0-rh # stable
git checkout v0.54.0-rh # next
```

1. **Branch Selection**: Switch to the appropriate version branch for your target OpenShift release
2. **Build System**: This is a Rust project using Cargo as the build system
3. **Integration**: The project is designed to be deployed and configured by the [cluster-logging-operator](https://github.com/openshift/cluster-logging-operator)
4. **Runtime**: Runs with configuration and `run.sh` files provided by the operator
When a fix applies to both branches, submit separate PRs per branch.

## Red Hat Specific Modifications
## Build Quick Reference

This fork includes several patches not found in upstream Vector:
The Makefile defaults to `FEATURES=ocp-logging` on Linux (not upstreams `default` feature set).

- **OpenSSL Integration** (LOG-2552): Replaces Ring with OpenSSL for Red Hat compliance
- **TLS Security Profile** (LOG-3398): Applies OpenShift TLS security settings
- **Syslog Sink** (LOG-2288): Custom syslog output capability
- **File Rotation** (LOG-3949): Enhanced file rotation controls
- **AWS Credentials** (LOG-6789): Fixes for AWS credentials file authentication
- **CloudWatch Limits** (LOG-7013): Updated to match current AWS event size limits
```bash
make build # cargo build --release --no-default-features --features ocp-logging
make test # cargo nextest run ... --features ocp-logging --test-threads 1
make fmt # rustfmt
make check-clippy # clippy lints
```

## Issue Reporting
Key differences from upstream builds:
- **`--features ocp-logging`** — curated subset of Vector, not the full feature set
- **`--test-threads 1`** — single-threaded tests to avoid OOM in CI
- **`lld` linker** — configured in `.cargo/config.toml` (GNU ld OOMs during linking)
- **`CFLAGS=-g0 -O3`** — set in the build target for release optimization

- File issues at [Red Hat JIRA](https://issues.redhat.com)
- Include [must-gather](https://github.com/openshift/cluster-logging-operator/tree/master/must-gather) output for debugging
- Reference the specific OpenShift Logging release version
## `ocp-logging` Feature Scope

## Development Notes
This is the curated subset of Vector components enabled for OpenShift. Only these are compiled into the Red Hat binary.

- This project is tightly integrated with the OpenShift ecosystem
- Testing and deployment should consider the cluster-logging-operator context
- Changes should maintain compatibility with Red Hat's OpenShift Logging architecture
- Consider upstream contribution potential when making modifications
**Sources:** stdin, file_descriptor, file, journald, kubernetes_logs, prometheus, internal_metrics, demo_logs, http_server, syslog, opentelemetry

**Transforms:** route, dedupe, filter, remap, log_to_metric, lua, throttle, reduce, detect_exceptions

**Sinks (v0.47.0-rh):** aws_cloudwatch_logs, aws_s3, azure_monitor_logs, elasticsearch, file, kafka, loki, console, prometheus, gcp, splunk_hec, http, socket, opentelemetry

**Sinks (v0.54.0-rh adds):** azure_logs_ingestion

**Also enabled:** api, api-client, unix

Components not in this list are **not available** in the Red Hat binary even though they exist in the source tree.

## Fork-Specific Files

These files exist in this fork but not in upstream:

| Path | Purpose |
|-------------------|----------------------------------------------------------------------------------------------------------|
| `patch/openssl/` | Forked openssl crate — enables FIPS compliance and TLS security profile enforcement (LOG-2552, LOG-3398) |
| `patch/hyper/` | Forked hyper crate — HTTP behavior modifications |
| `thirdparty/` | Bundled binaries: `protoc`, `cargo-nextest`, `cargo-deny` (no external downloads in CI) |
| `Dockerfile` | Developer/local build image (UBI 9 + rustup) |
| `Dockerfile.art` | ART production build (UBI 9 minimal + RPM rust-toolset, `make build-offline`) |
| `Dockerfile.unit` | CI unit test runner |
| `OWNERS` | Kubernetes-style code review assignments |
| `CLAUDE.md` | Stub pointing to this file |

## Patched Crates (`[patch.crates-io]`)

**Both branches:**
- `openssl = { path = "patch/openssl" }` — Red Hat's forked openssl crate for FIPS and TLS security profiles
- `hyper = { path = "patch/hyper" }` — HTTP behavior patches
- `ntapi` — pinned revision for Windows alignment bug (inherited from upstream)

**v0.47.0-rh only** (removed in v0.54.0-rh):
- `tokio-util` — vectordotdev fork for framed-read-continue-on-error
- `nix` — vectordotdev fork for memfd gnu/musl support

When modifying TLS or HTTP behavior, check `patch/openssl/` and `patch/hyper/` — the standard crate.io versions are not used.

## Jira Conventions

Issues are tracked in [Red Hat JIRA](https://issues.redhat.com) under the **LOG** project.

- Prefix commit messages with the Jira ticket: `LOG-XXXX: description`
- Reference tickets in PR descriptions
- Use conventional commit format for PR titles: `fix(scope): description`

## Code Review (OWNERS)

```
approvers: jcantrill, alanconway, xperimental
reviewers: jcantrill, Clee2691, syedriko, vparfonov, cahartma
```

Dockerfile changes get the `midstream/Dockerfile` label automatically.

## Upstream References

For generic Vector development, use these upstream docs (they apply to this fork's Rust code):

| Topic | Upstream Document |
|------------------------------------------------------------|-------------------------------------------------------------------------------------------------------|
| Dev commands, project structure, common patterns | [AGENTS.md](https://github.com/vectordotdev/vector/blob/master/AGENTS.md) |
| Contribution workflow, PR format | [CONTRIBUTING.md](https://github.com/vectordotdev/vector/blob/master/CONTRIBUTING.md) |
| Internal engine architecture (topology, fanout, buffering) | [docs/ARCHITECTURE.md](https://github.com/vectordotdev/vector/blob/master/docs/ARCHITECTURE.md) |
| Rust coding conventions (agent-oriented) | [docs/RUST_STYLE.md](https://github.com/vectordotdev/vector/blob/master/docs/RUST_STYLE.md) |
| Code style (logging, metrics, error handling, concurrency) | [STYLE.md](https://github.com/vectordotdev/vector/blob/master/STYLE.md) |
| Developer setup, testing, benchmarking | [docs/DEVELOPING.md](https://github.com/vectordotdev/vector/blob/master/docs/DEVELOPING.md) |
| Component specification | [docs/specs/component.md](https://github.com/vectordotdev/vector/blob/master/docs/specs/component.md) |
128 changes: 128 additions & 0 deletions ARCHITECTURE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,128 @@
# Architecture — Red Hat Vector Fork

This document covers what is architecturally specific to this fork. For Vector's internal engine architecture (topology, fanout, buffering, component construction), see the upstream [docs/ARCHITECTURE.md](https://github.com/vectordotdev/vector/blob/master/docs/ARCHITECTURE.md).

## Deployment Model

Vector runs as a **DaemonSet** on every OpenShift node, managed by the [cluster-logging-operator](https://github.com/openshift/cluster-logging-operator). It is not deployed standalone.

- Configuration is injected via ConfigMap by the operator
- The operator provides a `run.sh` entrypoint script
- Vector collects container logs from `/var/log/containers/` (file source) and infrastructure logs from journald
- Pod metadata is extracted from file paths and enriched by transforms

## `ocp-logging` Feature

The fork defines a custom Cargo feature `ocp-logging` in `Cargo.toml` that compiles only the Vector components needed for OpenShift. The Makefile sets `FEATURES=ocp-logging` by default on Linux.

### Enabled Components

**Sources:** stdin, file_descriptor, file, journald, kubernetes_logs, prometheus, internal_metrics, demo_logs, http_server, syslog, opentelemetry

**Transforms:** route, dedupe, filter, remap, log_to_metric, lua, throttle, reduce, detect_exceptions

**Sinks:**

| Sink | v0.47.0-rh | v0.54.0-rh |
|----------------------|:----------:|:----------:|
| aws_cloudwatch_logs | yes | yes |
| aws_s3 | yes | yes |
| azure_logs_ingestion | — | yes |
| azure_monitor_logs | yes | yes |
| elasticsearch | yes | yes |
| file | yes | yes |
| kafka | yes | yes |
| loki | yes | yes |
| console | yes | yes |
| prometheus | yes | yes |
| gcp | yes | yes |
| splunk_hec | yes | yes |
| http | yes | yes |
| socket | yes | yes |
| opentelemetry | yes | yes |

Components not in `ocp-logging` exist in the source tree but are not compiled into the Red Hat binary.

## Patched Dependencies

The fork carries local patches for two crates via `[patch.crates-io]` in `Cargo.toml`:

### `patch/openssl/`

Forked from the `openssl` crate. Two purposes:

1. **FIPS compliance** (LOG-2552): Vector uses OpenSSL instead of upstream's Ring for all TLS operations, enabling FIPS-validated cryptography.
2. **TLS security profiles** (LOG-3398): Enforces OpenShift's `TLSSecurityProfile` settings (cipher suites, TLS versions) at the Vector level, so cluster administrators control the security posture globally.

### `patch/hyper/`

Forked from the `hyper` crate. Contains HTTP behavior modifications for reliability in OpenShift environments.

### Other patches (v0.47.0-rh only)

- `tokio-util` — vectordotdev fork for framed-read-continue-on-error behavior
- `nix` — vectordotdev fork for memfd gnu/musl support

Both were dropped in v0.54.0-rh (fixed upstream).

## Build System

### Dockerfiles

| File | Base | Rust | Purpose |
|-------------------|--------------------|--------------------|-------------------------------------------------------------------------------------------------------------------------------------|
| `Dockerfile` | `ubi9/ubi` | rustup | Local/developer builds |
| `Dockerfile.art` | `ubi9/ubi-minimal` | RPM `rust-toolset` | ART production image (shipped to customers). Uses `make build-offline` for air-gapped builds |
| `Dockerfile.unit` | `ubi9/ubi` | rustup | CI unit test runner. Selective COPY for build cache efficiency |

### Build constraints

- **`lld` linker**: Configured in `.cargo/config.toml`. GNU ld runs out of memory during linking.
- **`protoc` bundled**: Pre-built binaries in `thirdparty/protoc/` (architecture-aware). No network download during build.
- **`cargo-nextest` and `cargo-deny` bundled**: In `thirdparty/` for CI reproducibility.
- **Single-threaded tests**: `--test-threads 1` in the Makefile `test` target to limit memory usage.

### CI/CD

No in-repo CI pipelines. Builds happen in Red Hat's internal ART/OSBS infrastructure:
- `Dockerfile.art` → production image via ART (Automated Release Tooling)

GitHub Actions on the dev branches are inherited from upstream and mostly not Red Hat-specific.

## Key Design Decisions

### OpenSSL over Ring (LOG-2552)

**Decision:** Replace Ring with OpenSSL for all cryptographic operations.
**Why:** FIPS compliance requires a validated crypto library. Ring is not FIPS-validated. OpenSSL is.
**Trade-off:** Adds the `patch/openssl/` maintenance burden and prevents trivial upstream merges of TLS-related code.

### TLS Security Profile Enforcement (LOG-3398)

**Decision:** All TLS connections respect OpenShift's `TLSSecurityProfile` (cipher suites, min/max TLS version).
**Why:** OpenShift cluster administrators expect uniform TLS policy enforcement across all components.
**Depends on:** The OpenSSL patch above — upstreams Ring-based TLS does not support security profile configuration.

### `detect_exceptions` Transform (LOG-6155)

**Decision:** A Red Hat-specific transform for multiline exception detection (Java stack traces, Python tracebacks, etc.) with configurable `message_key`.
**Why:** Upstream's multiline handling is less flexible. OpenShift workloads heavily use Java, and stack traces must be grouped into single log events.

## Differences Between Version Branches

| Aspect | v0.47.0-rh | v0.54.0-rh |
|----------------------------------|----------------|--------------------------|
| Upstream base | Vector v0.47.0 | Vector v0.54.0 |
| Rust edition | 2021 | 2024 |
| MSRV (rust-toolchain.toml) | 1.85 | 1.92 |
| `azure_logs_ingestion` sink | — | enabled |
| `tokio-util` / `nix` patches | present | removed (fixed upstream) |
| GCP Workload Identity Federation | — | LOG-9171 |
| TLS curve configuration | — | LOG-8968 |

## Upstream Sync Strategy

1. **Version-based merges:** Each branch is based on a specific upstream tag (e.g., v0.47.0). Upstream changes are incorporated by creating a new branch from the next upstream tag and replaying Red Hat patches.
2. **Patch minimization:** Features are contributed upstream when possible. Once accepted, the fork patch is simplified or removed on the next branch.
3. **Cherry-picks:** Critical fixes from upstream are cherry-picked into active branches between version merges.
4. **Conflict zones:** `Cargo.toml` (feature definitions, patch section) and TLS-related code are the primary conflict areas during merges.
Loading