From d1da36e9725c64b6dfba98bc6bd005c5a2034a3e Mon Sep 17 00:00:00 2001 From: Reto Gantenbein Date: Mon, 6 Apr 2026 16:04:00 +0200 Subject: [PATCH 1/6] openspec: Add gentoo-distfiles-proxy change spec --- .../gentoo-distfiles-proxy/.openspec.yaml | 2 + .../changes/gentoo-distfiles-proxy/design.md | 71 +++++++++++++++++++ .../gentoo-distfiles-proxy/proposal.md | 30 ++++++++ .../specs/cache-exclude/spec.md | 45 ++++++++++++ .../specs/e2e-multi-distro/spec.md | 17 +++++ .../specs/gentoo-distfiles/spec.md | 27 +++++++ .../changes/gentoo-distfiles-proxy/tasks.md | 28 ++++++++ 7 files changed, 220 insertions(+) create mode 100644 openspec/changes/gentoo-distfiles-proxy/.openspec.yaml create mode 100644 openspec/changes/gentoo-distfiles-proxy/design.md create mode 100644 openspec/changes/gentoo-distfiles-proxy/proposal.md create mode 100644 openspec/changes/gentoo-distfiles-proxy/specs/cache-exclude/spec.md create mode 100644 openspec/changes/gentoo-distfiles-proxy/specs/e2e-multi-distro/spec.md create mode 100644 openspec/changes/gentoo-distfiles-proxy/specs/gentoo-distfiles/spec.md create mode 100644 openspec/changes/gentoo-distfiles-proxy/tasks.md diff --git a/openspec/changes/gentoo-distfiles-proxy/.openspec.yaml b/openspec/changes/gentoo-distfiles-proxy/.openspec.yaml new file mode 100644 index 0000000..9b8557a --- /dev/null +++ b/openspec/changes/gentoo-distfiles-proxy/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-04-06 diff --git a/openspec/changes/gentoo-distfiles-proxy/design.md b/openspec/changes/gentoo-distfiles-proxy/design.md new file mode 100644 index 0000000..09e421a --- /dev/null +++ b/openspec/changes/gentoo-distfiles-proxy/design.md @@ -0,0 +1,71 @@ +## Context + +pkgproxy routes requests by stripping the first URL path segment as the repository name, then proxying the remainder to configured upstream mirrors. Cache candidacy is currently decided solely by file suffix (`IsCacheCandidate` in `cache.go`). Gentoo distfiles are content-addressed, permanent blobs with heterogeneous file extensions — the suffix model alone cannot represent "cache everything except a few metadata files". + +## Goals / Non-Goals + +**Goals:** +- Cache all Gentoo distfiles by default with a minimal exclude list for mirror-specific metadata. +- Introduce an `exclude` field that works independently of `"*"`, so operators can also exclude oversized individual files from any repo (e.g. `verylarge.rpm`). +- No changes to the proxy routing or transport layers — Gentoo fits the existing first-segment routing convention. + +**Non-Goals:** +- Computing or validating the BLAKE2B path prefix — pkgproxy is a transparent proxy; path correctness is portage's responsibility. +- Caching `layout.conf` — excluded by default in the Gentoo config entry; no special-case code needed. +- Supporting `mirror://gentoo/` pseudo-URI scheme in ebuilds — handled transparently when portage resolves it to a real URL. + +## Decisions + +### 1. `"*"` wildcard in `suffixes` means "cache all" + +**Decision:** A literal `"*"` entry in the `suffixes` list makes every proxied file a cache candidate, subject to `exclude` filtering. + +**Alternatives considered:** +- `cache_all: true` boolean flag — adds a new top-level field and duplicates semantics already expressible via `suffixes`. +- Empty `suffixes` list means cache all — inverts current behavior (empty = cache nothing) and is surprising. +- `suffixes: ["*"]` is explicit, additive, and requires no validator changes. + +**Edge case:** If `suffixes` contains both `"*"` and explicit entries (e.g. `["*", ".rpm"]`), the explicit entries are redundant. The config is accepted but `validateConfig` logs a warning naming the repository and the redundant suffixes. `IsCacheCandidate` treats this identically to `["*"]` alone. + +### 2. `exclude` matches both exact filenames and suffixes + +**Decision:** Each entry in `exclude` is tested against the filename as an exact match first, then as a suffix. This covers: +- Exact files: `layout.conf`, `timestamp.mirmon`, `timestamp.dev-local` +- Suffix-based: `.sig`, `.asc` if an operator wanted to exclude signatures + +**Alternatives considered:** +- Separate `exclude_names` and `exclude_suffixes` fields — more explicit but adds config verbosity for a simple feature. +- Glob/regex patterns — more powerful but over-engineered for current needs; can be added later. + +### 3. `exclude` is valid without `"*"` in suffixes + +**Decision:** The `exclude` field is always applied, regardless of whether `"*"` is present. When no `"*"` is present, it acts as an override on top of suffix matching — useful for excluding a specific large file from an otherwise suffix-matched repo. + +**Implementation:** `IsCacheCandidate` runs exclude check before suffix check. If any exclude entry matches, return false immediately. + +### 4. Gentoo config uses init7 + Adfinis as primary Swiss mirrors + +**Decision:** `mirror.init7.net` first, `pkg.adfinis-on-exoscale.ch` second, `distfiles.gentoo.org` as authoritative fallback. + +### 5. E2e test bootstraps portage snapshot and uses emerge --fetchonly + +**Decision:** Use `gentoo/stage3:latest`. The test script downloads `portage-latest.tar.xz` directly from `distfiles.gentoo.org` (bypassing the proxy — bootstrap only), unpacks it into `/var/db/repos/gentoo`, sets `GENTOO_MIRRORS` to pkgproxy, then runs `emerge --fetchonly app-text/tree`. This exercises the real portage fetch path including BLAKE2B path resolution. + +**Alternatives considered:** +- Raw `wget` of a known distfile URL — simpler and faster, but doesn't validate that portage's mirror resolution works end-to-end through pkgproxy. + +The test verifies: +1. `emerge --fetchonly app-text/tree` exits successfully with `GENTOO_MIRRORS` pointing at pkgproxy. +2. The tree source archive is cached on disk under `gentoo/distfiles/`. +3. `wget` of `distfiles/layout.conf` through the proxy succeeds but the file is NOT written to cache. + +## Risks / Trade-offs + +- **`"*"` caches everything including unexpected content** → Mitigated by the `exclude` list; operators can tune it. +- **Gentoo distfiles are large** → Cache disk usage is unbounded; this is an existing property of pkgproxy (no eviction). No change needed. +- **`portage-latest.tar.xz` snapshot download adds ~300 MB to each e2e test run** → Acceptable; Gentoo e2e tests are run manually on request, not in automated CI. +- **Mirror availability** → `distfiles.gentoo.org` as authoritative fallback ensures correctness. + +## Open Questions + +None — design is fully resolved by this document. diff --git a/openspec/changes/gentoo-distfiles-proxy/proposal.md b/openspec/changes/gentoo-distfiles-proxy/proposal.md new file mode 100644 index 0000000..768b86a --- /dev/null +++ b/openspec/changes/gentoo-distfiles-proxy/proposal.md @@ -0,0 +1,30 @@ +## Why + +pkgproxy supports caching for RPM, DEB, and Arch-based distros but not Gentoo. Gentoo users who build many packages fetch large source tarballs (distfiles) repeatedly across machines; a local caching proxy reduces bandwidth and improves build times. + +## What Changes + +- Add `exclude` field to the `Repository` config type: a list of filenames or suffixes that are **never** cached, even when `suffixes` contains `"*"`. +- Add `"*"` wildcard support to the existing `suffixes` field: when present, all proxied files are cache candidates except those matching `exclude` entries. +- Add a `gentoo` repository entry to `configs/pkgproxy.yaml` using Swiss mirrors (init7, Adfinis/Exoscale) with `suffixes: ["*"]` and `exclude` covering mirror-specific metadata files. +- Add a Gentoo e2e test (`TestGentoo`) that fetches a distfile via the proxy from a `gentoo/stage3` container and asserts it is cached. + +## Capabilities + +### New Capabilities + +- `gentoo-distfiles`: Proxy and cache Gentoo distfiles from configurable upstream mirrors, honoring the BLAKE2B hash-based directory layout (`distfiles//`). +- `cache-exclude`: Per-repository `exclude` list that prevents specific filenames or suffixes from being cached, complementing the existing `suffixes` include list and enabling the `"*"` wildcard use case. + +### Modified Capabilities + +- `e2e-multi-distro`: Gentoo is added as a supported distro with a corresponding e2e test. + +## Impact + +- `pkg/pkgproxy/repository.go`: Add `Exclude []string` field to `Repository` struct; update `validateConfig` (no required validation, field is optional). +- `pkg/cache/cache.go`: Update `CacheConfig` to carry the exclude list; update `IsCacheCandidate` to handle `"*"` wildcard and exclude matching. +- `configs/pkgproxy.yaml`: Add `gentoo` repository entry. +- `test/e2e/e2e_test.go`: Add `TestGentoo`. +- `README.md` and landing page: Add Gentoo `make.conf` snippet. +- `CHANGELOG.md`: Document new features. diff --git a/openspec/changes/gentoo-distfiles-proxy/specs/cache-exclude/spec.md b/openspec/changes/gentoo-distfiles-proxy/specs/cache-exclude/spec.md new file mode 100644 index 0000000..d791921 --- /dev/null +++ b/openspec/changes/gentoo-distfiles-proxy/specs/cache-exclude/spec.md @@ -0,0 +1,45 @@ +## ADDED Requirements + +### Requirement: Wildcard suffix caches all files +When the `suffixes` list for a repository contains `"*"`, the cache SHALL treat every proxied file as a cache candidate, subject to the `exclude` list. + +#### Scenario: File with uncommon extension is cached under wildcard repo +- **WHEN** a request is made for a file with an extension not in any explicit suffix list (e.g. `.crate`) under a repo with `suffixes: ["*"]` +- **THEN** `IsCacheCandidate` returns true + +#### Scenario: Wildcard does not affect repos without it +- **WHEN** a request is made for a file under a repo whose `suffixes` list does not contain `"*"` +- **THEN** `IsCacheCandidate` applies the existing suffix-match logic unchanged + +### Requirement: Exclude list prevents specific files from being cached +A repository MAY define an `exclude` list. Each entry is matched against the request filename as an exact name first, then as a suffix. If any entry matches, the file SHALL NOT be cached regardless of `suffixes`. + +#### Scenario: Exact filename match prevents caching +- **WHEN** a request is made for a file whose name exactly matches an `exclude` entry (e.g. `layout.conf`) +- **THEN** `IsCacheCandidate` returns false + +#### Scenario: Suffix match prevents caching +- **WHEN** a request is made for a file whose name ends with an `exclude` entry (e.g. `.sig`) +- **THEN** `IsCacheCandidate` returns false + +#### Scenario: Non-matching file is not excluded +- **WHEN** a request is made for a file that does not match any `exclude` entry +- **THEN** the `exclude` list has no effect on the cache candidacy decision + +#### Scenario: Exclude applies without wildcard suffix +- **WHEN** a repository has explicit suffixes (no `"*"`) and an `exclude` list, and a request is made for a file that matches both a suffix and an exclude entry +- **THEN** `IsCacheCandidate` returns false (exclude takes precedence) + +### Requirement: Explicit suffixes alongside wildcard are redundant but valid +When the `suffixes` list contains both `"*"` and explicit suffix entries, the configuration SHALL be accepted. pkgproxy SHALL log a warning identifying the repository and the redundant entries. Cache behavior is identical to having only `"*"`. + +#### Scenario: Mixed wildcard and explicit suffixes triggers a warning +- **WHEN** pkgproxy loads a repository config whose `suffixes` list contains `"*"` and at least one other entry +- **THEN** the repository is accepted without error, a warning is logged naming the repository and the redundant suffixes, and `IsCacheCandidate` behaves as if only `"*"` were present + +### Requirement: Exclude field is optional +The `exclude` field in a repository config SHALL be optional. Repositories without it SHALL behave identically to the current behavior. + +#### Scenario: Repository without exclude field +- **WHEN** pkgproxy loads a repository config with no `exclude` key +- **THEN** the repository is accepted without error and cache behavior is unchanged diff --git a/openspec/changes/gentoo-distfiles-proxy/specs/e2e-multi-distro/spec.md b/openspec/changes/gentoo-distfiles-proxy/specs/e2e-multi-distro/spec.md new file mode 100644 index 0000000..8eec505 --- /dev/null +++ b/openspec/changes/gentoo-distfiles-proxy/specs/e2e-multi-distro/spec.md @@ -0,0 +1,17 @@ +## ADDED Requirements + +### Requirement: Gentoo e2e test +The test suite SHALL include a Gentoo test function `TestGentoo` using a `docker.io/gentoo/stage3:latest` container. The test script SHALL: +1. Download the latest portage ebuild snapshot directly from `https://distfiles.gentoo.org/snapshots/portage-latest.tar.xz` (bypassing the proxy — this is bootstrap, not a distfile fetch). +2. Unpack the snapshot into `/var/db/repos/gentoo` inside the container. +3. Configure `GENTOO_MIRRORS` in `/etc/portage/make.conf` to point at the pkgproxy `gentoo` repository. +4. Run `emerge --fetchonly app-text/tree` to fetch the `tree` package sources through the proxy. +5. Fetch `http:///gentoo/distfiles/layout.conf` via `wget` to exercise the negative cache path. + +#### Scenario: emerge --fetchonly proxies and caches tree distfiles +- **WHEN** the Gentoo container runs `emerge --fetchonly app-text/tree` with `GENTOO_MIRRORS` pointing at pkgproxy +- **THEN** the command exits successfully and the tree source archive exists in the pkgproxy cache under the `gentoo/` subdirectory + +#### Scenario: layout.conf is proxied but not cached +- **WHEN** the Gentoo container fetches `http:///gentoo/distfiles/layout.conf` via `wget` +- **THEN** the request returns HTTP 200 and `layout.conf` does NOT exist in the pkgproxy cache under the `gentoo/` subdirectory diff --git a/openspec/changes/gentoo-distfiles-proxy/specs/gentoo-distfiles/spec.md b/openspec/changes/gentoo-distfiles-proxy/specs/gentoo-distfiles/spec.md new file mode 100644 index 0000000..af296e3 --- /dev/null +++ b/openspec/changes/gentoo-distfiles-proxy/specs/gentoo-distfiles/spec.md @@ -0,0 +1,27 @@ +## ADDED Requirements + +### Requirement: Gentoo distfiles repository config entry +The `configs/pkgproxy.yaml` SHALL include a `gentoo` repository configured with `suffixes: ["*"]`, an `exclude` list covering mirror-specific metadata files (`layout.conf`, `timestamp.mirmon`, `timestamp.dev-local`), and at least two Swiss HTTPS mirrors plus `distfiles.gentoo.org` as authoritative fallback. + +#### Scenario: Gentoo distfiles repository is configured +- **WHEN** pkgproxy loads its configuration +- **THEN** the `gentoo` repository is available with at least one upstream mirror + +#### Scenario: layout.conf is not cached +- **WHEN** a client fetches `/gentoo/distfiles/layout.conf` +- **THEN** pkgproxy proxies the file upstream but does not write it to the local cache + +#### Scenario: Distfile fetched via emerge --fetchonly is proxied and cached +- **WHEN** portage runs `emerge --fetchonly app-text/tree` with `GENTOO_MIRRORS` pointing at pkgproxy +- **THEN** pkgproxy proxies the distfile from the upstream mirror and saves it to the local cache under `gentoo/distfiles//` + +#### Scenario: Cached distfile is served from disk on subsequent request +- **WHEN** portage fetches the same distfile a second time +- **THEN** pkgproxy serves the file from the local cache without contacting the upstream mirror + +### Requirement: make.conf snippet in README and landing page +The README.md and HTTP landing page SHALL include a Gentoo `make.conf` snippet showing how to configure `GENTOO_MIRRORS` to point at the proxy. + +#### Scenario: Gentoo configuration snippet is present +- **WHEN** a user views the README or the pkgproxy landing page +- **THEN** a `make.conf` snippet with `GENTOO_MIRRORS="http:///gentoo"` is visible diff --git a/openspec/changes/gentoo-distfiles-proxy/tasks.md b/openspec/changes/gentoo-distfiles-proxy/tasks.md new file mode 100644 index 0000000..6bc8db5 --- /dev/null +++ b/openspec/changes/gentoo-distfiles-proxy/tasks.md @@ -0,0 +1,28 @@ +## 1. Cache exclude feature + +- [ ] 1.1 Add `Exclude []string` field to `Repository` struct in `pkg/pkgproxy/repository.go`; in `validateConfig`, if a repository's `suffixes` list contains `"*"` alongside other entries, log a `slog.Warn` naming the repository and the redundant suffixes +- [ ] 1.2 Add `Exclude []string` field to `CacheConfig` in `pkg/cache/cache.go` +- [ ] 1.3 Pass `Exclude` from `Repository` into `CacheConfig` when constructing upstreams in `proxy.go` +- [ ] 1.4 Update `IsCacheCandidate` in `cache.go` to: run exclude check first (exact name + suffix), then handle `"*"` wildcard, then existing suffix logic +- [ ] 1.5 Add unit tests for `IsCacheCandidate` covering: wildcard match, exclude exact name, exclude suffix, exclude overrides wildcard, exclude overrides explicit suffix, no exclude field +- [ ] 1.6 Add unit test for `validateConfig` covering: wildcard with redundant explicit suffixes emits a warning and returns no error + +## 2. Gentoo repository config + +- [ ] 2.1 Add `gentoo` entry to `configs/pkgproxy.yaml` with `suffixes: ["*"]`, `exclude: [layout.conf, timestamp.mirmon, timestamp.dev-local]`, and mirrors: `mirror.init7.net`, `pkg.adfinis-on-exoscale.ch`, `distfiles.gentoo.org` + +## 3. E2e test + +- [ ] 3.1 Add `assertNotCached` helper to `test/e2e/e2e_test.go` that asserts no file matching a given name exists anywhere under a cache subdirectory +- [ ] 3.2 Write `test/e2e/test-gentoo.sh` shell script that: downloads `portage-latest.tar.xz` directly from `distfiles.gentoo.org`, unpacks it to `/var/db/repos/gentoo`, sets `GENTOO_MIRRORS` in `make.conf` to point at pkgproxy, runs `emerge --fetchonly app-text/tree`, then fetches `distfiles/layout.conf` via `wget` through the proxy +- [ ] 3.3 Add `TestGentoo` to `test/e2e/e2e_test.go` using `docker.io/gentoo/stage3:latest`, mounting the script, asserting tree source archive is cached under `gentoo/distfiles/`, and asserting `layout.conf` is NOT cached using `assertNotCached` + +## 3b. Makefile + +- [ ] 3b.1 Add `gentoo → TestGentoo` mapping to the `distroToTest` macro in `Makefile` so `make e2e DISTRO=gentoo` works; add `gentoo` to the error message's list of valid values + +## 4. Documentation + +- [ ] 4.1 Add Gentoo `make.conf` snippet to `README.md` +- [ ] 4.2 Add Gentoo `make.conf` snippet to the HTTP landing page (`pkg/pkgproxy/landing.go` or template) +- [ ] 4.3 Update `CHANGELOG.md` `[Unreleased]` section with new features From 6d5b8e88c5b70ec28848023eef8924b2607b7842 Mon Sep 17 00:00:00 2001 From: Reto Gantenbein Date: Mon, 6 Apr 2026 16:11:03 +0200 Subject: [PATCH 2/6] Add Gentoo distfiles proxy support with cache exclude feature Introduce per-repository `exclude` config field and `"*"` wildcard support in `suffixes` to cache all files except specified exclusions. Add Gentoo repository config, e2e test, and documentation snippets. Co-Authored-By: Claude Sonnet 4.6 --- CHANGELOG.md | 6 ++ Makefile | 3 +- README.md | 9 ++- configs/pkgproxy.yaml | 11 +++ .../changes/gentoo-distfiles-proxy/tasks.md | 28 +++---- pkg/cache/cache.go | 26 ++++-- pkg/cache/cache_test.go | 81 +++++++++++++++++++ pkg/pkgproxy/landing.go | 3 + pkg/pkgproxy/proxy.go | 1 + pkg/pkgproxy/repository.go | 16 ++++ pkg/pkgproxy/repository_test.go | 56 +++++++++++++ test/e2e/e2e_test.go | 28 +++++++ test/e2e/test-gentoo.sh | 33 ++++++++ 13 files changed, 280 insertions(+), 21 deletions(-) create mode 100644 pkg/pkgproxy/repository_test.go create mode 100755 test/e2e/test-gentoo.sh diff --git a/CHANGELOG.md b/CHANGELOG.md index a722df7..8453953 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,12 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ## [Unreleased](https://github.com/ganto/pkgproxy/commits/HEAD/) +### Added + +- Per-repository `exclude` config field to prevent specific filenames or suffixes from being cached +- Support caching Gentoo distfiles with `suffixes: ["*"]` wildcard and `exclude` list +- Gentoo e2e test using `emerge --fetchonly` in a `gentoo/stage3` container + ## [v0.1.2](https://github.com/ganto/pkgproxy/releases/tag/v0.1.2) - 2026-03-28 ### Fixed diff --git a/Makefile b/Makefile index a1c3bab..c8d4cda 100644 --- a/Makefile +++ b/Makefile @@ -107,7 +107,8 @@ $(if $(filter rockylinux,$(1)),TestRockyLinux,\ $(if $(filter debian,$(1)),TestDebian,\ $(if $(filter ubuntu,$(1)),TestUbuntu,\ $(if $(filter archlinux,$(1)),TestArch,\ -$(error Unknown DISTRO: $(1). Use one of: fedora centos-stream almalinux rockylinux debian ubuntu archlinux))))))))) +$(if $(filter gentoo,$(1)),TestGentoo,\ +$(error Unknown DISTRO: $(1). Use one of: fedora centos-stream almalinux rockylinux debian ubuntu archlinux gentoo)))))))))) endef .PHONY: e2e diff --git a/README.md b/README.md index 58a7ff3..be09959 100644 --- a/README.md +++ b/README.md @@ -135,6 +135,13 @@ Can be used for any type of RPM-based enterprise distribution. E.g. `/etc/yum.re baseurl=http://:8080/epel/$releasever/Everything/$basearch/ ``` +### Gentoo + +`/etc/portage/make.conf`: +``` +GENTOO_MIRRORS="http://:8080/gentoo" +``` + ### Fedora `/etc/yum.repos.d/fedora.repo` (adjust other repositories accordingly): @@ -193,7 +200,7 @@ Run tests for a specific distribution and release: make e2e DISTRO=fedora RELEASE=42 ``` -Supported `DISTRO` values: `fedora`, `centos-stream`, `almalinux`, `rockylinux`, `debian`, `ubuntu`, `archlinux`. +Supported `DISTRO` values: `fedora`, `centos-stream`, `almalinux`, `rockylinux`, `debian`, `ubuntu`, `archlinux`, `gentoo`. When adding support for a new Linux distribution, corresponding e2e tests should be added as well. diff --git a/configs/pkgproxy.yaml b/configs/pkgproxy.yaml index cfb8ed3..f94aeff 100644 --- a/configs/pkgproxy.yaml +++ b/configs/pkgproxy.yaml @@ -56,6 +56,17 @@ repositories: mirrors: - https://mirror.init7.net/fedora/epel/ - https://dl.fedoraproject.org/pub/epel/ + gentoo: + suffixes: + - "*" + exclude: + - layout.conf + - timestamp.mirmon + - timestamp.dev-local + mirrors: + - https://mirror.init7.net/gentoo/ + - https://pkg.adfinis-on-exoscale.ch/gentoo/ + - https://distfiles.gentoo.org/ fedora: suffixes: - .drpm diff --git a/openspec/changes/gentoo-distfiles-proxy/tasks.md b/openspec/changes/gentoo-distfiles-proxy/tasks.md index 6bc8db5..b79e6b7 100644 --- a/openspec/changes/gentoo-distfiles-proxy/tasks.md +++ b/openspec/changes/gentoo-distfiles-proxy/tasks.md @@ -1,28 +1,28 @@ ## 1. Cache exclude feature -- [ ] 1.1 Add `Exclude []string` field to `Repository` struct in `pkg/pkgproxy/repository.go`; in `validateConfig`, if a repository's `suffixes` list contains `"*"` alongside other entries, log a `slog.Warn` naming the repository and the redundant suffixes -- [ ] 1.2 Add `Exclude []string` field to `CacheConfig` in `pkg/cache/cache.go` -- [ ] 1.3 Pass `Exclude` from `Repository` into `CacheConfig` when constructing upstreams in `proxy.go` -- [ ] 1.4 Update `IsCacheCandidate` in `cache.go` to: run exclude check first (exact name + suffix), then handle `"*"` wildcard, then existing suffix logic -- [ ] 1.5 Add unit tests for `IsCacheCandidate` covering: wildcard match, exclude exact name, exclude suffix, exclude overrides wildcard, exclude overrides explicit suffix, no exclude field -- [ ] 1.6 Add unit test for `validateConfig` covering: wildcard with redundant explicit suffixes emits a warning and returns no error +- [x] 1.1 Add `Exclude []string` field to `Repository` struct in `pkg/pkgproxy/repository.go`; in `validateConfig`, if a repository's `suffixes` list contains `"*"` alongside other entries, log a `slog.Warn` naming the repository and the redundant suffixes +- [x] 1.2 Add `Exclude []string` field to `CacheConfig` in `pkg/cache/cache.go` +- [x] 1.3 Pass `Exclude` from `Repository` into `CacheConfig` when constructing upstreams in `proxy.go` +- [x] 1.4 Update `IsCacheCandidate` in `cache.go` to: run exclude check first (exact name + suffix), then handle `"*"` wildcard, then existing suffix logic +- [x] 1.5 Add unit tests for `IsCacheCandidate` covering: wildcard match, exclude exact name, exclude suffix, exclude overrides wildcard, exclude overrides explicit suffix, no exclude field +- [x] 1.6 Add unit test for `validateConfig` covering: wildcard with redundant explicit suffixes emits a warning and returns no error ## 2. Gentoo repository config -- [ ] 2.1 Add `gentoo` entry to `configs/pkgproxy.yaml` with `suffixes: ["*"]`, `exclude: [layout.conf, timestamp.mirmon, timestamp.dev-local]`, and mirrors: `mirror.init7.net`, `pkg.adfinis-on-exoscale.ch`, `distfiles.gentoo.org` +- [x] 2.1 Add `gentoo` entry to `configs/pkgproxy.yaml` with `suffixes: ["*"]`, `exclude: [layout.conf, timestamp.mirmon, timestamp.dev-local]`, and mirrors: `mirror.init7.net`, `pkg.adfinis-on-exoscale.ch`, `distfiles.gentoo.org` ## 3. E2e test -- [ ] 3.1 Add `assertNotCached` helper to `test/e2e/e2e_test.go` that asserts no file matching a given name exists anywhere under a cache subdirectory -- [ ] 3.2 Write `test/e2e/test-gentoo.sh` shell script that: downloads `portage-latest.tar.xz` directly from `distfiles.gentoo.org`, unpacks it to `/var/db/repos/gentoo`, sets `GENTOO_MIRRORS` in `make.conf` to point at pkgproxy, runs `emerge --fetchonly app-text/tree`, then fetches `distfiles/layout.conf` via `wget` through the proxy -- [ ] 3.3 Add `TestGentoo` to `test/e2e/e2e_test.go` using `docker.io/gentoo/stage3:latest`, mounting the script, asserting tree source archive is cached under `gentoo/distfiles/`, and asserting `layout.conf` is NOT cached using `assertNotCached` +- [x] 3.1 Add `assertNotCached` helper to `test/e2e/e2e_test.go` that asserts no file matching a given name exists anywhere under a cache subdirectory +- [x] 3.2 Write `test/e2e/test-gentoo.sh` shell script that: downloads `portage-latest.tar.xz` directly from `distfiles.gentoo.org`, unpacks it to `/var/db/repos/gentoo`, sets `GENTOO_MIRRORS` in `make.conf` to point at pkgproxy, runs `emerge --fetchonly app-text/tree`, then fetches `distfiles/layout.conf` via `wget` through the proxy +- [x] 3.3 Add `TestGentoo` to `test/e2e/e2e_test.go` using `docker.io/gentoo/stage3:latest`, mounting the script, asserting tree source archive is cached under `gentoo/distfiles/`, and asserting `layout.conf` is NOT cached using `assertNotCached` ## 3b. Makefile -- [ ] 3b.1 Add `gentoo → TestGentoo` mapping to the `distroToTest` macro in `Makefile` so `make e2e DISTRO=gentoo` works; add `gentoo` to the error message's list of valid values +- [x] 3b.1 Add `gentoo → TestGentoo` mapping to the `distroToTest` macro in `Makefile` so `make e2e DISTRO=gentoo` works; add `gentoo` to the error message's list of valid values ## 4. Documentation -- [ ] 4.1 Add Gentoo `make.conf` snippet to `README.md` -- [ ] 4.2 Add Gentoo `make.conf` snippet to the HTTP landing page (`pkg/pkgproxy/landing.go` or template) -- [ ] 4.3 Update `CHANGELOG.md` `[Unreleased]` section with new features +- [x] 4.1 Add Gentoo `make.conf` snippet to `README.md` +- [x] 4.2 Add Gentoo `make.conf` snippet to the HTTP landing page (`pkg/pkgproxy/landing.go` or template) +- [x] 4.3 Update `CHANGELOG.md` `[Unreleased]` section with new features diff --git a/pkg/cache/cache.go b/pkg/cache/cache.go index 5fad759..c7ffb6c 100644 --- a/pkg/cache/cache.go +++ b/pkg/cache/cache.go @@ -56,6 +56,9 @@ type CacheConfig struct { // List of file suffixes that will be cached FileSuffixes []string + + // List of filenames or suffixes that are never cached + Exclude []string } func New(cfg *CacheConfig) FileCache { @@ -104,17 +107,30 @@ func (c *cache) GetFileSuffixes() []string { // Verifies if the given file URI is candidate to be cached func (c *cache) IsCacheCandidate(uri string) bool { - ca := false - name := utils.FilenameFromURI(uri) + + // Exclude check first: exact name match or suffix match. + for _, entry := range c.config.Exclude { + if name == entry || strings.HasSuffix(name, entry) { + return false + } + } + + // Wildcard: "*" in suffixes means cache everything (that wasn't excluded). + for _, suffix := range c.GetFileSuffixes() { + if suffix == "*" { + return true + } + } + + // Existing suffix-match logic. for _, suffix := range c.GetFileSuffixes() { if strings.HasSuffix(name, suffix) { - ca = true - break + return true } } - return ca + return false } // Verifies if the file is already cached diff --git a/pkg/cache/cache_test.go b/pkg/cache/cache_test.go index 54898fa..20e18ee 100644 --- a/pkg/cache/cache_test.go +++ b/pkg/cache/cache_test.go @@ -187,6 +187,87 @@ func TestCommitTempFile(t *testing.T) { }) } +func TestIsCacheCandidate(t *testing.T) { + tests := []struct { + name string + suffixes []string + exclude []string + uri string + want bool + }{ + { + name: "wildcard matches any file", + suffixes: []string{"*"}, + uri: "/repo/distfiles/ab/somefile.tar.xz", + want: true, + }, + { + name: "exclude exact name blocks caching", + suffixes: []string{"*"}, + exclude: []string{"layout.conf"}, + uri: "/repo/distfiles/layout.conf", + want: false, + }, + { + name: "exclude suffix blocks caching", + suffixes: []string{"*"}, + exclude: []string{".sig"}, + uri: "/repo/distfiles/ab/somefile.tar.xz.sig", + want: false, + }, + { + name: "exclude overrides wildcard", + suffixes: []string{"*"}, + exclude: []string{"timestamp.mirmon"}, + uri: "/repo/distfiles/timestamp.mirmon", + want: false, + }, + { + name: "exclude overrides explicit suffix", + suffixes: []string{".rpm"}, + exclude: []string{"verylarge.rpm"}, + uri: "/repo/verylarge.rpm", + want: false, + }, + { + name: "no exclude field behaves normally", + suffixes: []string{".rpm"}, + uri: "/repo/package.rpm", + want: true, + }, + { + name: "no exclude field rejects non-matching suffix", + suffixes: []string{".rpm"}, + uri: "/repo/readme.txt", + want: false, + }, + { + name: "wildcard without exclude matches everything", + suffixes: []string{"*"}, + uri: "/repo/anything", + want: true, + }, + { + name: "non-matching exclude does not affect caching", + suffixes: []string{"*"}, + exclude: []string{"layout.conf"}, + uri: "/repo/distfiles/ab/somefile.crate", + want: true, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + c := New(&CacheConfig{ + BasePath: "/cache", + FileSuffixes: tt.suffixes, + Exclude: tt.exclude, + }) + assert.Equal(t, tt.want, c.IsCacheCandidate(tt.uri)) + }) + } +} + func TestSaveToDiskStillWorks(t *testing.T) { baseDir := t.TempDir() c := New(&CacheConfig{BasePath: baseDir}) diff --git a/pkg/pkgproxy/landing.go b/pkg/pkgproxy/landing.go index 8fac227..e4724d8 100644 --- a/pkg/pkgproxy/landing.go +++ b/pkg/pkgproxy/landing.go @@ -82,6 +82,9 @@ var snippetFuncs = map[string]func(string) string{ "# metalink=https://mirrors.fedoraproject.org/metalink?repo=epel-$releasever&arch=$basearch\n" + "baseurl=http://" + addr + "/epel/$releasever/Everything/$basearch/" }, + "gentoo": func(addr string) string { + return "GENTOO_MIRRORS=\"http://" + addr + "/gentoo\"" + }, "fedora": func(addr string) string { return "[fedora]\n" + "# metalink=https://mirrors.fedoraproject.org/metalink?repo=fedora-$releasever&arch=$basearch\n" + diff --git a/pkg/pkgproxy/proxy.go b/pkg/pkgproxy/proxy.go index abd9f98..83814fa 100644 --- a/pkg/pkgproxy/proxy.go +++ b/pkg/pkgproxy/proxy.go @@ -134,6 +134,7 @@ func New(config *PkgProxyConfig) PkgProxy { cache: cache.New(&cache.CacheConfig{ BasePath: config.CacheBasePath, FileSuffixes: config.RepositoryConfig.Repositories[repo].CacheSuffixes, + Exclude: config.RepositoryConfig.Repositories[repo].Exclude, }), mirrors: mirrors, retries: retries, diff --git a/pkg/pkgproxy/repository.go b/pkg/pkgproxy/repository.go index 74996ce..a292916 100644 --- a/pkg/pkgproxy/repository.go +++ b/pkg/pkgproxy/repository.go @@ -5,6 +5,7 @@ package pkgproxy import ( "errors" "fmt" + "log/slog" "os" "path/filepath" "regexp" @@ -22,6 +23,7 @@ type RepoConfig struct { type Repository struct { CacheSuffixes []string `yaml:"suffixes"` + Exclude []string `yaml:"exclude,omitempty"` Mirrors []string `yaml:"mirrors"` Retries int `yaml:"retries,omitempty"` } @@ -62,6 +64,20 @@ func validateConfig(config *RepoConfig) error { if repoConfig.Mirrors == nil { return fmt.Errorf("missing required key for repository '%s': mirrors", handle) } + // Warn if suffixes contains "*" alongside other entries (redundant). + hasWildcard := false + var redundant []string + for _, s := range repoConfig.CacheSuffixes { + if s == "*" { + hasWildcard = true + } else { + redundant = append(redundant, s) + } + } + if hasWildcard && len(redundant) > 0 { + slog.Warn("repository has wildcard suffix '*' with redundant explicit suffixes", + "repository", handle, "redundant_suffixes", redundant) + } } return nil } diff --git a/pkg/pkgproxy/repository_test.go b/pkg/pkgproxy/repository_test.go new file mode 100644 index 0000000..72b7f4d --- /dev/null +++ b/pkg/pkgproxy/repository_test.go @@ -0,0 +1,56 @@ +// Copyright 2026 Reto Gantenbein +// SPDX-License-Identifier: Apache-2.0 +package pkgproxy + +import ( + "bytes" + "log/slog" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +func TestValidateConfigWildcardWithRedundantSuffixes(t *testing.T) { + var buf bytes.Buffer + handler := slog.NewTextHandler(&buf, &slog.HandlerOptions{Level: slog.LevelWarn}) + logger := slog.New(handler) + slog.SetDefault(logger) + + config := &RepoConfig{ + Repositories: map[string]Repository{ + "testrepo": { + CacheSuffixes: []string{"*", ".rpm", ".drpm"}, + Mirrors: []string{"https://example.com/"}, + }, + }, + } + + err := validateConfig(config) + require.NoError(t, err) + + logOutput := buf.String() + assert.Contains(t, logOutput, "testrepo") + assert.Contains(t, logOutput, "redundant") +} + +func TestValidateConfigWildcardAloneNoWarning(t *testing.T) { + var buf bytes.Buffer + handler := slog.NewTextHandler(&buf, &slog.HandlerOptions{Level: slog.LevelWarn}) + logger := slog.New(handler) + slog.SetDefault(logger) + + config := &RepoConfig{ + Repositories: map[string]Repository{ + "testrepo": { + CacheSuffixes: []string{"*"}, + Mirrors: []string{"https://example.com/"}, + }, + }, + } + + err := validateConfig(config) + require.NoError(t, err) + + assert.Empty(t, buf.String()) +} diff --git a/test/e2e/e2e_test.go b/test/e2e/e2e_test.go index aa26ae6..fe79979 100644 --- a/test/e2e/e2e_test.go +++ b/test/e2e/e2e_test.go @@ -150,6 +150,21 @@ func runContainer(t *testing.T, image string, mounts []string, cmdArgs []string) require.NoError(t, err, "container command failed") } +func assertNotCached(t *testing.T, cacheDir string, repoPrefix string, name string) { + t.Helper() + var matches []string + filepath.Walk(filepath.Join(cacheDir, repoPrefix), func(path string, info os.FileInfo, err error) error { + if err != nil { + return nil + } + if !info.IsDir() && filepath.Base(path) == name { + matches = append(matches, path) + } + return nil + }) + assert.Empty(t, matches, "expected no %s files under %s/%s, but found: %v", name, cacheDir, repoPrefix, matches) +} + func assertCachedFiles(t *testing.T, cacheDir string, repoPrefix string, suffix string) { t.Helper() var matches []string @@ -418,3 +433,16 @@ deb http://%s/ubuntu-security %s-security main restricted universe multiverse ) assertCachedFiles(t, cacheDir, "ubuntu", ".deb") } + +func TestGentoo(t *testing.T) { + proxyAddr, cacheDir := setupPkgproxy(t) + + runContainer(t, "docker.io/gentoo/stage3:latest", + []string{ + filepath.Join(scriptDir(), "test-gentoo.sh") + ":/test-gentoo.sh:ro,z", + }, + []string{"bash", "/test-gentoo.sh", proxyAddr}, + ) + assertCachedFiles(t, cacheDir, "gentoo/distfiles", "") + assertNotCached(t, cacheDir, "gentoo", "layout.conf") +} diff --git a/test/e2e/test-gentoo.sh b/test/e2e/test-gentoo.sh new file mode 100755 index 0000000..1d151d6 --- /dev/null +++ b/test/e2e/test-gentoo.sh @@ -0,0 +1,33 @@ +#!/bin/bash +# test-gentoo.sh — Fetch a package via emerge --fetchonly through pkgproxy. +# Usage: test-gentoo.sh +set -euo pipefail + +PROXY_ADDR="$1" + +echo "==> Proxy: ${PROXY_ADDR}" + +# Bootstrap: download portage snapshot directly (bypassing the proxy). +echo "==> Downloading portage snapshot..." +wget -q https://distfiles.gentoo.org/snapshots/portage-latest.tar.xz -O /tmp/portage-latest.tar.xz +echo "==> Unpacking portage snapshot..." +mkdir -p /var/db/repos/gentoo +tar xf /tmp/portage-latest.tar.xz -C /var/db/repos/gentoo --strip-components=1 +rm /tmp/portage-latest.tar.xz + +# Configure GENTOO_MIRRORS to point at pkgproxy. +echo "GENTOO_MIRRORS=\"http://${PROXY_ADDR}/gentoo\"" >> /etc/portage/make.conf + +echo "==> make.conf:" +echo "--- /etc/portage/make.conf ---" +cat /etc/portage/make.conf + +# Fetch distfiles for app-text/tree through the proxy. +echo "==> Running emerge --fetchonly app-text/tree..." +emerge --fetchonly app-text/tree + +# Exercise the negative cache path: fetch layout.conf through the proxy. +echo "==> Fetching layout.conf through proxy (should not be cached)..." +wget -q "http://${PROXY_ADDR}/gentoo/distfiles/layout.conf" -O /dev/null + +echo "==> Done" From ed1d3e90fe320d541127ac2cb2b1293033c094b1 Mon Sep 17 00:00:00 2001 From: Reto Gantenbein Date: Mon, 6 Apr 2026 16:21:46 +0200 Subject: [PATCH 3/6] github-workflow: Add Gentoo to e2e test matrix --- .github/workflows/e2e.yaml | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/.github/workflows/e2e.yaml b/.github/workflows/e2e.yaml index f1d4dfb..d402abc 100644 --- a/.github/workflows/e2e.yaml +++ b/.github/workflows/e2e.yaml @@ -28,7 +28,7 @@ jobs: e2e: if: github.event_name == 'workflow_dispatch' runs-on: ubuntu-latest - timeout-minutes: 5 + timeout-minutes: ${{ matrix.timeout || 5 }} strategy: fail-fast: false @@ -55,6 +55,10 @@ jobs: - name: Arch Linux latest test: TestArch release: latest + - name: Gentoo latest + test: TestGentoo + release: latest + timeout: 10 name: ${{ matrix.name }} @@ -71,7 +75,7 @@ jobs: env: E2E_RELEASE: ${{ matrix.release }} CONTAINER_RUNTIME: docker - run: go test -tags e2e -v -race -timeout 5m -run ${{ matrix.test }} ./test/e2e/ + run: go test -tags e2e -v -race -timeout ${{ matrix.timeout || 5 }}m -run ${{ matrix.test }} ./test/e2e/ report-status: if: always() && github.event_name == 'workflow_dispatch' From d39d9cf6f8dac967905fc9aa1fc1e16ecf95932b Mon Sep 17 00:00:00 2001 From: Reto Gantenbein Date: Mon, 6 Apr 2026 16:23:00 +0200 Subject: [PATCH 4/6] test: Fix error handling in e2e assertNotCached Propagate filepath.Walk callback errors and check the returned error with require.NoError to prevent false-passing tests on walk failures. Co-Authored-By: Claude Sonnet 4.6 --- test/e2e/e2e_test.go | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/test/e2e/e2e_test.go b/test/e2e/e2e_test.go index fe79979..d9f594e 100644 --- a/test/e2e/e2e_test.go +++ b/test/e2e/e2e_test.go @@ -153,15 +153,16 @@ func runContainer(t *testing.T, image string, mounts []string, cmdArgs []string) func assertNotCached(t *testing.T, cacheDir string, repoPrefix string, name string) { t.Helper() var matches []string - filepath.Walk(filepath.Join(cacheDir, repoPrefix), func(path string, info os.FileInfo, err error) error { + err := filepath.Walk(filepath.Join(cacheDir, repoPrefix), func(path string, info os.FileInfo, err error) error { if err != nil { - return nil + return err } if !info.IsDir() && filepath.Base(path) == name { matches = append(matches, path) } return nil }) + require.NoError(t, err, "failed to walk %s/%s", cacheDir, repoPrefix) assert.Empty(t, matches, "expected no %s files under %s/%s, but found: %v", name, cacheDir, repoPrefix, matches) } From e6ff9d6014f3d20967f4ab3685765ed97d7ee64d Mon Sep 17 00:00:00 2001 From: Reto Gantenbein Date: Mon, 6 Apr 2026 16:24:55 +0200 Subject: [PATCH 5/6] test: Restore default slog logger after validateConfig tests Save and restore the global slog default via t.Cleanup to prevent leaking a custom logger into other tests. Co-Authored-By: Claude Sonnet 4.6 --- pkg/pkgproxy/repository_test.go | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/pkg/pkgproxy/repository_test.go b/pkg/pkgproxy/repository_test.go index 72b7f4d..124d3ad 100644 --- a/pkg/pkgproxy/repository_test.go +++ b/pkg/pkgproxy/repository_test.go @@ -12,6 +12,9 @@ import ( ) func TestValidateConfigWildcardWithRedundantSuffixes(t *testing.T) { + prev := slog.Default() + t.Cleanup(func() { slog.SetDefault(prev) }) + var buf bytes.Buffer handler := slog.NewTextHandler(&buf, &slog.HandlerOptions{Level: slog.LevelWarn}) logger := slog.New(handler) @@ -35,6 +38,9 @@ func TestValidateConfigWildcardWithRedundantSuffixes(t *testing.T) { } func TestValidateConfigWildcardAloneNoWarning(t *testing.T) { + prev := slog.Default() + t.Cleanup(func() { slog.SetDefault(prev) }) + var buf bytes.Buffer handler := slog.NewTextHandler(&buf, &slog.HandlerOptions{Level: slog.LevelWarn}) logger := slog.New(handler) From c5f5fdb94e6a4f16a1b9be390168347ce8a2af38 Mon Sep 17 00:00:00 2001 From: Reto Gantenbein Date: Mon, 6 Apr 2026 16:32:15 +0200 Subject: [PATCH 6/6] openspec: Archive gentoo-distfiles-proxy change spec --- .../.openspec.yaml | 0 .../design.md | 0 .../proposal.md | 0 .../specs/cache-exclude/spec.md | 0 .../specs/e2e-multi-distro/spec.md | 0 .../specs/gentoo-distfiles/spec.md | 0 .../tasks.md | 0 openspec/specs/cache-exclude/spec.md | 45 +++++++++++++++++++ openspec/specs/e2e-multi-distro/spec.md | 16 +++++++ openspec/specs/gentoo-distfiles/spec.md | 27 +++++++++++ 10 files changed, 88 insertions(+) rename openspec/changes/{gentoo-distfiles-proxy => archive/2026-04-06-gentoo-distfiles-proxy}/.openspec.yaml (100%) rename openspec/changes/{gentoo-distfiles-proxy => archive/2026-04-06-gentoo-distfiles-proxy}/design.md (100%) rename openspec/changes/{gentoo-distfiles-proxy => archive/2026-04-06-gentoo-distfiles-proxy}/proposal.md (100%) rename openspec/changes/{gentoo-distfiles-proxy => archive/2026-04-06-gentoo-distfiles-proxy}/specs/cache-exclude/spec.md (100%) rename openspec/changes/{gentoo-distfiles-proxy => archive/2026-04-06-gentoo-distfiles-proxy}/specs/e2e-multi-distro/spec.md (100%) rename openspec/changes/{gentoo-distfiles-proxy => archive/2026-04-06-gentoo-distfiles-proxy}/specs/gentoo-distfiles/spec.md (100%) rename openspec/changes/{gentoo-distfiles-proxy => archive/2026-04-06-gentoo-distfiles-proxy}/tasks.md (100%) create mode 100644 openspec/specs/cache-exclude/spec.md create mode 100644 openspec/specs/gentoo-distfiles/spec.md diff --git a/openspec/changes/gentoo-distfiles-proxy/.openspec.yaml b/openspec/changes/archive/2026-04-06-gentoo-distfiles-proxy/.openspec.yaml similarity index 100% rename from openspec/changes/gentoo-distfiles-proxy/.openspec.yaml rename to openspec/changes/archive/2026-04-06-gentoo-distfiles-proxy/.openspec.yaml diff --git a/openspec/changes/gentoo-distfiles-proxy/design.md b/openspec/changes/archive/2026-04-06-gentoo-distfiles-proxy/design.md similarity index 100% rename from openspec/changes/gentoo-distfiles-proxy/design.md rename to openspec/changes/archive/2026-04-06-gentoo-distfiles-proxy/design.md diff --git a/openspec/changes/gentoo-distfiles-proxy/proposal.md b/openspec/changes/archive/2026-04-06-gentoo-distfiles-proxy/proposal.md similarity index 100% rename from openspec/changes/gentoo-distfiles-proxy/proposal.md rename to openspec/changes/archive/2026-04-06-gentoo-distfiles-proxy/proposal.md diff --git a/openspec/changes/gentoo-distfiles-proxy/specs/cache-exclude/spec.md b/openspec/changes/archive/2026-04-06-gentoo-distfiles-proxy/specs/cache-exclude/spec.md similarity index 100% rename from openspec/changes/gentoo-distfiles-proxy/specs/cache-exclude/spec.md rename to openspec/changes/archive/2026-04-06-gentoo-distfiles-proxy/specs/cache-exclude/spec.md diff --git a/openspec/changes/gentoo-distfiles-proxy/specs/e2e-multi-distro/spec.md b/openspec/changes/archive/2026-04-06-gentoo-distfiles-proxy/specs/e2e-multi-distro/spec.md similarity index 100% rename from openspec/changes/gentoo-distfiles-proxy/specs/e2e-multi-distro/spec.md rename to openspec/changes/archive/2026-04-06-gentoo-distfiles-proxy/specs/e2e-multi-distro/spec.md diff --git a/openspec/changes/gentoo-distfiles-proxy/specs/gentoo-distfiles/spec.md b/openspec/changes/archive/2026-04-06-gentoo-distfiles-proxy/specs/gentoo-distfiles/spec.md similarity index 100% rename from openspec/changes/gentoo-distfiles-proxy/specs/gentoo-distfiles/spec.md rename to openspec/changes/archive/2026-04-06-gentoo-distfiles-proxy/specs/gentoo-distfiles/spec.md diff --git a/openspec/changes/gentoo-distfiles-proxy/tasks.md b/openspec/changes/archive/2026-04-06-gentoo-distfiles-proxy/tasks.md similarity index 100% rename from openspec/changes/gentoo-distfiles-proxy/tasks.md rename to openspec/changes/archive/2026-04-06-gentoo-distfiles-proxy/tasks.md diff --git a/openspec/specs/cache-exclude/spec.md b/openspec/specs/cache-exclude/spec.md new file mode 100644 index 0000000..4f3598d --- /dev/null +++ b/openspec/specs/cache-exclude/spec.md @@ -0,0 +1,45 @@ +## Requirements + +### Requirement: Wildcard suffix caches all files +When the `suffixes` list for a repository contains `"*"`, the cache SHALL treat every proxied file as a cache candidate, subject to the `exclude` list. + +#### Scenario: File with uncommon extension is cached under wildcard repo +- **WHEN** a request is made for a file with an extension not in any explicit suffix list (e.g. `.crate`) under a repo with `suffixes: ["*"]` +- **THEN** `IsCacheCandidate` returns true + +#### Scenario: Wildcard does not affect repos without it +- **WHEN** a request is made for a file under a repo whose `suffixes` list does not contain `"*"` +- **THEN** `IsCacheCandidate` applies the existing suffix-match logic unchanged + +### Requirement: Exclude list prevents specific files from being cached +A repository MAY define an `exclude` list. Each entry is matched against the request filename as an exact name first, then as a suffix. If any entry matches, the file SHALL NOT be cached regardless of `suffixes`. + +#### Scenario: Exact filename match prevents caching +- **WHEN** a request is made for a file whose name exactly matches an `exclude` entry (e.g. `layout.conf`) +- **THEN** `IsCacheCandidate` returns false + +#### Scenario: Suffix match prevents caching +- **WHEN** a request is made for a file whose name ends with an `exclude` entry (e.g. `.sig`) +- **THEN** `IsCacheCandidate` returns false + +#### Scenario: Non-matching file is not excluded +- **WHEN** a request is made for a file that does not match any `exclude` entry +- **THEN** the `exclude` list has no effect on the cache candidacy decision + +#### Scenario: Exclude applies without wildcard suffix +- **WHEN** a repository has explicit suffixes (no `"*"`) and an `exclude` list, and a request is made for a file that matches both a suffix and an exclude entry +- **THEN** `IsCacheCandidate` returns false (exclude takes precedence) + +### Requirement: Explicit suffixes alongside wildcard are redundant but valid +When the `suffixes` list contains both `"*"` and explicit suffix entries, the configuration SHALL be accepted. pkgproxy SHALL log a warning identifying the repository and the redundant entries. Cache behavior is identical to having only `"*"`. + +#### Scenario: Mixed wildcard and explicit suffixes triggers a warning +- **WHEN** pkgproxy loads a repository config whose `suffixes` list contains `"*"` and at least one other entry +- **THEN** the repository is accepted without error, a warning is logged naming the repository and the redundant suffixes, and `IsCacheCandidate` behaves as if only `"*"` were present + +### Requirement: Exclude field is optional +The `exclude` field in a repository config SHALL be optional. Repositories without it SHALL behave identically to the current behavior. + +#### Scenario: Repository without exclude field +- **WHEN** pkgproxy loads a repository config with no `exclude` key +- **THEN** the repository is accepted without error and cache behavior is unchanged diff --git a/openspec/specs/e2e-multi-distro/spec.md b/openspec/specs/e2e-multi-distro/spec.md index b2d87db..b7b7d03 100644 --- a/openspec/specs/e2e-multi-distro/spec.md +++ b/openspec/specs/e2e-multi-distro/spec.md @@ -83,3 +83,19 @@ COPR repository configuration for non-Fedora DNF-based distros (CentOS Stream, A #### Scenario: Non-Fedora COPR uses epel pattern - **WHEN** the Go test generates a COPR repo file for AlmaLinux - **THEN** the baseurl uses the pattern `http:///copr/ganto/jo/epel-$releasever-$basearch/` + +### Requirement: Gentoo e2e test +The test suite SHALL include a Gentoo test function `TestGentoo` using a `docker.io/gentoo/stage3:latest` container. The test script SHALL: +1. Download the latest portage ebuild snapshot directly from `https://distfiles.gentoo.org/snapshots/portage-latest.tar.xz` (bypassing the proxy — this is bootstrap, not a distfile fetch). +2. Unpack the snapshot into `/var/db/repos/gentoo` inside the container. +3. Configure `GENTOO_MIRRORS` in `/etc/portage/make.conf` to point at the pkgproxy `gentoo` repository. +4. Run `emerge --fetchonly app-text/tree` to fetch the `tree` package sources through the proxy. +5. Fetch `http:///gentoo/distfiles/layout.conf` via `wget` to exercise the negative cache path. + +#### Scenario: emerge --fetchonly proxies and caches tree distfiles +- **WHEN** the Gentoo container runs `emerge --fetchonly app-text/tree` with `GENTOO_MIRRORS` pointing at pkgproxy +- **THEN** the command exits successfully and the tree source archive exists in the pkgproxy cache under the `gentoo/` subdirectory + +#### Scenario: layout.conf is proxied but not cached +- **WHEN** the Gentoo container fetches `http:///gentoo/distfiles/layout.conf` via `wget` +- **THEN** the request returns HTTP 200 and `layout.conf` does NOT exist in the pkgproxy cache under the `gentoo/` subdirectory diff --git a/openspec/specs/gentoo-distfiles/spec.md b/openspec/specs/gentoo-distfiles/spec.md new file mode 100644 index 0000000..57b46f1 --- /dev/null +++ b/openspec/specs/gentoo-distfiles/spec.md @@ -0,0 +1,27 @@ +## Requirements + +### Requirement: Gentoo distfiles repository config entry +The `configs/pkgproxy.yaml` SHALL include a `gentoo` repository configured with `suffixes: ["*"]`, an `exclude` list covering mirror-specific metadata files (`layout.conf`, `timestamp.mirmon`, `timestamp.dev-local`), and at least two Swiss HTTPS mirrors plus `distfiles.gentoo.org` as authoritative fallback. + +#### Scenario: Gentoo distfiles repository is configured +- **WHEN** pkgproxy loads its configuration +- **THEN** the `gentoo` repository is available with at least one upstream mirror + +#### Scenario: layout.conf is not cached +- **WHEN** a client fetches `/gentoo/distfiles/layout.conf` +- **THEN** pkgproxy proxies the file upstream but does not write it to the local cache + +#### Scenario: Distfile fetched via emerge --fetchonly is proxied and cached +- **WHEN** portage runs `emerge --fetchonly app-text/tree` with `GENTOO_MIRRORS` pointing at pkgproxy +- **THEN** pkgproxy proxies the distfile from the upstream mirror and saves it to the local cache under `gentoo/distfiles//` + +#### Scenario: Cached distfile is served from disk on subsequent request +- **WHEN** portage fetches the same distfile a second time +- **THEN** pkgproxy serves the file from the local cache without contacting the upstream mirror + +### Requirement: make.conf snippet in README and landing page +The README.md and HTTP landing page SHALL include a Gentoo `make.conf` snippet showing how to configure `GENTOO_MIRRORS` to point at the proxy. + +#### Scenario: Gentoo configuration snippet is present +- **WHEN** a user views the README or the pkgproxy landing page +- **THEN** a `make.conf` snippet with `GENTOO_MIRRORS="http:///gentoo"` is visible