diff --git a/docs/Architecture.md b/docs/Architecture.md new file mode 100644 index 000000000..580786b52 --- /dev/null +++ b/docs/Architecture.md @@ -0,0 +1,307 @@ +# compose-go loading pipeline + +This document describes how `compose-go` turns one or more Compose YAML +files into a `*types.Project`. It targets contributors who need to +extend the loader, debug an unexpected merge result, or understand why +a given fixture surfaces a particular value. + +## Goals + +The loader is built around three invariants: + +1. **A YAML node never loses its origin.** Every scalar carries an + implicit "where did this come from" — file path, working directory, + environment of the layer that introduced it. The pipeline preserves + that link so interpolation, path resolution and diagnostics can + reason about it per scalar. +2. **Lazy resolution at the right scope.** Variables, `env_file` + contents, `extends` targets and `include` directives are resolved in + the scope of the layer that declared them, not the global project + scope. A variable introduced by an include's `env_file` is visible + to the scalars from that include and to nothing else. +3. **One canonical merged tree.** All files end up in a single + `*yaml.Node` tree that two helpers project into either a + `map[string]any` (for `LoadModelWithContext`) or a typed + `*types.Project` (for `LoadWithContext`). + +The implementation stays in the `loader` package; supporting helpers +live in `internal/node`, `paths`, `override`, `interpolation`, +`transform`, `validation` and `normalize`. + +## Public entry points + +```go +loader.LoadWithContext(ctx, configDetails, opts...) (*types.Project, error) +loader.LoadModelWithContext(ctx, configDetails, opts...) (map[string]any, error) +``` + +Both share the same `Options` struct, run the same pipeline, and differ +only in their final projection. There is no other supported way to drive +the loader — internal helpers (`load`, `nodeToProject`, `nodeToModel`) are +not exported. + +## Core types + +The pipeline works on a small set of values that survive across phases: + +- **`internal/node.Layer`** holds one parsed compose document. It wraps + the document's `*yaml.Node` together with the `SourceContext` that + describes where the document came from. A layer is the unit of merge + and the unit of resource scope. +- **`internal/node.SourceContext`** records, for one layer, the absolute + file path, the working directory used to resolve its relative paths, + the environment in effect during its load, and the `env_file` entries + that contributed to that environment. It also carries a pointer to its + parent context so include / extends chains can walk back to the top. +- **`origins map[*yaml.Node]*node.SourceContext`** is a side-table + populated after every layer has been parsed and after extends have run + in place. It attributes every reachable node to the layer that + produced it; the path resolver, the environment resolver and the + interpolation step all consult it to pick the right scope for each + scalar. + +## Pipeline + +`load(ctx, cd, opts)` in `loader/load.go` is the single orchestrator. It +runs the steps below; numbers match the doc comment at the top of the +function. + +### 1. Parse + per-document hygiene + +For every `ConfigFile` in `ConfigDetails`, `LoadLayer` (see +`loader/load_layer.go`): + +- reads the YAML into one or more documents (`---` separated), +- runs `internal/node.ResolveResetOverride` to extract every `!reset` + and `!override` path while removing the tags from the tree, +- runs `internal/node.NormalizeAliases` to unfold anchors / merge keys + (with a cap on total deep-copies to defend against alias bombs), +- runs `checkStringKeys` to surface non-string mapping keys with a + precise diagnostic, +- wraps the resulting tree in a `Layer` with its `SourceContext`. + +### 2. Recursive `include` + +`CollectIncludeLayers` (in `loader/load_include.go`) reads the top-level +`include:` block on a layer, interpolates only that block in the parent +environment, then for each entry: + +- resolves the `path` through the registered `ResourceLoader`s, +- computes the include's `project_directory` (absolute) and loads the + declared `env_file`s on top of the parent environment, +- recursively expands its own includes (cycle detected via the absolute + filename chain in `expandIncludes`), +- (when `opts.ResolvePaths` is on) runs a sub-resolve so paths inside + the included file are absolutized in the include's working directory. + +The included layers are appended in order so the parent's overrides win +later in the merge. + +### 3. `extends` + +`applyExtendsPerLayer` (in `loader/load.go`) iterates every service in +every layer. For each `extends:` directive: + +- `parseExtendsRef` extracts the `(service, file)` tuple and emits the + `extends` Listener event, +- `loadExtendsBaseLayer` loads the base file if needed, with a child + `Options` whose `ResourceLoaders` is re-rooted at the base file's + directory, +- the chain recurses (cycle tracked by `cycleTracker.Add(file, service)`), +- the base service is `deepCloneNode`d, `!reset`/`!override` paths from + the current layer are pre-applied to the clone, then + `override.MergeNode` merges base + derived under the canonical + `services.x` path, +- `resolveExtendedServicePaths` runs path resolution on the merged + service against the v2-compatible relative WorkingDir carried in + `Options.extendsRelativeDir`, so cross-file extends accumulate the + expected relative form. + +`extends` mutates the layer in place — the merged service body +replaces its original entry inside the layer's tree. + +### 4. Origins side-table + +`populateOrigins` walks every layer once and records +`origins[node] = layer.Context`. Entries already present are preserved, +which leaves room for follow-ups that pre-stamp extends clones with the +base layer's context. + +### 5. Cross-file merge + +`mergeLayers` folds the layers left-to-right with `override.MergeNode`. +`ConfigFiles[0]` is the base, every later layer overrides. The combined +`!reset` / `!override` path list is collected; `node.ApplyResetPaths` +applies it to the merged tree once the fold is complete. The `include:` +key is then deleted from the result — it was consumed in phase 2 and +must not appear in the final project. + +### 6. Lazy interpolation + +`interpolateMerged` walks the merged tree and substitutes `${VAR}` per +scalar using `origins[scalar].Environment` as the lookup. The +interpolation hook also stamps `node.Tag` according to +`tagsForCasts()` so a `published: "80"` declared as a quoted string +decodes as an integer downstream without any explicit cast pass. + +### 7. Schema validation + +`validateAndStripVersion` runs the JSON-schema validator on a decoded +view of the merged tree before any canonicalization. Catching structural +errors here keeps the canonical transform free of defensive checks. +A successful validation also strips the legacy top-level `version` +attribute and emits the obsolete-version warning per file. + +### 8. Per-scalar bare-key environment resolution + +`ResolveEnvironmentNode` rewrites every bare `KEY` entry under +`services.*.environment`, `secrets.*.environment` and +`configs.*.environment` to `KEY=value`, picking each `value` from +`origins[scalar].Environment`. Bare keys without a match in scope are +left as-is, matching the v2 behavior that separates "interpolation +produced an empty string" from "no value found". + +### 9. Secret / config `environment:` capture + +`CaptureSecretConfigContent` returns two `name -> resolved value` maps +recorded *before* CanonicalNode reshuffles pointers. The maps are +applied at the very end of the pipeline (`ApplySecretConfigContent`) +after validation, so the synthesized `content:` does not trip the +content/environment mutual-exclusivity rule. + +### 10. Path resolution (pre-canonical) + +When `ResolvePaths` is on, `paths.ResolveRelativePathsNode` walks the +tree with a `WorkingDirFor` closure that picks each scalar's WorkingDir +from the origins map. Each handler in `paths/node.go` is path-pattern +keyed and operates on its specific shape (`absScalar`, `absVolumeMount`, +`absEnvFileShortForm`, ...). Paths that look already absolute (Unix or +Windows) are skipped; relative paths are joined against the right WD. + +### 11. Canonicalization + +`buildServiceContexts` snapshots, *before* `transform.CanonicalNode` +re-encodes the tree, a `service name -> WorkingDir` map. The canonical +transform converts every short form to its long form by going through a +`map[string]any` round-trip; node pointers are not stable across it, so +this snapshot is how the next two phases keep per-service attribution. + +### 12. Defaults + post-canonical path resolution + +`setDefaultValuesNode` runs the v2 `transform.SetDefaultValues` through +a temporary map projection. `resolveDefaultBuildContext` then walks the +default-`.` build contexts and rewrites them with the service's +WorkingDir from the snapshot. `resolveServiceVolumeSources` does the +same for short-form bind volumes whose source still carries the leading +`.`/`..` indicator — they were skipped pre-canonical because +`absVolumeMount` only matches the long form. + +### 13. Compose-rule validation + +`validation.ValidateNode` runs the cross-cutting rules (volumes +referenced by services exist, secrets / configs declare exactly one +source, network drivers and IPAM consistency, ...). The project name +is required to be non-empty at this point unless `SkipValidation` is +set. + +### 14. Normalization + +`NormalizeNode` applies the canonical defaults (default network, implicit +`depends_on` derived from `network_mode`, models containing files, ...). +The current implementation reuses the v2 `Normalize` through a +map roundtrip; a Node-native port is on the roadmap. + +### 15. Trim + finalize + +`omitEmptyNode` drops entries whose value collapsed to empty after +interpolation (`dns: ${UNSET}` produces `dns: ""` which is then +removed). `ApplySecretConfigContent` injects the captured +secret/config `content` scalars. The returned `*yaml.Node` is the +canonical merged tree. + +## Projection + +The merged tree is fed into one of two helpers: + +- **`nodeToProject(root, opts, cd)`** strips the `name` scalar so it + cannot override `opts.projectName`, `root.Decode(&project)` projects + the tree into `*types.Project` via the per-type `UnmarshalYAML` + methods registered on every compose-go type (no mapstructure, + no map intermediate). It then applies the project-level + post-decode passes: `decodeKnownExtensions` re-decodes registered + `x-*` targets, `EnvFileScopes` is stamped from `opts.envFileScopes`, + `WithProfiles` / `WithSelectedServices` / `WithoutUnnecessaryResources` + prune the project, `WithServicesEnvironmentResolved` and + `WithServicesLabelsResolved` finish the environment plumbing. +- **`nodeToModel(root)`** projects the tree into a `map[string]any` via + a single `root.Decode(&dict)` call. `OmitEmpty` and the + secret/config environment resolution have already run at the node + level, so the dict matches the legacy v2 loadYamlModel output. + +`Project.EnvFileScopes` is the side-table that ties the two halves +together: `load` records, for every `env_file` path, the environment +of the layer that declared it, and +`WithServicesEnvironmentResolved` consults it when interpolating +the file content. This is what lets a `secret` declared inside an +included file see variables introduced by the include's own `env_file`. + +## Extension points + +- **`ResourceLoader`** (`loader.ResourceLoader`) plugs in custom + protocols for `include.path` and `extends.file`. The built-in + `localResourceLoader` is always appended last so every other loader + has a chance to claim the URI first. Each loader exposes `Accept`, + `Load` (returns a local path), `Dir` (parent directory rendered + relative to the project root when possible). +- **`KnownExtensions`** maps `x-foo` to a target type. After decode, + `decodeKnownExtensions` walks every Extensions map (project, service, + network, volume, config, secret) and re-decodes registered entries + into the target type via a yaml round-trip. +- **`Listeners`** receive structured events emitted by the pipeline + (`extends`, `include`, ...). They are append-only metadata and do + not influence the merge result. + +## File map + +``` +loader/ + load.go # orchestrator (load), pipeline glue + load_layer.go # parse + reset + alias normalization + load_include.go # CollectIncludeLayers, env_file scope plumbing + load_extends.go # ApplyExtendsToLayer, chained extends + loader.go # Options, ResourceLoader, public entry points, + # nodeToProject / nodeToModel projections + normalize_node.go # NormalizeNode bridge + resolve_environment_node.go # bare-key + secret/config env resolution + reset.go # !reset / !override processor + interpolate.go # interpolateMerged + transform/ # CanonicalNode + per-rule short -> long form + internal/node/ # Layer, SourceContext, walker, merge primitives, + # alias normalization +override/ # MergeNode + EnforceUnicityNode + per-path rules +paths/ # ResolveRelativePathsNode + per-path resolvers +interpolation/ # Per-scalar substitution engine +validation/ # ValidateNode + per-rule checks +schema/ # JSON Schema definitions + validator +types/ # Project, ServiceConfig, ..., UnmarshalYAML + # methods (yaml.v4 native, no mapstructure) +``` + +## Adding a new transform / validation rule + +1. **Decide whether the rule is structural or behavioral.** + Schema-level checks (top-level kind, required fields, enum values) + belong in `schema/`. Compose-specific cross-references (a service's + `volumes_from` actually exists, a network is declared, ...) belong in + `validation/`. +2. **Pick the right pipeline phase.** Path-shaped values land in `paths/` + (pre-canonical) or in the post-canonical helpers of `load.go` when + they need per-service attribution. Short-form / long-form rewrites + land in `transform/`. +3. **Register a per-path handler.** The walkers in `paths/`, + `override/` and `validation/` are all keyed by `tree.Path` patterns; + add an entry and a function. +4. **Add a fixture.** Each transformer, resolver and validator has a + matching fixture under `loader/testdata/`; add one that covers the + new rule, declare its expected `*Project` in a `*_test.go` and run + it through `LoadWithContext`. diff --git a/docs/Interpolation.md b/docs/Interpolation.md new file mode 100644 index 000000000..a558eee0a --- /dev/null +++ b/docs/Interpolation.md @@ -0,0 +1,245 @@ +# Interpolation of YAML values + +This document describes how `${VAR}` expressions are substituted into +compose files, which scope each scalar is interpolated against, and how +`include` with `env_file` changes that scope. + +## What gets interpolated + +Interpolation operates on **every scalar in the merged tree** after +includes and extends have been folded in (steps 5–6 of the pipeline, +see `docs/Architecture.md`). It happens in place on the `*yaml.Node` +tree, so the substitution preserves the original Style (single-quoted, +double-quoted, plain, ...). Type casting is applied at the same time by +rewriting `Node.Tag` (see "Type casts" below). + +The syntax is a strict subset of Bash parameter expansion (full list in +`template/`): + +| Form | Behavior | +| --------------------- | ------------------------------------------------------------------ | +| `${VAR}` / `$VAR` | Substitute the value, fail in strict mode if unset. | +| `${VAR-default}` | Use `default` when `VAR` is unset. | +| `${VAR:-default}` | Use `default` when `VAR` is unset *or* empty. | +| `${VAR?error}` | Fail with `error` when `VAR` is unset. | +| `${VAR:?error}` | Fail with `error` when `VAR` is unset *or* empty. | +| `${VAR+value}` | Use `value` when `VAR` is set (even to ""). | +| `${VAR:+value}` | Use `value` when `VAR` is set and non-empty. | +| `$$` / `${$}` | Literal `$`, no substitution. | + +Bracket bodies may nest (`${OUTER:-${INNER:-fallback}}`) and the +default / error operands themselves go through substitution. + +## The "lazy" principle + +The classic Compose loader interpolated each file in its own scope **at +parse time**, before merge. `compose-go` v3 merges first and +interpolates last, but it does so **per scalar** rather than at the +top of the tree. Every scalar carries an implicit "where did this come +from" via the `origins` side-table, and the substitution engine picks +the lookup function for that scalar from the layer's +`SourceContext.Environment`: + +```go +interp.Options{ + LookupValue: func(node *yaml.Node, key string) (string, bool) { + ctx := origins[node] + if ctx == nil { + return "", false + } + v, ok := ctx.Environment[key] + return v, ok + }, +} +``` + +This is the **lazy interpolation principle**: a value introduced by one +layer's environment is only visible to the scalars that came from that +layer. Two consequences fall out of it: + +- A variable declared in an include's `env_file` is visible to scalars + declared inside that include (and to scalars from files that include + declares in turn), but never to the parent that did the include. +- A variable declared in the top-level shell environment is visible to + every scalar **unless** an include layer happens to redefine it; in + that case the include's own value wins inside the include scope. + +## Composing the environment of a layer + +`SourceContext.Environment` is built layer by layer. Going down the +chain: + +1. **Root context.** `cd.Environment` (the value the caller passes to + `LoadWithContext`) seeds the root layer. +2. **`COMPOSE_PROJECT_NAME`.** `projectName(cd, opts)` stamps the + resolved project name onto `cd.Environment` before any layer is + parsed, so it is visible to every scalar in every file. +3. **`include.env_file`.** When an entry under the top-level `include:` + block carries an `env_file:` (relative paths are resolved against + the parent WorkingDir), `resolveIncludeEnvironment` loads each file + in declaration order on top of the parent environment via + `Mapping.Merge`. **Existing keys win**: a key already present in the + parent context keeps its parent value, the include's `env_file` + value is dropped. This matches the v2 behavior where the shell + environment overrides `.env` entries. +4. **Implicit `/.env`.** If an include declares no + explicit `env_file:` *and* a `.env` file exists at the include's + `project_directory`, it is loaded as if it had been listed + explicitly. A single `/dev/null` entry in `env_file:` disables this + implicit lookup *and* skips the listed entry. +5. **Extends.** Layers loaded through `extends.file` inherit the parent + layer's environment as-is. There is no `env_file` mechanism on the + `extends` block itself. + +The resulting `SourceContext.Environment` is the lookup table used by +every scalar that originates from that layer. + +## Worked example: include scope + +Consider this fixture (close to `testdata/include/env_file/`): + +```yaml +# compose.yaml +include: + - path: sub/compose.yaml + env_file: + - sub/.env.include + +services: + parent: + image: ${IMG:-base} +``` + +```yaml +# sub/compose.yaml +services: + app: + image: alpine + env_file: + - extra.env +``` + +```sh +# sub/.env.include +BAR=bar +IMG=ignored-because-shell-wins +``` + +```sh +# sub/extra.env +FOO=$BAR +OVR=${BAR:-fallback} +``` + +Loaded with `cd.Environment = {"IMG": "shell"}`: + +| Scalar | Layer | Lookup observes | Result | +| ----------------------------------------- | ---------------------- | --------------------------------------- | ---------------------------- | +| `services.parent.image: ${IMG:-base}` | top-level | shell `IMG=shell` | `image: shell` | +| `services.app.image: alpine` | sub include | shell + include `env_file` | `image: alpine` (literal) | +| `extra.env` `FOO=$BAR` | sub include `env_file` | shell + include `env_file` (`BAR=bar`) | `FOO=bar` | +| `extra.env` `OVR=${BAR:-fallback}` | sub include `env_file` | shell + include `env_file` (`BAR=bar`) | `OVR=bar` | + +Three things to note: + +- `IMG=shell` from the caller's environment wins over the include + `env_file` value because parent-wins is enforced by `Mapping.Merge`. +- `BAR` is **not** visible to the top-level `parent` service even + though it lives in the same project — the scope of `BAR` is the + include layer. +- `extra.env` is itself processed in the include's scope: its content + is interpolated against the **include's `env_file`**, not against the + caller's shell environment, so `$BAR` resolves to `bar`. + +## `env_file` on a service (vs on `include`) + +`services.*.env_file` is a different code path from `include.env_file`, +but the same lazy principle applies. `WithServicesEnvironmentResolved` +(see `types/project.go`) reads each file, interpolates `${VAR}` inside +the file content, then merges the result into the service's +`Environment` map. The lookup function it passes prefers, in order: + +1. The service's already-resolved `Environment` (so a variable set on + the service can be referenced by a later `env_file` entry). +2. `Project.EnvFileScopes[envFile.Path]` — the *layer* environment + captured at `env_file` declaration time. This is what makes the + scope honor lazy interpolation across includes. +3. The project-wide `Environment` (fallback when the entry was + declared at the top level and no scope was captured). + +`Project.EnvFileScopes` is populated during `load`: when a layer +declares an `env_file` entry, the loader records +`scopes[absoluteEnvFilePath] = layer.Context.Environment`. The +resulting map is attached to the `*Project` by `nodeToProject` and +preserved across `deepCopy`. From the consumer side, calling +`WithServicesEnvironmentResolved` on a project built with an include +that brought its own `env_file` produces the resolved values that +match what an interpolation pass *inside* that include would produce. + +## `secrets:` / `configs:` declared with `environment:` + +When a `secrets.NAME` or `configs.NAME` entry has the +`environment: VARNAME` shorthand, the value of `VARNAME` is looked up +in the layer that declared the secret/config — exactly the same scope +the surrounding scalar interpolation would use. The lookup happens +during `load` via `CaptureSecretConfigContent`, which walks the tree +pre-canonical (where origin pointers are still valid) and records +`name -> resolved value`. The map is applied to the post-canonical +tree by `ApplySecretConfigContent` so the synthesized `content:` +scalar reaches `SecretConfig.UnmarshalYAML` / `ConfigObjConfig.UnmarshalYAML`. + +Practical effect: a secret declared inside an included file can pull +its value from a variable introduced by the include's own +`env_file`. The fixture under `testdata/include/secret_env/` +covers this scenario. + +## Type casts + +Schema-driven type conversions are wired in at the interpolation step +rather than as a separate post-pass. `tagsForCasts()` maps a +`tree.Path` pattern to a YAML tag (`!!int`, `!!bool`, `!!float`), and +the interpolation hook rewrites `node.Tag` accordingly after +substitution. yaml.v4 then performs the conversion natively at decode +time. The two consequences: + +- `published: "${PORT}"` with `PORT=80` decodes as an integer because + `services.*.ports.*.published` is tagged `!!int`. +- `init: "${INIT}"` with `INIT=true` decodes as a boolean for the same + reason on `services.*.init`. + +The list of cast targets matches the v2 `interpolateTypeCastMapping`; +adding a new one is a one-line entry in `tagsForCasts()`. + +## Strict vs lenient mode + +`Options.Interpolate.Substitute` is the entry point of the substitution +engine; `cd.LookupEnv` is the default `LookupValue`. The default mode +treats an unset variable as the empty string and emits a warning. To +fail fast on missing variables, use the strict variants in the source +file: + +```sh +image: nginx:${TAG:?TAG must be set} +``` + +The error surfaces at the scalar where the unset variable was +referenced, with the file / line / column of that scalar. + +## Gotchas + +- **Double dollar.** A literal `$` inside a value must be escaped as + `$$`, otherwise the engine tries to interpret it. In particular + `command: "echo $$PATH"` produces `echo $PATH` at runtime. +- **Quoted scalars.** Interpolation operates on the parsed scalar + value, not on the YAML source. `image: "${TAG}"` and `image: ${TAG}` + produce the same result (modulo type casts: only the unquoted form is + eligible for a `!!int` rewrite, the quoted form stays `!!str`). +- **Include order matters.** The include layers are merged in + declaration order; a later include that ships an + `env_file:` value for a key already present in an earlier include + scope does *not* override the earlier scope. Each include keeps its + own environment composition. +- **Implicit `.env`.** The `.env` lookup at the project root is + performed by the *caller* (the CLI), not by `compose-go`. The + loader honors `cd.Environment` as the single source of truth for + the root context. diff --git a/docs/Migration.md b/docs/Migration.md new file mode 100644 index 000000000..9c4017fd0 --- /dev/null +++ b/docs/Migration.md @@ -0,0 +1,274 @@ +# Migrating from compose-go v2 to v3 + +v3 is a major release. The public entry points (`LoadWithContext`, +`LoadModelWithContext`, `Options`, `*types.Project`) keep the same +signature, so most callers compile against v3 with only the module +path change. This document lists the breaking changes a caller might +trip over and the behavioral changes a caller might rely on. + +## Module path + +```go +// v2 +import "github.com/compose-spec/compose-go/v2/loader" + +// v3 +import "github.com/compose-spec/compose-go/v3/loader" +``` + +Update every import path in your project from `v2` to `v3`. + +## Removed APIs + +| v2 symbol | v3 replacement | +| ---------------------------------------- | ------------------------------------------------------------------------- | +| `loader.Transform(source, target)` | `(*yaml.Node).Decode(&target)` (yaml.v4 native, no mapstructure) | +| `loader.ModelToProject(dict, opts, cd)` | `loader.LoadWithContext(ctx, cd, opts...)` returns `*types.Project` | +| `loader.ApplyInclude(...)` | Internal -- includes are processed by `loader.CollectIncludeLayers` | +| `loader.ApplyExtends(...)` | Internal -- `loader.ApplyExtendsToLayer` on the yaml.Node tree | +| `loader.Normalize(dict, env)` | Still works as a wrapper, prefer `loader.NormalizeNode(*yaml.Node, env)` | +| `loader.OmitEmpty(dict)` | Internal -- runs as part of `Load` | +| `loader.ResolveEnvironment(dict, env)` | Internal -- per-scalar `loader.ResolveEnvironmentNode` | +| `types..DecodeMapstructure(value)` | Replaced by `UnmarshalYAML(value *yaml.Node) error` on every type | + +The `Transform` removal is the most visible break: code that turned an +arbitrary `map[string]any` into a typed compose-go struct via +`loader.Transform` should call `yaml.Marshal` + `yaml.Unmarshal` (or +`(*yaml.Node).Decode`) instead. + +## Removed dependency + +`github.com/go-viper/mapstructure/v2` is no longer in `go.mod`. Every +compose-go type now exposes `UnmarshalYAML(*yaml.Node) error` and the +loader projects directly into `*types.Project` via yaml.v4. A +downstream module that imported mapstructure only because compose-go +required it can drop it too. + +## YAML tags + +Two structs gained explicit `yaml:` tags on fields that previously +relied on yaml.v3's lowercased-field-name fallback. The serialized +form is unchanged; the tags are required because yaml.v4 is stricter: + +```go +type WeightDevice struct { + Path string `yaml:"path,omitempty"` + Weight uint16 `yaml:"weight,omitempty"` + // ... +} + +type ThrottleDevice struct { + Path string `yaml:"path,omitempty"` + Rate UnitBytes `yaml:"rate,omitempty"` + // ... +} +``` + +Downstream forks that embed these structs in their own types should +mirror the tags. + +## Error types + +Most user-facing failures now surface as `*errdefs.Diagnostic`. The +type carries the source file, line, column and dotted compose path +alongside the underlying cause: + +```go +type Diagnostic struct { + File string + Line int + Column int + Path string + Cause error +} +``` + +`Diagnostic.Error()` renders as `file:line:col: path: cause` (each +segment elided when missing). Existing error handling that matched on +the legacy string format needs to switch to substring matching or use +`errors.As` to inspect the typed value: + +```go +var diag *errdefs.Diagnostic +if errors.As(err, &diag) { + fmt.Printf("at %s:%d:%d: %s\n", diag.File, diag.Line, diag.Column, diag.Cause) +} +``` + +`errors.Is` and `errors.As` still walk through the wrapped Cause. + +The following sites are now wrapped: + +- JSON Schema validation (`schema.Validate` failures) +- Compose-rule validation (`validation.ValidateNode` failures) +- Interpolation strict mode (`${VAR:?msg}` against unset variables) +- Include cycle (`include cycle detected`) +- `include must be a list` and per-entry shape errors +- `extends.NAME.service is required` and related extends ref errors +- `services.NAME must be a mapping` +- `cannot extend service in F: no services section` +- `cannot extend service in F: service Q not found` + +Tests that asserted on the exact legacy strings need to update to the +new prefixed form, e.g. + +```diff +- assert.Error(t, err, "extends.test.service is required") ++ assert.Error(t, err, "(inline):7:7: services.test.extends: extends.test.service is required") +``` + +A typed assertion via `errors.As(err, &diag)` is more robust. + +## Behavioral changes + +### Lazy, per-scalar interpolation + +`${VAR}` is now substituted per scalar in the merged tree, with the +lookup scoped to the `SourceContext.Environment` of the layer that +declared that scalar. The classic v2 behavior interpolated each file +in its own scope at parse time; the result for a flat project is +unchanged, but two scenarios behave differently: + +- A variable declared by an include's `env_file` is now visible to + scalars declared inside that include, including the content of + service-level `env_file` entries declared in the included file. +- A variable from the parent shell environment is still visible to + every scalar (parent-wins via `Mapping.Merge`). + +See `docs/Interpolation.md` for the full scope composition rules. + +### Per-include path resolution + +Relative paths inside an included file are resolved against the +include's own `project_directory`, not the project root. v2 always +joined relative paths with the project root, which was a known +limitation. + +### `Project.EnvFileScopes` + +The `*types.Project` returned by `LoadWithContext` now carries an +`EnvFileScopes map[string]Mapping` side-table keyed by absolute +env_file path. `WithServicesEnvironmentResolved` consults it when +interpolating env_file content so a file referenced from an include +block resolves variables against the include env_file values rather +than only the project-wide environment. The map is hidden from +`yaml.Marshal` / `json.Marshal` (`yaml:"-" json:"-"`) and preserved +by `deepCopy`. + +### `Project.Sources` (opt-in) + +The `*types.Project` returned by `LoadWithContext` can carry a populated +`Sources types.Sources` map (dotted compose path -> file:line:column) +when the loader is invoked with `loader.WithDiagnostics`: + +```go +project, err := loader.LoadWithContext(ctx, cd, loader.WithDiagnostics) +if err != nil { ... } +if loc, ok := project.Sources["services.web.image"]; ok { + fmt.Printf("declared at %s:%d:%d\n", loc.File, loc.Line, loc.Column) +} +``` + +`types.Location` is the small struct that holds the position. The map +covers every mapping path reachable from the merged tree at the moment +the loader snapshots it (just before CanonicalNode reshuffles +pointers). Paths under sequences are stable per index only when those +entries survived the canonical transform without re-encoding; consumers +should treat missing entries as "position not recorded" rather than as +an error. + +Without `WithDiagnostics` the field stays nil, so the project shape is +unchanged for callers that do not opt in. The map is preserved by +`Project.deepCopy` so chained `WithProfiles` / `WithSelectedServices` +calls keep it. + +### `dns: ${UNSET}` collapsing + +The legacy "empty dns drops the list" behavior was a map-level +`OmitEmpty` pass. It now runs on the yaml.Node tree at the same +position in the pipeline. A `dns: ${UNSET}` that interpolates to an +empty string still drops the entry; the decoded `Project.Services.X.DNS` +remains an empty slice. + +### FileMode parsing + +`type FileMode` accepts the same set of source forms (`"0440"`, +`0440`, `288`, `"288"`), but the precedence order changed: octal is +tried first, then decimal as fallback. The motivation is the YAML +round-trip done by extends / canonical, which can re-emit an octal +literal as its decimal equivalent. + +## New helpers + +A handful of building blocks landed in v3 that didn't exist in v2: + +- `internal/node.Layer`, `internal/node.SourceContext`, + `internal/node.NormalizeAliases`, `internal/node.ResolveResetOverride` + -- the yaml.Node-centric primitives. Unexported package + (`internal`), not for public use. +- `loader.LoadLayer`, `loader.CollectIncludeLayers`, + `loader.ApplyExtendsToLayer` -- per-phase entry points if you need + to drive the loader in pieces. +- `loader.ResolveEnvironmentNode`, `loader.CaptureSecretConfigContent`, + `loader.ApplySecretConfigContent` -- node-level passes called by the + orchestrator. Exported because they can be useful in custom + pipelines. + +For a tour of how everything fits together, see `docs/Architecture.md`. + +## Common upgrade recipes + +### "My code called `loader.Transform`" + +```diff +- err := loader.Transform(source, &target) ++ buf, err := yaml.Marshal(source) ++ if err != nil { return err } ++ err = yaml.Unmarshal(buf, &target) +``` + +Or, when `source` is already a `*yaml.Node`: + +```diff +- err := loader.Transform(asMap, &target) ++ err := node.Decode(&target) +``` + +### "My code called `loader.ModelToProject`" + +Switch to the public `LoadWithContext`: + +```diff +- dict, err := loader.LoadModelWithContext(ctx, cd, opts...) +- if err != nil { return nil, err } +- project, err := loader.ModelToProject(dict, optsStruct, cd) ++ project, err := loader.LoadWithContext(ctx, cd, opts...) +``` + +`Options` is built and threaded internally; pass the same option +functions you already pass to `LoadModelWithContext`. + +### "My tests asserted on `validating X: ...` strings" + +The new format is `file:line:col: path: cause`. Either widen the +match to `strings.Contains` on the cause, or assert against the +typed `*errdefs.Diagnostic`: + +```go +var diag *errdefs.Diagnostic +if assert.Check(t, errors.As(err, &diag)) { + assert.Equal(t, diag.Path, "services.web.image") + assert.Assert(t, strings.Contains(diag.Cause.Error(), "must be a string")) +} +``` + +### "My type embedded `WeightDevice` / `ThrottleDevice`" + +Add the explicit `yaml:` tags on the fields: + +```go +Path string `yaml:"path,omitempty" json:"path,omitempty"` +Weight uint16 `yaml:"weight,omitempty" json:"weight,omitempty"` +// or, for ThrottleDevice +Rate UnitBytes `yaml:"rate,omitempty" json:"rate,omitempty"` +``` diff --git a/errdefs/diagnostic.go b/errdefs/diagnostic.go new file mode 100644 index 000000000..5c990422f --- /dev/null +++ b/errdefs/diagnostic.go @@ -0,0 +1,96 @@ +/* + Copyright 2020 The Compose Specification Authors. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +*/ + +package errdefs + +import ( + "fmt" + "strings" +) + +// Diagnostic wraps an error with the file location of the offending +// YAML node. It is the format the loader uses to surface +// interpolation, validation and merge failures with a "file:line:col: +// cause" prefix that points back at the source the user wrote. +// +// File is the absolute path of the source file, or "(inline)" when the +// document was built from in-memory bytes. Line and Column are 1-based; +// zero on either field is rendered as missing. +// +// Diagnostic intentionally implements Unwrap so errors.Is / errors.As +// on the wrapped Cause keep working. +type Diagnostic struct { + File string + Line int + Column int + // Path is the dotted compose path of the offending value + // (e.g. "services.web.ports.0.published"). Optional; included in + // the rendered message when set. + Path string + Cause error +} + +// Error renders the diagnostic. Examples: +// +// /abs/compose.yaml:12:5: services.web.image: invalid value +// /abs/compose.yaml:12: services.web.image: invalid value +// services.web.image: invalid value +func (d *Diagnostic) Error() string { + if d == nil || d.Cause == nil { + return "" + } + var b strings.Builder + if d.File != "" { + b.WriteString(d.File) + if d.Line > 0 { + fmt.Fprintf(&b, ":%d", d.Line) + if d.Column > 0 { + fmt.Fprintf(&b, ":%d", d.Column) + } + } + b.WriteString(": ") + } + if d.Path != "" { + b.WriteString(d.Path) + b.WriteString(": ") + } + b.WriteString(d.Cause.Error()) + return b.String() +} + +// Unwrap exposes the underlying Cause so errors.Is / errors.As walk +// through to the inner error. +func (d *Diagnostic) Unwrap() error { + if d == nil { + return nil + } + return d.Cause +} + +// Diagnose wraps cause as a Diagnostic. Returns nil when cause is nil +// so callers can pass through error returns without an extra check. +func Diagnose(cause error, file string, line, column int, path string) error { + if cause == nil { + return nil + } + return &Diagnostic{ + File: file, + Line: line, + Column: column, + Path: path, + Cause: cause, + } +} diff --git a/go.mod b/go.mod index 52a44a163..9e5e7fcb3 100644 --- a/go.mod +++ b/go.mod @@ -6,7 +6,6 @@ require ( github.com/distribution/reference v0.5.0 github.com/docker/go-connections v0.4.0 github.com/docker/go-units v0.5.0 - github.com/go-viper/mapstructure/v2 v2.4.0 github.com/google/go-cmp v0.5.9 github.com/mattn/go-shellwords v1.0.12 github.com/opencontainers/go-digest v1.0.0 diff --git a/go.sum b/go.sum index b921dc5fc..40a2dddc6 100644 --- a/go.sum +++ b/go.sum @@ -9,8 +9,6 @@ github.com/docker/go-connections v0.4.0 h1:El9xVISelRB7BuFusrZozjnkIM5YnzCViNKoh github.com/docker/go-connections v0.4.0/go.mod h1:Gbd7IOopHjR8Iph03tsViu4nIes5XhDvyHbTtUxmeec= github.com/docker/go-units v0.5.0 h1:69rxXcBk27SvSaaxTtLh/8llcHD8vYHT7WSdRZ/jvr4= github.com/docker/go-units v0.5.0/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk= -github.com/go-viper/mapstructure/v2 v2.4.0 h1:EBsztssimR/CONLSZZ04E8qAkxNYq4Qp9LvH92wZUgs= -github.com/go-viper/mapstructure/v2 v2.4.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM= github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= github.com/google/go-cmp v0.5.9 h1:O2Tfq5qg4qc4AmwVlvv0oLiVAGB7enBSJ2x2DqQFi38= github.com/google/go-cmp v0.5.9/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY= diff --git a/internal/node/aliases.go b/internal/node/aliases.go new file mode 100644 index 000000000..c3c323b83 --- /dev/null +++ b/internal/node/aliases.go @@ -0,0 +1,228 @@ +/* + Copyright 2020 The Compose Specification Authors. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +*/ + +package node + +import ( + "fmt" + + "go.yaml.in/yaml/v4" +) + +// NormalizeAliases walks the yaml.Node tree and removes every AliasNode by +// substituting a deep copy of the alias target in its place, then folds YAML +// merge keys (`<<: *ref`, `<<: [*a, *b]`) into the surrounding mapping with +// surrounding-wins semantics. +// +// After NormalizeAliases returns, no AliasNode remains in the tree and no +// mapping has a `<<` key. The subsequent pipeline phases (cross-file merge, +// interpolation, transform, decode) can therefore operate without any alias +// indirection, which is what makes the per-file Layer model self-contained. +// +// Aliases are deep-copied (rather than reused) because the merge phase +// mutates nodes in place: a node shared between two locations would otherwise +// be corrupted by the first merge involving it. Anchor names are not +// preserved on the copies; once the unfold pass completes, no anchor remains +// reachable. +// +// Cycles in alias chains (A references B which references A) are detected +// during the unfold pass and reported as errors. Cycles created by merge +// keys that resolve to the surrounding mapping are detected the same way +// because the merge value is itself an alias. +func NormalizeAliases(root *yaml.Node) error { + if root == nil { + return nil + } + st := &aliasState{ + inProgress: map[*yaml.Node]bool{}, + cleaned: map[*yaml.Node]bool{}, + sizes: map[*yaml.Node]int{}, + maxNodes: defaultMaxAliasNodes, + } + if err := unfoldAliases(root, st); err != nil { + return err + } + foldMergeKeys(root) + return nil +} + +// defaultMaxAliasNodes caps the total number of nodes created by +// unfoldAliases as it deep-copies alias targets. Sized to accommodate +// large real-world compose files while still rejecting alias-bomb +// documents (e.g. B9_N9 with 9^9 effective nodes after expansion). +const defaultMaxAliasNodes = 1_000_000 + +type aliasState struct { + inProgress map[*yaml.Node]bool + cleaned map[*yaml.Node]bool + // sizes caches the (post-unfold) node count of an anchor target, so a + // reused anchor adds size(target) per reference rather than walking + // the target's subtree again. + sizes map[*yaml.Node]int + created int + maxNodes int +} + +// unfoldAliases replaces AliasNode children of n with deep copies of their +// resolved targets. inProgress tracks targets whose unfolding is on the +// current call stack so cycles are detected; cleaned remembers targets that +// have already been fully unfolded so anchor reuse stays linear in the +// number of distinct anchors. The aliasState.created counter is checked +// against maxNodes to abort exponentially blown-up alias graphs (excessive +// aliasing) before they exhaust memory. +func unfoldAliases(n *yaml.Node, st *aliasState) error { + if n == nil { + return nil + } + for i, child := range n.Content { + if child == nil { + continue + } + if child.Kind == yaml.AliasNode { + target := child.Alias + if target == nil { + continue + } + if st.inProgress[target] { + return fmt.Errorf("cycle detected in alias chain at line %d", child.Line) + } + if !st.cleaned[target] { + st.inProgress[target] = true + if err := unfoldAliases(target, st); err != nil { + return err + } + delete(st.inProgress, target) + st.cleaned[target] = true + st.sizes[target] = countNodes(target) + } + st.created += st.sizes[target] + if st.created > st.maxNodes { + return fmt.Errorf("excessive aliasing: alias expansion exceeded %d nodes", st.maxNodes) + } + n.Content[i] = deepCopy(target) + continue + } + if err := unfoldAliases(child, st); err != nil { + return err + } + } + return nil +} + +// countNodes returns the total number of nodes reachable from n, used by +// unfoldAliases to charge each alias reuse against the expansion cap. +func countNodes(n *yaml.Node) int { + if n == nil { + return 0 + } + total := 1 + for _, c := range n.Content { + total += countNodes(c) + } + return total +} + +// deepCopy returns a structural copy of n with all nested content cloned. +// Anchor and Alias fields are cleared on the copy: the result is a plain +// concrete subtree, no longer participating in the YAML anchor graph. +// Position information (Line, Column) and Style are preserved so diagnostics +// downstream still point at the original source location, even though the +// node has been duplicated. +func deepCopy(n *yaml.Node) *yaml.Node { + if n == nil { + return nil + } + clone := &yaml.Node{ + Kind: n.Kind, + Tag: n.Tag, + Value: n.Value, + Style: n.Style, + Line: n.Line, + Column: n.Column, + HeadComment: n.HeadComment, + LineComment: n.LineComment, + FootComment: n.FootComment, + } + if len(n.Content) > 0 { + clone.Content = make([]*yaml.Node, len(n.Content)) + for i, c := range n.Content { + clone.Content[i] = deepCopy(c) + } + } + return clone +} + +// foldMergeKeys eliminates `<<` entries from every MappingNode in the tree. +// For each MappingNode, the explicit keys defined on the mapping itself take +// precedence; then, for each merge source in declaration order, any key not +// yet present is appended. A merge value can be a single mapping or a +// sequence of mappings (the YAML 1.1 merge key spec); sequence entries are +// processed in order, with earlier entries winning over later ones — the +// same semantics yaml.Decoder would apply when decoding the unfolded tree +// directly. +// +// Recursion is depth-first so that inner mappings fold their own `<<` +// entries before their parents see them. By this point in the pipeline, +// aliases have already been unfolded, so every merge value is a concrete +// mapping (or sequence of mappings) and no alias indirection remains. +func foldMergeKeys(n *yaml.Node) { + if n == nil { + return + } + for _, c := range n.Content { + foldMergeKeys(c) + } + if n.Kind != yaml.MappingNode { + return + } + + var result []*yaml.Node + var mergeSources []*yaml.Node + seen := map[string]bool{} + + for i := 0; i+1 < len(n.Content); i += 2 { + key := n.Content[i] + value := n.Content[i+1] + if key.Tag == "!!merge" || key.Value == "<<" { + switch value.Kind { + case yaml.MappingNode: + mergeSources = append(mergeSources, value) + case yaml.SequenceNode: + for _, item := range value.Content { + if item != nil && item.Kind == yaml.MappingNode { + mergeSources = append(mergeSources, item) + } + } + } + continue + } + seen[key.Value] = true + result = append(result, key, value) + } + + for _, src := range mergeSources { + for i := 0; i+1 < len(src.Content); i += 2 { + key := src.Content[i] + value := src.Content[i+1] + if seen[key.Value] { + continue + } + seen[key.Value] = true + result = append(result, key, value) + } + } + n.Content = result +} diff --git a/internal/node/aliases_fuzz_test.go b/internal/node/aliases_fuzz_test.go new file mode 100644 index 000000000..d7bb24296 --- /dev/null +++ b/internal/node/aliases_fuzz_test.go @@ -0,0 +1,72 @@ +/* + Copyright 2020 The Compose Specification Authors. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +*/ + +package node + +import ( + "testing" + + "go.yaml.in/yaml/v4" +) + +// FuzzNormalizeAliases feeds the alias unfolder arbitrary YAML +// documents and checks that the function terminates (the +// defaultMaxAliasNodes cap is the production-grade defense against +// alias bombs; the fuzz target validates that the cap is actually +// honored across the input space). +func FuzzNormalizeAliases(f *testing.F) { + corpus := []string{ + `x-a: &a {k: v} +services: + web: + image: nginx`, + `x-a: &a + k: v +x-b: &b + <<: *a + z: 1 +services: + s: + <<: *b`, + `x-a: &a [1, 2, 3] +x-b: &b [*a, *a] +x-c: [*b, *b]`, + `x-a: &a {k: v} +x-b: &b [*a, *a, *a] +x-c: &c [*b, *b, *b] +services: + svc: + image: alpine`, + ``, + } + for _, s := range corpus { + f.Add(s) + } + f.Fuzz(func(t *testing.T, src string) { + var n yaml.Node + if err := yaml.Unmarshal([]byte(src), &n); err != nil { + t.Skip() + } + if n.Kind == 0 { + t.Skip() + } + // NormalizeAliases either returns nil (bounded unfold), an + // "excessive aliasing" cap hit, or a cycle error -- all three + // are acceptable terminations. A panic would fail the fuzz + // harness automatically. + _ = NormalizeAliases(&n) + }) +} diff --git a/internal/node/aliases_test.go b/internal/node/aliases_test.go new file mode 100644 index 000000000..42129d635 --- /dev/null +++ b/internal/node/aliases_test.go @@ -0,0 +1,280 @@ +/* + Copyright 2020 The Compose Specification Authors. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +*/ + +package node + +import ( + "strings" + "testing" + + "go.yaml.in/yaml/v4" + "gotest.tools/v3/assert" + + "github.com/compose-spec/compose-go/v3/tree" +) + +func normalize(t *testing.T, src string) *yaml.Node { + t.Helper() + var doc yaml.Node + assert.NilError(t, yaml.Unmarshal([]byte(src), &doc)) + assert.NilError(t, NormalizeAliases(&doc)) + return &doc +} + +// findAlias returns true if any node in the subtree is an AliasNode. +func findAlias(n *yaml.Node) bool { + if n == nil { + return false + } + if n.Kind == yaml.AliasNode { + return true + } + for _, c := range n.Content { + if findAlias(c) { + return true + } + } + return false +} + +// findMergeKey returns true if any MappingNode in the subtree still has a +// "<<" key (which NormalizeAliases is supposed to remove). +func findMergeKey(n *yaml.Node) bool { + if n == nil { + return false + } + if n.Kind == yaml.MappingNode { + for i := 0; i+1 < len(n.Content); i += 2 { + if n.Content[i].Value == "<<" { + return true + } + } + } + for _, c := range n.Content { + if findMergeKey(c) { + return true + } + } + return false +} + +func decodeMap(t *testing.T, n *yaml.Node) map[string]any { + t.Helper() + var m map[string]any + assert.NilError(t, n.Decode(&m)) + return m +} + +func TestNormalizeAliasesUnfoldsSimpleAlias(t *testing.T) { + src := ` +defaults: &defaults + image: nginx + restart: always +services: + web: *defaults +` + root := normalize(t, src) + assert.Assert(t, !findAlias(root), "no AliasNode should remain after NormalizeAliases") + + m := decodeMap(t, root) + web := m["services"].(map[string]any)["web"].(map[string]any) + assert.Equal(t, web["image"], "nginx") + assert.Equal(t, web["restart"], "always") +} + +func TestNormalizeAliasesFoldsMergeKeyWithSurroundingWins(t *testing.T) { + src := ` +defaults: &defaults + image: nginx + restart: always +services: + web: + <<: *defaults + image: caddy +` + root := normalize(t, src) + assert.Assert(t, !findAlias(root)) + assert.Assert(t, !findMergeKey(root)) + + m := decodeMap(t, root) + web := m["services"].(map[string]any)["web"].(map[string]any) + // Surrounding mapping wins over merge source. + assert.Equal(t, web["image"], "caddy") + assert.Equal(t, web["restart"], "always") +} + +func TestNormalizeAliasesFoldsMergeKeySequence(t *testing.T) { + // YAML 1.1 merge key with a sequence value: earlier entries win over + // later ones; both lose to keys defined in the surrounding mapping. + src := ` +common: &common + image: nginx + ports: ["80:80"] +overrides: &overrides + image: caddy + restart: always +services: + web: + <<: [*common, *overrides] +` + root := normalize(t, src) + assert.Assert(t, !findAlias(root)) + assert.Assert(t, !findMergeKey(root)) + + m := decodeMap(t, root) + web := m["services"].(map[string]any)["web"].(map[string]any) + assert.Equal(t, web["image"], "nginx", "first merge source wins") + assert.Equal(t, web["restart"], "always") + ports := web["ports"].([]any) + assert.Equal(t, len(ports), 1) + assert.Equal(t, ports[0], "80:80") +} + +func TestNormalizeAliasesDeepCopiesSoMutationsAreIsolated(t *testing.T) { + src := ` +defaults: &defaults + ports: ["80:80"] +services: + web: + <<: *defaults + api: + <<: *defaults +` + root := normalize(t, src) + + // Mutate web.ports by appending to its yaml.Node Content directly. + // If web and api shared the same Node, api would see the mutation. + m := decodeMap(t, root) + web := m["services"].(map[string]any)["web"].(map[string]any) + api := m["services"].(map[string]any)["api"].(map[string]any) + assert.DeepEqual(t, web["ports"], []any{"80:80"}) + assert.DeepEqual(t, api["ports"], []any{"80:80"}) + + // Inspect the Content pointers to confirm divergence after the deep copy. + var webPorts, apiPorts *yaml.Node + for _, top := range root.Content[0].Content { // unwrap doc → root mapping + // ignore the unwrap mechanics; instead walk to find both ports + _ = top + } + _ = Walk(root, func(p tree.Path, n *yaml.Node) error { + switch p.String() { + case "services.web.ports": + webPorts = n + case "services.api.ports": + apiPorts = n + } + return nil + }) + assert.Assert(t, webPorts != nil && apiPorts != nil) + assert.Assert(t, webPorts != apiPorts, "deep copy must produce distinct Node pointers") +} + +func TestNormalizeAliasesPreservesLineForDiagnostics(t *testing.T) { + src := `defaults: &defaults + image: nginx +services: + web: *defaults +` + root := normalize(t, src) + var imageLine int + _ = Walk(root, func(p tree.Path, n *yaml.Node) error { + if p.String() == "services.web.image" { + imageLine = n.Line + } + return nil + }) + // image: nginx is on line 2 of the source; the deep copy preserves it. + assert.Equal(t, imageLine, 2) +} + +func TestNormalizeAliasesRejectsCycle(t *testing.T) { + src := ` +a: &a + loop: *a +` + var doc yaml.Node + assert.NilError(t, yaml.Unmarshal([]byte(src), &doc)) + err := NormalizeAliases(&doc) + assert.ErrorContains(t, err, "cycle detected in alias chain") +} + +func TestNormalizeAliasesHandlesAliasBomb(t *testing.T) { + // Branching factor 3, depth 10: 3^10 = ~59k logical references; without + // the `cleaned` cache the unfold would explode. With the cache, each + // anchor is unfolded once. + src := ` +x-a: &a {k: v} +x-b: &b [*a, *a, *a] +x-c: &c [*b, *b, *b] +x-d: &d [*c, *c, *c] +x-e: &e [*d, *d, *d] +x-f: &f [*e, *e, *e] +x-g: &g [*f, *f, *f] +x-h: &h [*g, *g, *g] +x-i: &i [*h, *h, *h] +x-j: &j [*i, *i, *i] +x-k: &k [*j, *j, *j] +services: + svc: + image: alpine +` + var doc yaml.Node + assert.NilError(t, yaml.Unmarshal([]byte(src), &doc)) + assert.NilError(t, NormalizeAliases(&doc)) + assert.Assert(t, !findAlias(&doc)) +} + +func TestNormalizeAliasesHandlesNullSafely(t *testing.T) { + // An empty top-level document must not panic. + var doc yaml.Node + assert.NilError(t, yaml.Unmarshal([]byte(""), &doc)) + assert.NilError(t, NormalizeAliases(&doc)) +} + +func TestNormalizeAliasesNestedAliasInsideAlias(t *testing.T) { + src := ` +inner: &inner + k: v +outer: &outer + ref: *inner +target: *outer +` + root := normalize(t, src) + assert.Assert(t, !findAlias(root)) + m := decodeMap(t, root) + target := m["target"].(map[string]any) + ref := target["ref"].(map[string]any) + assert.Equal(t, ref["k"], "v") +} + +func TestNormalizeAliasesPreservesAcrossMultipleCalls(t *testing.T) { + src := ` +common: &common {image: nginx} +services: + web: + <<: *common +` + root := normalize(t, src) + // A second call is a no-op (idempotent). + assert.NilError(t, NormalizeAliases(root)) + m := decodeMap(t, root) + web := m["services"].(map[string]any)["web"].(map[string]any) + assert.Equal(t, web["image"], "nginx") + // Sanity: no stray markers left in source. + out, err := yaml.Marshal(root) + assert.NilError(t, err) + assert.Assert(t, !strings.Contains(string(out), "<<"), "no merge key in output") +} diff --git a/internal/node/apply_reset.go b/internal/node/apply_reset.go new file mode 100644 index 000000000..6c0373f7e --- /dev/null +++ b/internal/node/apply_reset.go @@ -0,0 +1,80 @@ +/* + Copyright 2020 The Compose Specification Authors. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +*/ + +package node + +import ( + "go.yaml.in/yaml/v4" + + "github.com/compose-spec/compose-go/v3/tree" +) + +// ApplyResetPaths removes every mapping entry in root whose path matches one +// of the recorded reset paths. !override entries also feature in the list +// but their replacement semantics are handled by the merge phase (the +// override layer wins outright), so the value at the path on the right-hand +// merge tree is already correct; deleting an entry whose path matches a +// stored !override pattern is a no-op when the override layer carried a +// concrete value at that path. +// +// Sequence elements are not currently supported: !reset on an array entry +// is rejected by v3 with an explicit error rather than silently being +// applied, matching the decision recorded in the plan. +// +// ApplyResetPaths mutates root in place. Returns root for convenience. +func ApplyResetPaths(root *yaml.Node, paths []tree.Path) *yaml.Node { + if root == nil || len(paths) == 0 { + return root + } + target := root + if target.Kind == yaml.DocumentNode && len(target.Content) == 1 { + target = target.Content[0] + } + applyResetPaths(target, tree.NewPath(), paths) + return root +} + +func applyResetPaths(n *yaml.Node, p tree.Path, patterns []tree.Path) { + if n == nil { + return + } + switch n.Kind { + case yaml.MappingNode: + filtered := make([]*yaml.Node, 0, len(n.Content)) + for i := 0; i+1 < len(n.Content); i += 2 { + next := p.Next(n.Content[i].Value) + if matchesAny(next, patterns) { + continue + } + applyResetPaths(n.Content[i+1], next, patterns) + filtered = append(filtered, n.Content[i], n.Content[i+1]) + } + n.Content = filtered + case yaml.SequenceNode: + for _, c := range n.Content { + applyResetPaths(c, p.Next(tree.PathMatchList), patterns) + } + } +} + +func matchesAny(p tree.Path, patterns []tree.Path) bool { + for _, pattern := range patterns { + if p.Matches(pattern) { + return true + } + } + return false +} diff --git a/internal/node/layer.go b/internal/node/layer.go new file mode 100644 index 000000000..1c3ea8ef8 --- /dev/null +++ b/internal/node/layer.go @@ -0,0 +1,132 @@ +/* + Copyright 2020 The Compose Specification Authors. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +*/ + +// Package node holds the yaml.Node-centric building blocks used by the v3 +// loader pipeline. A Layer pairs a parsed YAML tree with the SourceContext +// that produced it, so per-node parsing context (working directory, env +// variables, source file/line) can be preserved across cross-file merges +// and applied lazily during interpolation and path resolution. +package node + +import ( + "go.yaml.in/yaml/v4" + + "github.com/compose-spec/compose-go/v3/tree" + "github.com/compose-spec/compose-go/v3/types" +) + +// SourceInline is used as SourceContext.File when a Layer is built from +// in-memory bytes with no associated filename. +const SourceInline = "(inline)" + +// SourceContext carries the parsing context attached to a YAML subtree. +// It is the unit of information needed to interpolate a scalar lazily and to +// resolve a relative path against the appropriate working directory after +// cross-file merge. +type SourceContext struct { + // File is the absolute path of the source file, or SourceInline when the + // layer was constructed from in-memory content. + File string + + // WorkingDir is the directory against which relative paths inside this + // subtree are resolved. For an included file it is the include's + // project_directory, not the project root. + WorkingDir string + + // Environment is the variable lookup table effective for this subtree. + // It is the result of merging the shell environment with any env_file + // declared by the layer's loader (top-level, include, or extends). + Environment types.Mapping + + // EnvFiles lists the env_file paths, in load order, that contributed to + // Environment. Kept for diagnostics; not consulted at lookup time. + EnvFiles []string + + // Parent points to the SourceContext that triggered loading this one + // (via include or extends). Nil for the root context. The chain enables + // "in file X included from file Y" style diagnostics. + Parent *SourceContext + + // PathsPreResolved is set to true once the include sub-load has + // absolutized every relative path scalar inside the layer's tree. The + // orchestrator path resolution pass consults this flag to skip + // re-resolving scalars that already went through the include's own + // resolution, which would otherwise double-join when the include + // project_directory was relative. + PathsPreResolved bool +} + +// Layer is a parsed YAML document paired with its SourceContext. +// +// Node is the document root as returned by yaml.Decoder (the DocumentNode +// wrapper is typically stripped before storing the inner MappingNode). The +// node retains all position information and the original Kind/Tag/Style of +// every scalar, which v3 uses both for diagnostics and to drive type +// conversion at decode time. +// +// origins is a sparse side-table mapping individual *yaml.Node values to a +// SourceContext different from the layer default. Until a cross-file merge +// rewires nodes from other layers into this tree, the map is empty and +// Origin returns the layer Context for any node. +type Layer struct { + Node *yaml.Node + Context *SourceContext + + origins map[*yaml.Node]*SourceContext + resetPaths []tree.Path +} + +// NewLayer returns a Layer that pairs node with ctx. The origins side-table +// is allocated on first SetOrigin; until then Origin returns ctx for any +// queried node. +func NewLayer(node *yaml.Node, ctx *SourceContext) *Layer { + return &Layer{Node: node, Context: ctx} +} + +// Origin returns the SourceContext governing the interpretation of n. When no +// explicit origin has been recorded for n, the layer default Context is +// returned. +func (l *Layer) Origin(n *yaml.Node) *SourceContext { + if l == nil { + return nil + } + if ctx, ok := l.origins[n]; ok { + return ctx + } + return l.Context +} + +// SetOrigin records an explicit origin for n. Used by the merge phase when a +// node from another layer is grafted into this layer's tree. +func (l *Layer) SetOrigin(n *yaml.Node, ctx *SourceContext) { + if l.origins == nil { + l.origins = make(map[*yaml.Node]*SourceContext) + } + l.origins[n] = ctx +} + +// SetResetPaths records the tree.Paths where !reset / !override tags were +// found during ResolveResetOverride. The merge phase consults this list to +// drop or replace values from base layers at those paths. +func (l *Layer) SetResetPaths(paths []tree.Path) { + l.resetPaths = paths +} + +// ResetPaths returns the list of paths recorded by SetResetPaths, in the +// order they were collected. +func (l *Layer) ResetPaths() []tree.Path { + return l.resetPaths +} diff --git a/internal/node/reset.go b/internal/node/reset.go new file mode 100644 index 000000000..daa30e386 --- /dev/null +++ b/internal/node/reset.go @@ -0,0 +1,298 @@ +/* + Copyright 2020 The Compose Specification Authors. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +*/ + +package node + +import ( + "fmt" + "strconv" + "strings" + + "go.yaml.in/yaml/v4" + + "github.com/compose-spec/compose-go/v3/tree" +) + +// DefaultMaxNodeVisits caps the total number of node visits performed by +// ResolveResetOverride per document. The value is sized to accommodate large +// real-world compose files while still rejecting documents that would cause +// unbounded traversal (alias bombs). Callers can override it by passing a +// non-zero limit to ResolveResetOverride. +const DefaultMaxNodeVisits = 100_000 + +// nodeCache stores a resolved node and the relative sub-paths within its +// subtree that carried !reset/!override tags, so cache hits at different call +// sites can replay them rather than re-traversing the shared subtree. +type nodeCache struct { + node *yaml.Node + relativePaths []tree.Path +} + +// ResolveResetOverride detects !reset and !override tags inside a yaml.Node +// tree and produces a cleaned tree together with the list of paths where one +// of those tags was found. +// +// - Nodes tagged !reset are removed from cleaned (their value contributes +// nothing to this layer) but their path is recorded so the merge phase +// can also drop any value contributed at the same path by a base layer. +// - Nodes tagged !override are kept in cleaned with their value; their path +// is recorded so the merge phase replaces (rather than merges with) any +// value from a base layer at that path. +// +// maxNodeVisits caps the total number of recursive resolution calls; pass 0 +// to use DefaultMaxNodeVisits. Exceeding the cap returns an error rather than +// silently truncating, which is the v2 alias-bomb defense. +// +// Aliases are followed once per call site through an internal cache, so a +// shared anchor used at multiple sites is traversed only once and the +// recorded !reset/!override paths are replayed at each subsequent site. +func ResolveResetOverride(root *yaml.Node, maxNodeVisits int) (*yaml.Node, []tree.Path, error) { + if maxNodeVisits <= 0 { + maxNodeVisits = DefaultMaxNodeVisits + } + r := &resolver{ + visitedNodes: make(map[*yaml.Node][]string), + resolvedNodes: make(map[*yaml.Node]nodeCache), + maxNodeVisits: maxNodeVisits, + } + // A DocumentNode is a transparent wrapper around the actual root; unwrap + // it so callers that pass the result of yaml.Unmarshal directly get the + // same behavior as the v2 path, where yaml.Decoder hands the inner node + // to UnmarshalYAML. + target := root + if target.Kind == yaml.DocumentNode && len(target.Content) == 1 { + target = target.Content[0] + } + resolved, err := r.resolveReset(target, tree.NewPath()) + if err != nil { + return nil, nil, err + } + return resolved, r.paths, nil +} + +type resolver struct { + paths []tree.Path + visitedNodes map[*yaml.Node][]string + resolvedNodes map[*yaml.Node]nodeCache + visitCount int + maxNodeVisits int +} + +func (r *resolver) resolveReset(node *yaml.Node, path tree.Path) (*yaml.Node, error) { + r.visitCount++ + if r.visitCount > r.maxNodeVisits { + return nil, fmt.Errorf("compose file exceeds maximum node visit limit (%d)", r.maxNodeVisits) + } + + pathStr := path.String() + // A merge key (`<<`) appears as a synthetic ".<<" segment in the path; the + // recorded path must elide it so downstream consumers can match it against + // the user-visible structure. + if strings.Contains(pathStr, ".<<") { + path = tree.NewPath(strings.Replace(pathStr, ".<<", "", 1)) + } + + if node.Tag == "!reset" { + r.paths = append(r.paths, path) + return nil, nil + } + if node.Tag == "!override" { + r.paths = append(r.paths, path) + return node, nil + } + + if node.Kind == yaml.AliasNode { + if err := r.checkForCycle(node.Alias, path); err != nil { + return nil, err + } + target := node.Alias + if target.Tag == "!reset" { + r.paths = append(r.paths, path) + return nil, nil + } + if target.Tag == "!override" { + r.paths = append(r.paths, path) + return target, nil + } + return r.cachedResolve(target, path) + } + + if node.Kind == yaml.SequenceNode || node.Kind == yaml.MappingNode { + return r.cachedResolve(node, path) + } + + return node, nil +} + +// cachedResolve resolves a container node (Sequence or Mapping), serving from +// cache on repeat visits so a shared anchor is only traversed once. The cache +// is keyed by the original node pointer; on a cache hit, the recorded +// relative paths are replayed under the current base path. +func (r *resolver) cachedResolve(node *yaml.Node, path tree.Path) (*yaml.Node, error) { + if cached, ok := r.resolvedNodes[node]; ok { + for _, rel := range cached.relativePaths { + r.paths = append(r.paths, joinPath(path, rel)) + } + return cached.node, nil + } + + startIdx := len(r.paths) + resolved, err := r.resolveContainer(node, path) + if err != nil { + return nil, err + } + + var relPaths []tree.Path + for _, addedPath := range r.paths[startIdx:] { + rel, err := subPath(addedPath, path) + if err != nil { + return nil, err + } + relPaths = append(relPaths, rel) + } + r.resolvedNodes[node] = nodeCache{node: resolved, relativePaths: relPaths} + return resolved, nil +} + +// resolveContainer recurses into a Sequence or Mapping node's children. +// AliasNodes are preserved as-is in the output Content so the YAML library +// can resolve them at decode time; only the resolved value is consulted for +// tag inspection. Mapping keys are checked for duplicates and the error +// carries the offending line numbers for diagnostics. +func (r *resolver) resolveContainer(node *yaml.Node, path tree.Path) (*yaml.Node, error) { + switch node.Kind { + case yaml.SequenceNode: + var nodes []*yaml.Node + for idx, v := range node.Content { + next := path.Next(strconv.Itoa(idx)) + resolved, err := r.resolveReset(v, next) + if err != nil { + return nil, err + } + if resolved == nil { + continue + } + if v.Kind == yaml.AliasNode { + nodes = append(nodes, v) + } else { + nodes = append(nodes, resolved) + } + } + node.Content = nodes + case yaml.MappingNode: + keys := map[string]int{} + var key string + var nodes []*yaml.Node + for idx, v := range node.Content { + if idx%2 == 0 { + key = v.Value + if line, seen := keys[key]; seen { + return nil, fmt.Errorf("line %d: mapping key %#v already defined at line %d", v.Line, key, line) + } + keys[key] = v.Line + } else { + resolved, err := r.resolveReset(v, path.Next(key)) + if err != nil { + return nil, err + } + if resolved == nil { + continue + } + if v.Kind == yaml.AliasNode { + nodes = append(nodes, node.Content[idx-1], v) + } else { + nodes = append(nodes, node.Content[idx-1], resolved) + } + } + } + node.Content = nodes + } + return node, nil +} + +func (r *resolver) checkForCycle(node *yaml.Node, path tree.Path) error { + paths := r.visitedNodes[node] + pathStr := path.String() + + for _, prevPath := range paths { + if pathStr == prevPath { + continue + } + // Merge keys (`<<`) are legitimate YAML merging, not a cycle. + if strings.Contains(prevPath, "<<") || strings.Contains(pathStr, "<<") { + continue + } + // Only consider it a cycle if one path is contained within the other + // and they're not in different service definitions. + if (strings.HasPrefix(pathStr, prevPath+".") || + strings.HasPrefix(prevPath, pathStr+".")) && + !areInDifferentServices(pathStr, prevPath) { + return fmt.Errorf("cycle detected: node at path %s references node at path %s", pathStr, prevPath) + } + } + + r.visitedNodes[node] = append(paths, pathStr) + return nil +} + +// areInDifferentServices returns true when both paths traverse the `services` +// top-level key but land on different service names. A shared anchor used by +// two different services is not a cycle, even if both paths share a common +// prefix below the service name. +func areInDifferentServices(path1, path2 string) bool { + parts1 := strings.Split(path1, ".") + parts2 := strings.Split(path2, ".") + for i := 0; i < len(parts1) && i < len(parts2); i++ { + if parts1[i] == "services" && i+1 < len(parts1) && + parts2[i] == "services" && i+1 < len(parts2) { + return parts1[i+1] != parts2[i+1] + } + } + return false +} + +// subPath strips base from full to produce a relative path stored in the +// cache. Returns "" when full == base (the !reset/!override tag is on the +// node root itself). Returns an error when full is not rooted at base, which +// would indicate a logic error in resolveReset/cachedResolve. +func subPath(full, base tree.Path) (tree.Path, error) { + if base == "" { + return full, nil + } + fullStr := string(full) + baseStr := string(base) + if fullStr == baseStr { + return "", nil + } + prefix := baseStr + "." + if strings.HasPrefix(fullStr, prefix) { + return tree.Path(fullStr[len(prefix):]), nil + } + return "", fmt.Errorf("internal error: path %q is not a sub-path of %q", fullStr, baseStr) +} + +// joinPath reconstructs an absolute path from a call-site base and a cached +// relative path. A relative path of "" means the tag was on the node root, so +// base is returned unchanged. +func joinPath(base, rel tree.Path) tree.Path { + if rel == "" { + return base + } + if base == "" { + return rel + } + return tree.Path(string(base) + "." + string(rel)) +} diff --git a/internal/node/reset_test.go b/internal/node/reset_test.go new file mode 100644 index 000000000..5efa7683f --- /dev/null +++ b/internal/node/reset_test.go @@ -0,0 +1,140 @@ +/* + Copyright 2020 The Compose Specification Authors. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +*/ + +package node + +import ( + "fmt" + "strings" + "testing" + + "go.yaml.in/yaml/v4" + "gotest.tools/v3/assert" +) + +func resolveYAML(t *testing.T, src string) (*yaml.Node, []string, error) { + t.Helper() + var doc yaml.Node + if err := yaml.Unmarshal([]byte(src), &doc); err != nil { + return nil, nil, err + } + resolved, paths, err := ResolveResetOverride(&doc, 0) + if err != nil { + return nil, nil, err + } + strs := make([]string, len(paths)) + for i, p := range paths { + strs[i] = p.String() + } + return resolved, strs, nil +} + +func TestResolveResetTagRemovesNode(t *testing.T) { + src := ` +services: + web: + image: nginx + command: !reset null +` + resolved, paths, err := resolveYAML(t, src) + assert.NilError(t, err) + assert.DeepEqual(t, paths, []string{"services.web.command"}) + + // Confirm command is no longer present in the resolved tree. + out, err := yaml.Marshal(resolved) + assert.NilError(t, err) + assert.Assert(t, !strings.Contains(string(out), "command"), "command should be stripped from tree, got:\n%s", out) +} + +func TestResolveOverrideTagKeepsNode(t *testing.T) { + src := ` +services: + web: + command: !override ["echo", "hi"] +` + resolved, paths, err := resolveYAML(t, src) + assert.NilError(t, err) + assert.DeepEqual(t, paths, []string{"services.web.command"}) + + out, err := yaml.Marshal(resolved) + assert.NilError(t, err) + assert.Assert(t, strings.Contains(string(out), "command"), "command must survive !override, got:\n%s", out) + assert.Assert(t, strings.Contains(string(out), "echo"), "command value preserved, got:\n%s", out) +} + +func TestResolveNoTagsReturnsEmptyPaths(t *testing.T) { + src := ` +services: + web: + image: nginx +` + _, paths, err := resolveYAML(t, src) + assert.NilError(t, err) + assert.Equal(t, len(paths), 0) +} + +func TestResolveAliasCycleRejected(t *testing.T) { + // A mapping that merges its own ancestor through `<<` creates an + // alias cycle reachable via path containment, which resolveReset + // detects. Pattern lifted from the loader-level TestResetCycle + // "direct_self_reference_cycle" case. + src := ` +name: test +x-healthcheck: &healthcheck + egress-service: + <<: *healthcheck +` + _, _, err := resolveYAML(t, src) + assert.ErrorContains(t, err, "cycle detected") +} + +func TestResolveMaxNodeVisitsExceeded(t *testing.T) { + var sb strings.Builder + sb.WriteString("name: test\nentries:\n") + for i := 0; i < 200; i++ { + fmt.Fprintf(&sb, " k%d: v\n", i) + } + var doc yaml.Node + assert.NilError(t, yaml.Unmarshal([]byte(sb.String()), &doc)) + + _, _, err := ResolveResetOverride(&doc, 50) + assert.ErrorContains(t, err, "exceeds maximum node visit limit (50)") +} + +func TestResolveSharedAnchorReplaysRelativePaths(t *testing.T) { + // A shared anchor used by two services must record !reset at both call + // sites, not just the first one. This is the regression covered by the + // loader-level TestResetTagWithSharedAlias. + src := ` +x-base: &base + command: !reset null + +services: + web: *base + api: *base +` + _, paths, err := resolveYAML(t, src) + assert.NilError(t, err) + // The anchor itself is at x-base.command; replayed paths are + // services.web.command and services.api.command. + assert.Assert(t, len(paths) >= 3, "expected at least 3 reset paths, got %v", paths) + have := map[string]bool{} + for _, p := range paths { + have[p] = true + } + assert.Assert(t, have["services.web.command"], "services.web.command should be recorded: %v", paths) + assert.Assert(t, have["services.api.command"], "services.api.command should be recorded: %v", paths) +} diff --git a/internal/node/walk.go b/internal/node/walk.go new file mode 100644 index 000000000..1fa35d5bb --- /dev/null +++ b/internal/node/walk.go @@ -0,0 +1,101 @@ +/* + Copyright 2020 The Compose Specification Authors. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +*/ + +package node + +import ( + "go.yaml.in/yaml/v4" + + "github.com/compose-spec/compose-go/v3/tree" +) + +// Visit is the callback invoked by Walk at every meaningful position in a +// yaml.Node tree. +// +// path is the dotted tree.Path leading to n; an empty Path denotes the root. +// Sequence elements are represented by the tree.PathMatchList token "[]" to +// stay consistent with the patterns used by override, paths, transform and +// validation throughout the codebase. +// +// Returning a non-nil error aborts the walk; that same error is returned by +// Walk. +type Visit func(path tree.Path, n *yaml.Node) error + +// Walk traverses a yaml.Node tree depth-first, invoking fn at every node +// reachable from root that maps to a meaningful Compose path: +// +// - the root itself, with an empty Path; +// - every value of every MappingNode, with the path extended by the key; +// - every element of every SequenceNode, with the path extended by "[]". +// +// DocumentNodes are unwrapped transparently (Walk recurses into Content +// without invoking fn for them). AliasNodes are followed once: their target is +// visited at the alias's path. Cycles between aliases are broken silently to +// avoid infinite recursion; reset / override resolution is responsible for +// reporting them. +// +// Mapping keys themselves are not visited; only their values are. Callers that +// need to inspect a key alongside its value can retrieve the key from +// n.Content[i] when visiting the parent MappingNode in a separate pass. +func Walk(root *yaml.Node, fn Visit) error { + return walk(root, tree.NewPath(), fn, map[*yaml.Node]struct{}{}) +} + +func walk(n *yaml.Node, path tree.Path, fn Visit, seen map[*yaml.Node]struct{}) error { + if n == nil { + return nil + } + if n.Kind == yaml.DocumentNode { + for _, child := range n.Content { + if err := walk(child, path, fn, seen); err != nil { + return err + } + } + return nil + } + if n.Kind == yaml.AliasNode { + target := n.Alias + if target == nil { + return nil + } + if _, cycle := seen[target]; cycle { + return nil + } + seen[target] = struct{}{} + defer delete(seen, target) + return walk(target, path, fn, seen) + } + if err := fn(path, n); err != nil { + return err + } + switch n.Kind { + case yaml.MappingNode: + for i := 0; i+1 < len(n.Content); i += 2 { + key := n.Content[i] + value := n.Content[i+1] + if err := walk(value, path.Next(key.Value), fn, seen); err != nil { + return err + } + } + case yaml.SequenceNode: + for _, child := range n.Content { + if err := walk(child, path.Next(tree.PathMatchList), fn, seen); err != nil { + return err + } + } + } + return nil +} diff --git a/internal/node/walk_test.go b/internal/node/walk_test.go new file mode 100644 index 000000000..1dcb20642 --- /dev/null +++ b/internal/node/walk_test.go @@ -0,0 +1,183 @@ +/* + Copyright 2020 The Compose Specification Authors. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +*/ + +package node + +import ( + "errors" + "testing" + + "go.yaml.in/yaml/v4" + "gotest.tools/v3/assert" + + "github.com/compose-spec/compose-go/v3/tree" +) + +func parse(t *testing.T, src string) *yaml.Node { + t.Helper() + var doc yaml.Node + assert.NilError(t, yaml.Unmarshal([]byte(src), &doc)) + return &doc +} + +func collectPaths(t *testing.T, root *yaml.Node) []string { + t.Helper() + var paths []string + err := Walk(root, func(p tree.Path, _ *yaml.Node) error { + paths = append(paths, p.String()) + return nil + }) + assert.NilError(t, err) + return paths +} + +func TestWalkFlatMapping(t *testing.T) { + root := parse(t, ` +name: demo +version: "3" +`) + got := collectPaths(t, root) + assert.DeepEqual(t, got, []string{"", "name", "version"}) +} + +func TestWalkNestedMappingAndSequence(t *testing.T) { + root := parse(t, ` +services: + web: + image: nginx + ports: + - "80:80" + - "443:443" +`) + got := collectPaths(t, root) + assert.DeepEqual(t, got, []string{ + "", + "services", + "services.web", + "services.web.image", + "services.web.ports", + "services.web.ports.[]", + "services.web.ports.[]", + }) +} + +func TestWalkUnwrapsDocumentNode(t *testing.T) { + root := parse(t, `key: value`) + assert.Equal(t, root.Kind, yaml.DocumentNode) + var rootKind yaml.Kind + err := Walk(root, func(p tree.Path, n *yaml.Node) error { + if p == "" { + rootKind = n.Kind + } + return nil + }) + assert.NilError(t, err) + assert.Equal(t, rootKind, yaml.MappingNode) +} + +func TestWalkFollowsAliasOnce(t *testing.T) { + root := parse(t, ` +defaults: &d + image: nginx + ports: + - "80:80" +services: + web: *d +`) + var webImage *yaml.Node + err := Walk(root, func(p tree.Path, n *yaml.Node) error { + if p == "services.web.image" { + webImage = n + } + return nil + }) + assert.NilError(t, err) + assert.Assert(t, webImage != nil, "alias target reachable via services.web.image") + assert.Equal(t, webImage.Value, "nginx") +} + +func TestWalkBreaksAliasCycle(t *testing.T) { + // Construct an artificial cycle: a MappingNode whose only value is an + // AliasNode pointing back to that mapping. The YAML library does not + // allow expressing this in source, so we build it by hand. + mapping := &yaml.Node{Kind: yaml.MappingNode} + key := &yaml.Node{Kind: yaml.ScalarNode, Value: "self"} + alias := &yaml.Node{Kind: yaml.AliasNode, Alias: mapping} + mapping.Content = []*yaml.Node{key, alias} + + count := 0 + err := Walk(mapping, func(_ tree.Path, _ *yaml.Node) error { + count++ + if count > 100 { + return errors.New("walk did not terminate") + } + return nil + }) + assert.NilError(t, err) +} + +func TestWalkPropagatesError(t *testing.T) { + root := parse(t, ` +services: + web: + image: nginx +`) + boom := errors.New("boom") + err := Walk(root, func(p tree.Path, _ *yaml.Node) error { + if p == "services.web.image" { + return boom + } + return nil + }) + assert.ErrorIs(t, err, boom) +} + +func TestLayerOriginDefaultsToContext(t *testing.T) { + root := parse(t, `key: value`) + ctx := &SourceContext{File: "test.yaml", WorkingDir: "/work"} + layer := NewLayer(root, ctx) + + var scalar *yaml.Node + err := Walk(root, func(p tree.Path, n *yaml.Node) error { + if p == "key" { + scalar = n + } + return nil + }) + assert.NilError(t, err) + assert.Equal(t, layer.Origin(scalar), ctx) +} + +func TestLayerSetOriginOverridesDefault(t *testing.T) { + root := parse(t, `key: value`) + defaultCtx := &SourceContext{File: "main.yaml"} + otherCtx := &SourceContext{File: "included.yaml"} + layer := NewLayer(root, defaultCtx) + + var scalar *yaml.Node + _ = Walk(root, func(p tree.Path, n *yaml.Node) error { + if p == "key" { + scalar = n + } + return nil + }) + layer.SetOrigin(scalar, otherCtx) + + assert.Equal(t, layer.Origin(scalar), otherCtx) + + other := &yaml.Node{Kind: yaml.ScalarNode, Value: "untracked"} + assert.Equal(t, layer.Origin(other), defaultCtx) +} diff --git a/interpolation/node.go b/interpolation/node.go new file mode 100644 index 000000000..8801258ba --- /dev/null +++ b/interpolation/node.go @@ -0,0 +1,133 @@ +/* + Copyright 2020 The Compose Specification Authors. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +*/ + +package interpolation + +import ( + "errors" + + "go.yaml.in/yaml/v4" + + "github.com/compose-spec/compose-go/v3/internal/node" + "github.com/compose-spec/compose-go/v3/template" + "github.com/compose-spec/compose-go/v3/tree" +) + +// NodeOptions configures InterpolateNode. +type NodeOptions struct { + // LookupValueFor returns the variable lookup function to consult for a + // given scalar node. It is invoked once per scalar visited, which is + // what enables lazy per-Layer interpolation: callers can hand back a + // different lookup based on the SourceContext attached to each scalar + // after a cross-file merge. + // + // When nil, every scalar uses LookupValue. + LookupValueFor func(*yaml.Node) LookupValue + + // LookupValue is the fall-back single lookup used when LookupValueFor is + // nil. Provided for parity with v2 callers that have only one environment + // for the entire document. + LookupValue LookupValue + + // Substitute is the template substitution function; defaults to + // template.Substitute when nil. The shape matches v2. + Substitute func(string, template.Mapping) (string, error) + + // Tags maps tree.Path patterns to a YAML tag ("!!int", "!!bool", + // "!!float", ...). After substitution, scalars whose path matches a + // pattern have their Tag updated so the eventual (*yaml.Node).Decode + // produces the right Go type — replacing the legacy mapstructure cast + // hook with native YAML decoding semantics. + Tags map[tree.Path]string +} + +// InterpolateNode walks a yaml.Node tree and substitutes ${VAR} references +// in every scalar value, using a LookupValue that may be picked per-node. +// This is the v3 interpolation phase: it runs after the cross-file merge so +// each scalar can be interpolated in the SourceContext of its layer of +// origin — the bug fix that motivates the whole refactor. +// +// Mapping keys are not interpolated (matching v2 behavior). When a scalar's +// path matches an entry in opts.Tags, its Tag is rewritten so that yaml.v4 +// converts the value to the expected target type at decode time. +// +// The tree is mutated in place. An error from the substitution function or +// from the template parser short-circuits the walk; the returned error is +// wrapped with the source path for diagnostics. +func InterpolateNode(root *yaml.Node, opts NodeOptions) error { + if opts.Substitute == nil { + opts.Substitute = template.Substitute + } + if opts.LookupValueFor == nil { + if opts.LookupValue == nil { + return errors.New("interpolation: LookupValueFor or LookupValue must be set") + } + lookup := opts.LookupValue + opts.LookupValueFor = func(*yaml.Node) LookupValue { return lookup } + } + return node.Walk(root, func(p tree.Path, n *yaml.Node) error { + if n == nil || n.Kind != yaml.ScalarNode { + return nil + } + // !!null scalars carry no substitutable content. + if n.Tag == "!!null" { + return nil + } + lookup := opts.LookupValueFor(n) + substituted, err := opts.Substitute(n.Value, template.Mapping(lookup)) + if err != nil { + return &Error{Path: p, Node: n, Cause: newPathError(p, err)} + } + n.Value = substituted + if tag, ok := tagFor(p, opts.Tags); ok { + n.Tag = tag + } + return nil + }) +} + +// Error is returned by InterpolateNode when substitution fails on a +// scalar. It carries the offending *yaml.Node and tree.Path so the +// loader can wrap it with the source file from the origins side-table +// and surface an errdefs.Diagnostic. +type Error struct { + Path tree.Path + Node *yaml.Node + Cause error +} + +func (e *Error) Error() string { + if e == nil || e.Cause == nil { + return "" + } + return e.Cause.Error() +} + +func (e *Error) Unwrap() error { + if e == nil { + return nil + } + return e.Cause +} + +func tagFor(p tree.Path, tags map[tree.Path]string) (string, bool) { + for pattern, tag := range tags { + if p.Matches(pattern) { + return tag, true + } + } + return "", false +} diff --git a/interpolation/node_fuzz_test.go b/interpolation/node_fuzz_test.go new file mode 100644 index 000000000..99253bc0d --- /dev/null +++ b/interpolation/node_fuzz_test.go @@ -0,0 +1,77 @@ +/* + Copyright 2020 The Compose Specification Authors. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +*/ + +package interpolation + +import ( + "testing" + + "go.yaml.in/yaml/v4" + + "github.com/compose-spec/compose-go/v3/template" +) + +// FuzzInterpolateNode feeds InterpolateNode arbitrary scalar templates +// with a small fixed environment. The fuzz target verifies that the +// substitution engine terminates and never panics on any input that +// the YAML parser accepts; behavioral correctness of the substitution +// is covered by the unit tests in the template package. +func FuzzInterpolateNode(f *testing.F) { + corpus := []string{ + `services: + web: + image: nginx:${TAG}`, + `services: + web: + image: nginx:${TAG:-1.0} + command: echo ${CMD:?cmd required}`, + `services: + web: + environment: + FOO: ${BAR:-fallback} + DOUBLE: $$LITERAL`, + `services: + web: + image: ${A}-${B}-${C}`, + ``, + } + for _, s := range corpus { + f.Add(s) + } + f.Fuzz(func(t *testing.T, src string) { + var n yaml.Node + if err := yaml.Unmarshal([]byte(src), &n); err != nil { + t.Skip() + } + if n.Kind == 0 { + t.Skip() + } + env := map[string]string{ + "TAG": "2.0", + "BAR": "bar", + "A": "alpha", + "B": "beta", + "C": "gamma", + } + _ = InterpolateNode(&n, NodeOptions{ + Substitute: template.Substitute, + LookupValue: func(key string) (string, bool) { + v, ok := env[key] + return v, ok + }, + }) + }) +} diff --git a/interpolation/node_test.go b/interpolation/node_test.go new file mode 100644 index 000000000..0296bd360 --- /dev/null +++ b/interpolation/node_test.go @@ -0,0 +1,207 @@ +/* + Copyright 2020 The Compose Specification Authors. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +*/ + +package interpolation + +import ( + "testing" + + "go.yaml.in/yaml/v4" + "gotest.tools/v3/assert" + + "github.com/compose-spec/compose-go/v3/tree" +) + +func parseNode(t *testing.T, src string) *yaml.Node { + t.Helper() + var doc yaml.Node + assert.NilError(t, yaml.Unmarshal([]byte(src), &doc)) + return &doc +} + +func decode(t *testing.T, n *yaml.Node) map[string]any { + t.Helper() + var m map[string]any + assert.NilError(t, n.Decode(&m)) + return m +} + +func mappingLookup(m map[string]string) LookupValue { + return func(key string) (string, bool) { v, ok := m[key]; return v, ok } +} + +func TestInterpolateNode_BasicSubstitution(t *testing.T) { + root := parseNode(t, ` +services: + web: + image: nginx:${TAG} + ports: + - "${HOST_PORT}:80" +`) + err := InterpolateNode(root, NodeOptions{ + LookupValue: mappingLookup(map[string]string{ + "TAG": "1.2.3", + "HOST_PORT": "8080", + }), + }) + assert.NilError(t, err) + m := decode(t, root) + web := m["services"].(map[string]any)["web"].(map[string]any) + assert.Equal(t, web["image"], "nginx:1.2.3") + assert.DeepEqual(t, web["ports"], []any{"8080:80"}) +} + +func TestInterpolateNode_LazyPerScalarLookup(t *testing.T) { + // The same ${TAG} variable resolves differently depending on which scalar + // we are interpolating: scalar A uses lookupA, scalar B uses lookupB. + // This is the lazy-interpolation pattern the v3 include / extends paths + // rely on to honor per-Layer SourceContext. + root := parseNode(t, ` +services: + web: + image: nginx:${TAG} + api: + image: caddy:${TAG} +`) + + lookupForWeb := mappingLookup(map[string]string{"TAG": "from-web"}) + lookupForAPI := mappingLookup(map[string]string{"TAG": "from-api"}) + + // Pre-locate the two scalars by walking the tree (starting from the + // inner mapping, after unwrapping the DocumentNode). + var webImage, apiImage *yaml.Node + var walk func(*yaml.Node, string) + walk = func(n *yaml.Node, parentKey string) { + if n.Kind == yaml.DocumentNode { + for _, c := range n.Content { + walk(c, parentKey) + } + return + } + if n.Kind == yaml.MappingNode { + for i := 0; i+1 < len(n.Content); i += 2 { + walk(n.Content[i+1], n.Content[i].Value) + } + } + if n.Kind == yaml.ScalarNode && parentKey == "image" { + switch n.Value { + case "nginx:${TAG}": + webImage = n + case "caddy:${TAG}": + apiImage = n + } + } + } + walk(root, "") + assert.Assert(t, webImage != nil && apiImage != nil) + + err := InterpolateNode(root, NodeOptions{ + LookupValueFor: func(n *yaml.Node) LookupValue { + if n == apiImage { + return lookupForAPI + } + return lookupForWeb + }, + }) + assert.NilError(t, err) + + m := decode(t, root) + assert.Equal(t, m["services"].(map[string]any)["web"].(map[string]any)["image"], "nginx:from-web") + assert.Equal(t, m["services"].(map[string]any)["api"].(map[string]any)["image"], "caddy:from-api") +} + +func TestInterpolateNode_TagApplied(t *testing.T) { + root := parseNode(t, ` +services: + web: + ports: + - target: "${PORT}" + protocol: tcp +`) + err := InterpolateNode(root, NodeOptions{ + LookupValue: mappingLookup(map[string]string{"PORT": "80"}), + Tags: map[tree.Path]string{ + "services.*.ports.[].target": "!!int", + }, + }) + assert.NilError(t, err) + + // After interpolation + tag rewrite, the scalar Value is "80" and the + // Tag is "!!int", so decoding to a struct with an int field succeeds + // natively. + type Port struct { + Target int `yaml:"target"` + Protocol string `yaml:"protocol"` + } + type WebService struct { + Ports []Port `yaml:"ports"` + } + type ServicesBlock struct { + Web WebService `yaml:"web"` + } + type Config struct { + Services ServicesBlock `yaml:"services"` + } + var c Config + assert.NilError(t, root.Decode(&c)) + assert.Equal(t, c.Services.Web.Ports[0].Target, 80) +} + +func TestInterpolateNode_MissingVariableLeavesScalar(t *testing.T) { + // template.Substitute without a strict mode leaves unmatched variables + // as empty string by default; the same behavior is preserved here. + root := parseNode(t, ` +key: value-${MISSING} +`) + err := InterpolateNode(root, NodeOptions{ + LookupValue: mappingLookup(map[string]string{}), + }) + assert.NilError(t, err) + m := decode(t, root) + assert.Equal(t, m["key"], "value-") +} + +func TestInterpolateNode_NullScalarSkipped(t *testing.T) { + root := parseNode(t, ` +key: ~ +other: ${VAL} +`) + err := InterpolateNode(root, NodeOptions{ + LookupValue: mappingLookup(map[string]string{"VAL": "hello"}), + }) + assert.NilError(t, err) + m := decode(t, root) + assert.Assert(t, m["key"] == nil) + assert.Equal(t, m["other"], "hello") +} + +func TestInterpolateNode_PreservesStyle(t *testing.T) { + // A double-quoted scalar must stay double-quoted in the marshaled output. + root := parseNode(t, `key: "value-${VAR}"`) + err := InterpolateNode(root, NodeOptions{ + LookupValue: mappingLookup(map[string]string{"VAR": "x"}), + }) + assert.NilError(t, err) + out, err := yaml.Marshal(root) + assert.NilError(t, err) + assert.Equal(t, string(out), "key: \"value-x\"\n") +} + +func TestInterpolateNode_NoLookupReturnsError(t *testing.T) { + root := parseNode(t, `key: value`) + err := InterpolateNode(root, NodeOptions{}) + assert.ErrorContains(t, err, "LookupValueFor or LookupValue must be set") +} diff --git a/loader/diagnostics_test.go b/loader/diagnostics_test.go new file mode 100644 index 000000000..21fa22c66 --- /dev/null +++ b/loader/diagnostics_test.go @@ -0,0 +1,364 @@ +/* + Copyright 2020 The Compose Specification Authors. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +*/ + +package loader + +import ( + "context" + "errors" + "path/filepath" + "strings" + "testing" + + "github.com/compose-spec/compose-go/v3/errdefs" + "github.com/compose-spec/compose-go/v3/types" + "gotest.tools/v3/assert" +) + +// TestDiagnostic_ProjectSourcesOptIn confirms that the WithDiagnostics +// option populates *Project.Sources with the source Location of every +// reachable compose path, so downstream tooling can resolve a path +// (e.g. "services.web.image") to its file:line:column. +func TestDiagnostic_ProjectSourcesOptIn(t *testing.T) { + dir := t.TempDir() + writeFile(t, dir, "compose.yaml", ` +services: + web: + image: nginx +`) + + withDiag := func(opts *Options) { + opts.SetProjectName("diag-sources", true) + WithDiagnostics(opts) + } + p, err := LoadWithContext(context.TODO(), types.ConfigDetails{ + WorkingDir: dir, + ConfigFiles: []types.ConfigFile{{ + Filename: filepath.Join(dir, "compose.yaml"), + }}, + Environment: map[string]string{}, + }, withDiag) + + assert.NilError(t, err) + assert.Assert(t, p.Sources != nil, "Project.Sources should be populated") + + imgLoc, ok := p.Sources["services.web.image"] + assert.Assert(t, ok, "expected services.web.image in Sources, have %v", + p.Sources) + assert.Equal(t, imgLoc.File, filepath.Join(dir, "compose.yaml")) + assert.Assert(t, imgLoc.Line > 0, "Line should be > 0, got %d", imgLoc.Line) + assert.Assert(t, imgLoc.Column > 0, "Column should be > 0, got %d", imgLoc.Column) +} + +// TestDiagnostic_ProjectSourcesDefaultOff confirms that without +// WithDiagnostics, Project.Sources stays nil so the project shape is +// unchanged for callers that did not opt in. +func TestDiagnostic_ProjectSourcesDefaultOff(t *testing.T) { + dir := t.TempDir() + writeFile(t, dir, "compose.yaml", ` +services: + web: + image: nginx +`) + + p, err := LoadWithContext(context.TODO(), types.ConfigDetails{ + WorkingDir: dir, + ConfigFiles: []types.ConfigFile{{ + Filename: filepath.Join(dir, "compose.yaml"), + }}, + Environment: map[string]string{}, + }, withProjectName("diag-off", true)) + + assert.NilError(t, err) + assert.Assert(t, p.Sources == nil, + "Project.Sources should be nil without WithDiagnostics, got %v", p.Sources) +} + +// TestDiagnostic_IncludeMustBeAList confirms that an `include:` value +// that isn't a sequence surfaces with the file / line / column of the +// offending node. +func TestDiagnostic_IncludeMustBeAList(t *testing.T) { + dir := t.TempDir() + writeFile(t, dir, "compose.yaml", ` +include: + path: other.yaml +services: + foo: + image: alpine +`) + + _, err := LoadWithContext(context.TODO(), types.ConfigDetails{ + WorkingDir: dir, + ConfigFiles: []types.ConfigFile{{ + Filename: filepath.Join(dir, "compose.yaml"), + }}, + Environment: map[string]string{}, + }, withProjectName("diag-include-list", true)) + + var diag *errdefs.Diagnostic + assert.Assert(t, errors.As(err, &diag), + "expected *errdefs.Diagnostic, got %T: %v", err, err) + assert.Equal(t, diag.File, filepath.Join(dir, "compose.yaml")) + assert.Equal(t, diag.Path, "include") + assert.Assert(t, diag.Line > 0, "Line must be set, got %d", diag.Line) + assert.Assert(t, strings.Contains(diag.Cause.Error(), + "`include` must be a list"), + "unexpected cause: %v", diag.Cause) +} + +// TestDiagnostic_IncludeCycleHasFile confirms that a self-including +// compose file surfaces an "include cycle detected" diagnostic whose +// File points at the offending source. +func TestDiagnostic_IncludeCycleHasFile(t *testing.T) { + dir := t.TempDir() + writeFile(t, dir, "compose.yaml", ` +include: + - compose.yaml +services: + foo: + image: alpine +`) + target := filepath.Join(dir, "compose.yaml") + + _, err := LoadWithContext(context.TODO(), types.ConfigDetails{ + WorkingDir: dir, + ConfigFiles: []types.ConfigFile{{ + Filename: target, + }}, + Environment: map[string]string{}, + }, withProjectName("diag-cycle", true)) + + var diag *errdefs.Diagnostic + assert.Assert(t, errors.As(err, &diag), + "expected *errdefs.Diagnostic, got %T: %v", err, err) + assert.Equal(t, diag.File, target) + assert.Assert(t, strings.Contains(diag.Cause.Error(), + "include cycle detected"), + "unexpected cause: %v", diag.Cause) +} + +// TestDiagnostic_ExtendsServiceNotFound confirms that an extends.file +// pointing at a service the base file does not declare surfaces with +// the file / line / column of the extends node on the derived service. +func TestDiagnostic_ExtendsServiceNotFound(t *testing.T) { + dir := t.TempDir() + writeFile(t, dir, "base.yaml", ` +services: + other: + image: alpine +`) + writeFile(t, dir, "compose.yaml", ` +services: + derived: + extends: + file: base.yaml + service: ghost +`) + + _, err := LoadWithContext(context.TODO(), types.ConfigDetails{ + WorkingDir: dir, + ConfigFiles: []types.ConfigFile{{ + Filename: filepath.Join(dir, "compose.yaml"), + }}, + Environment: map[string]string{}, + }, withProjectName("diag-extends-missing", true)) + + var diag *errdefs.Diagnostic + assert.Assert(t, errors.As(err, &diag), + "expected *errdefs.Diagnostic, got %T: %v", err, err) + assert.Equal(t, diag.File, filepath.Join(dir, "compose.yaml")) + assert.Equal(t, diag.Path, "services.derived.extends") + assert.Assert(t, diag.Line > 0, "Line must be set, got %d", diag.Line) + assert.Assert(t, strings.Contains(diag.Cause.Error(), + `service "ghost" not found`), + "unexpected cause: %v", diag.Cause) +} + +// TestDiagnostic_ExtendsMissingService confirms that an extends mapping +// without the required `service` key surfaces with the position of the +// extends node. +func TestDiagnostic_ExtendsMissingService(t *testing.T) { + dir := t.TempDir() + writeFile(t, dir, "compose.yaml", ` +services: + derived: + extends: + file: base.yaml +`) + + _, err := LoadWithContext(context.TODO(), types.ConfigDetails{ + WorkingDir: dir, + ConfigFiles: []types.ConfigFile{{ + Filename: filepath.Join(dir, "compose.yaml"), + }}, + Environment: map[string]string{}, + }, withProjectName("diag-extends-noservice", true)) + + var diag *errdefs.Diagnostic + assert.Assert(t, errors.As(err, &diag), + "expected *errdefs.Diagnostic, got %T: %v", err, err) + assert.Equal(t, diag.File, filepath.Join(dir, "compose.yaml")) + assert.Equal(t, diag.Path, "services.derived.extends") + assert.Assert(t, diag.Line > 0, "Line must be set, got %d", diag.Line) + assert.Assert(t, strings.Contains(diag.Cause.Error(), + "extends.derived.service is required"), + "unexpected cause: %v", diag.Cause) +} + +// TestDiagnostic_ValidationKeepsPositionAcrossCanonical confirms that +// CanonicalNode's node-level walker preserves Line / Column on every +// node it does not actually reshape, so a post-canonical +// compose-rule validation failure still points at the line and column +// the user wrote (rather than zero, which the full-tree decode/encode +// bridge used to produce). +func TestDiagnostic_ValidationKeepsPositionAcrossCanonical(t *testing.T) { + dir := t.TempDir() + src := ` +services: + foo: + image: alpine +configs: + bad: + file: /tmp/cfg + environment: VAR +` + writeFile(t, dir, "compose.yaml", src) + + _, err := LoadWithContext(context.TODO(), types.ConfigDetails{ + WorkingDir: dir, + ConfigFiles: []types.ConfigFile{{ + Filename: filepath.Join(dir, "compose.yaml"), + }}, + Environment: map[string]string{}, + }, withProjectName("diag-canonical", true)) + + var diag *errdefs.Diagnostic + assert.Assert(t, errors.As(err, &diag), + "expected *errdefs.Diagnostic, got %T: %v", err, err) + assert.Equal(t, diag.Path, "configs.bad") + assert.Assert(t, diag.Line > 0, + "Line must survive CanonicalNode walk, got %d", diag.Line) + assert.Assert(t, diag.Column > 0, + "Column must survive CanonicalNode walk, got %d", diag.Column) +} + +// TestDiagnostic_InterpolationStrictModeIncludesFileLineColumn confirms +// that a strict-mode unset variable surfaces with the file, line and +// column of the offending scalar. +func TestDiagnostic_InterpolationStrictModeIncludesFileLineColumn(t *testing.T) { + dir := t.TempDir() + src := ` +services: + web: + image: nginx:${MISSING:?must be set} +` + writeFile(t, dir, "compose.yaml", src) + + _, err := LoadWithContext(context.TODO(), types.ConfigDetails{ + WorkingDir: dir, + ConfigFiles: []types.ConfigFile{{ + Filename: filepath.Join(dir, "compose.yaml"), + }}, + Environment: map[string]string{}, + }, withProjectName("diag-interp", true)) + + assert.Assert(t, err != nil, "expected interpolation error") + + var diag *errdefs.Diagnostic + assert.Assert(t, errors.As(err, &diag), + "expected *errdefs.Diagnostic, got %T: %v", err, err) + assert.Equal(t, diag.File, filepath.Join(dir, "compose.yaml")) + assert.Assert(t, diag.Line > 0, "Line must be set, got %d", diag.Line) + assert.Equal(t, diag.Path, "services.web.image") + assert.Assert(t, strings.Contains(diag.Cause.Error(), "must be set"), + "unexpected cause: %v", diag.Cause) +} + +// TestDiagnostic_SchemaErrorIncludesFileLineColumn confirms that a +// JSON Schema failure surfaces the file, line and column the user +// wrote, via *errdefs.Diagnostic. +func TestDiagnostic_SchemaErrorIncludesFileLineColumn(t *testing.T) { + dir := t.TempDir() + src := ` +services: + bad: + image: 42 +` + writeFile(t, dir, "compose.yaml", src) + + _, err := LoadWithContext(context.TODO(), types.ConfigDetails{ + WorkingDir: dir, + ConfigFiles: []types.ConfigFile{{ + Filename: filepath.Join(dir, "compose.yaml"), + }}, + Environment: map[string]string{}, + }, withProjectName("diag-schema", true)) + + assert.Assert(t, err != nil, "expected schema error") + + var diag *errdefs.Diagnostic + assert.Assert(t, errors.As(err, &diag), + "expected *errdefs.Diagnostic, got %T: %v", err, err) + assert.Equal(t, diag.File, filepath.Join(dir, "compose.yaml")) + assert.Assert(t, diag.Line > 0, "Line must be set, got %d", diag.Line) + assert.Assert(t, strings.HasPrefix(diag.Path, "services.bad"), + "path should target the offending value, got %q", diag.Path) +} + +// TestDiagnostic_ValidateNodeIncludesFileLineColumn confirms that a +// validation error surfaces the source file, line and column of the +// offending node alongside the failure reason, via *errdefs.Diagnostic. +func TestDiagnostic_ValidateNodeIncludesFileLineColumn(t *testing.T) { + dir := t.TempDir() + src := ` +services: + foo: + image: alpine +secrets: + bad: + file: /tmp/secret + environment: VAR +` + writeFile(t, dir, "compose.yaml", src) + + _, err := LoadWithContext(context.TODO(), types.ConfigDetails{ + WorkingDir: dir, + ConfigFiles: []types.ConfigFile{{ + Filename: filepath.Join(dir, "compose.yaml"), + }}, + Environment: map[string]string{}, + }, withProjectName("diag-test", true)) + + assert.Assert(t, err != nil, "expected validation error") + + var diag *errdefs.Diagnostic + assert.Assert(t, errors.As(err, &diag), + "expected *errdefs.Diagnostic, got %T: %v", err, err) + + assert.Equal(t, diag.File, filepath.Join(dir, "compose.yaml")) + assert.Assert(t, diag.Line > 0, "Line must be set, got %d", diag.Line) + assert.Assert(t, diag.Column > 0, "Column must be set, got %d", diag.Column) + assert.Equal(t, diag.Path, "secrets.bad") + assert.Assert(t, strings.Contains(diag.Cause.Error(), + "file|environment attributes are mutually exclusive"), + "unexpected cause: %v", diag.Cause) + // Rendered form: file:line:col: path: cause + rendered := diag.Error() + assert.Assert(t, strings.HasPrefix(rendered, diag.File+":"), + "diagnostic should start with file: %q", rendered) + assert.Assert(t, strings.Contains(rendered, "secrets.bad"), + "diagnostic should include path: %q", rendered) +} diff --git a/loader/environment.go b/loader/environment.go deleted file mode 100644 index a250e7757..000000000 --- a/loader/environment.go +++ /dev/null @@ -1,110 +0,0 @@ -/* - Copyright 2020 The Compose Specification Authors. - - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. -*/ - -package loader - -import ( - "fmt" - - "github.com/compose-spec/compose-go/v3/types" -) - -// ResolveEnvironment update the environment variables for the format {- VAR} (without interpolation) -func ResolveEnvironment(dict map[string]any, environment types.Mapping) { - resolveServicesEnvironment(dict, environment) - resolveSecretsEnvironment(dict, environment) - resolveConfigsEnvironment(dict, environment) -} - -func resolveServicesEnvironment(dict map[string]any, environment types.Mapping) { - services, ok := dict["services"].(map[string]any) - if !ok { - return - } - - for service, cfg := range services { - serviceConfig, ok := cfg.(map[string]any) - if !ok { - continue - } - serviceEnv, ok := serviceConfig["environment"].([]any) - if !ok { - continue - } - envs := []any{} - for _, env := range serviceEnv { - varEnv, ok := env.(string) - if !ok { - continue - } - if found, ok := environment[varEnv]; ok { - envs = append(envs, fmt.Sprintf("%s=%s", varEnv, found)) - } else { - // either does not exist or it was already resolved in interpolation - envs = append(envs, varEnv) - } - } - serviceConfig["environment"] = envs - services[service] = serviceConfig - } - dict["services"] = services -} - -func resolveSecretsEnvironment(dict map[string]any, environment types.Mapping) { - secrets, ok := dict["secrets"].(map[string]any) - if !ok { - return - } - - for name, cfg := range secrets { - secret, ok := cfg.(map[string]any) - if !ok { - continue - } - env, ok := secret["environment"].(string) - if !ok { - continue - } - if found, ok := environment[env]; ok { - secret[types.SecretConfigXValue] = found - } - secrets[name] = secret - } - dict["secrets"] = secrets -} - -func resolveConfigsEnvironment(dict map[string]any, environment types.Mapping) { - configs, ok := dict["configs"].(map[string]any) - if !ok { - return - } - - for name, cfg := range configs { - config, ok := cfg.(map[string]any) - if !ok { - continue - } - env, ok := config["environment"].(string) - if !ok { - continue - } - if found, ok := environment[env]; ok { - config["content"] = found - } - configs[name] = config - } - dict["configs"] = configs -} diff --git a/loader/extends.go b/loader/extends.go deleted file mode 100644 index 290a29d91..000000000 --- a/loader/extends.go +++ /dev/null @@ -1,221 +0,0 @@ -/* - Copyright 2020 The Compose Specification Authors. - - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. -*/ - -package loader - -import ( - "context" - "fmt" - "path/filepath" - - "github.com/compose-spec/compose-go/v3/consts" - "github.com/compose-spec/compose-go/v3/override" - "github.com/compose-spec/compose-go/v3/paths" - "github.com/compose-spec/compose-go/v3/types" -) - -func ApplyExtends(ctx context.Context, dict map[string]any, opts *Options, tracker *cycleTracker, post PostProcessor) error { - a, ok := dict["services"] - if !ok { - return nil - } - services, ok := a.(map[string]any) - if !ok { - return fmt.Errorf("services must be a mapping") - } - for name := range services { - merged, err := applyServiceExtends(ctx, name, services, opts, tracker, post) - if err != nil { - return err - } - services[name] = merged - } - dict["services"] = services - return nil -} - -func applyServiceExtends(ctx context.Context, name string, services map[string]any, opts *Options, tracker *cycleTracker, post PostProcessor) (any, error) { - s := services[name] - if s == nil { - return nil, nil - } - service, ok := s.(map[string]any) - if !ok { - return nil, fmt.Errorf("services.%s must be a mapping", name) - } - extends, ok := service["extends"] - if !ok { - return s, nil - } - filename := ctx.Value(consts.ComposeFileKey{}).(string) - var ( - err error - ref string - file any - ) - switch v := extends.(type) { - case map[string]any: - ref, ok = v["service"].(string) - if !ok { - return nil, fmt.Errorf("extends.%s.service is required", name) - } - file = v["file"] - opts.ProcessEvent("extends", v) - case string: - ref = v - opts.ProcessEvent("extends", map[string]any{"service": ref}) - } - - var ( - base any - processor = post - ) - - if file != nil { - refFilename := file.(string) - services, processor, err = getExtendsBaseFromFile(ctx, name, ref, filename, refFilename, opts, tracker) - if err != nil { - return nil, err - } - filename = refFilename - } else { - _, ok := services[ref] - if !ok { - return nil, fmt.Errorf("cannot extend service %q in %s: service %q not found", name, filename, ref) - } - } - - tracker, err = tracker.Add(filename, name) - if err != nil { - return nil, err - } - - // recursively apply `extends` - base, err = applyServiceExtends(ctx, ref, services, opts, tracker, processor) - if err != nil { - return nil, err - } - - if base == nil { - return service, nil - } - source := deepClone(base).(map[string]any) - - err = post.Apply(map[string]any{ - "services": map[string]any{ - name: source, - }, - }) - if err != nil { - return nil, err - } - - merged, err := override.ExtendService(source, service) - if err != nil { - return nil, err - } - - delete(merged, "extends") - services[name] = merged - return merged, nil -} - -func getExtendsBaseFromFile( - ctx context.Context, - name, ref string, - path, refPath string, - opts *Options, - ct *cycleTracker, -) (map[string]any, PostProcessor, error) { - for _, loader := range opts.ResourceLoaders { - if !loader.Accept(refPath) { - continue - } - local, err := loader.Load(ctx, refPath) - if err != nil { - return nil, nil, err - } - localdir := filepath.Dir(local) - relworkingdir := loader.Dir(refPath) - - extendsOpts := opts.clone() - // replace localResourceLoader with a new flavour, using extended file base path - extendsOpts.ResourceLoaders = append(opts.RemoteResourceLoaders(), localResourceLoader{ - WorkingDir: localdir, - }) - extendsOpts.ResolvePaths = false // we do relative path resolution after file has been loaded - extendsOpts.SkipNormalization = true - extendsOpts.SkipConsistencyCheck = true - extendsOpts.SkipInclude = true - extendsOpts.SkipExtends = true // we manage extends recursively based on raw service definition - extendsOpts.SkipValidation = true // we validate the merge result - extendsOpts.SkipDefaultValues = true - source, processor, err := loadYamlFile(ctx, types.ConfigFile{Filename: local}, - extendsOpts, relworkingdir, nil, ct, map[string]any{}, nil) - if err != nil { - return nil, nil, err - } - m, ok := source["services"] - if !ok { - return nil, nil, fmt.Errorf("cannot extend service %q in %s: no services section", name, local) - } - services, ok := m.(map[string]any) - if !ok { - return nil, nil, fmt.Errorf("cannot extend service %q in %s: services must be a mapping", name, local) - } - _, ok = services[ref] - if !ok { - return nil, nil, fmt.Errorf( - "cannot extend service %q in %s: service %q not found in %s", - name, - path, - ref, - refPath, - ) - } - - var remotes []paths.RemoteResource - for _, loader := range opts.RemoteResourceLoaders() { - remotes = append(remotes, loader.Accept) - } - err = paths.ResolveRelativePaths(source, relworkingdir, remotes) - if err != nil { - return nil, nil, err - } - - return services, processor, nil - } - return nil, nil, fmt.Errorf("cannot read %s", refPath) -} - -func deepClone(value any) any { - switch v := value.(type) { - case []any: - cp := make([]any, len(v)) - for i, e := range v { - cp[i] = deepClone(e) - } - return cp - case map[string]any: - cp := make(map[string]any, len(v)) - for k, e := range v { - cp[k] = deepClone(e) - } - return cp - default: - return value - } -} diff --git a/loader/extends_test.go b/loader/extends_test.go index a2c4ce8ad..d4b25523c 100644 --- a/loader/extends_test.go +++ b/loader/extends_test.go @@ -208,7 +208,7 @@ services: options.ResolvePaths = false options.SkipValidation = true }) - assert.Error(t, err, "extends.test.service is required") + assert.Error(t, err, "(inline):7:7: services.test.extends: extends.test.service is required") } func TestIncludeWithExtends(t *testing.T) { diff --git a/loader/fix.go b/loader/fix.go deleted file mode 100644 index 7a6e88d81..000000000 --- a/loader/fix.go +++ /dev/null @@ -1,36 +0,0 @@ -/* - Copyright 2020 The Compose Specification Authors. - - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. -*/ - -package loader - -// fixEmptyNotNull is a workaround for https://github.com/xeipuuv/gojsonschema/issues/141 -// as go-yaml `[]` will load as a `[]any(nil)`, which is not the same as an empty array -func fixEmptyNotNull(value any) interface{} { - switch v := value.(type) { - case []any: - if v == nil { - return []any{} - } - for i, e := range v { - v[i] = fixEmptyNotNull(e) - } - case map[string]any: - for k, e := range v { - v[k] = fixEmptyNotNull(e) - } - } - return value -} diff --git a/loader/include.go b/loader/include.go deleted file mode 100644 index 8158bb882..000000000 --- a/loader/include.go +++ /dev/null @@ -1,223 +0,0 @@ -/* - Copyright 2020 The Compose Specification Authors. - - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. -*/ - -package loader - -import ( - "context" - "fmt" - "os" - "path/filepath" - "strings" - - "github.com/compose-spec/compose-go/v3/dotenv" - interp "github.com/compose-spec/compose-go/v3/interpolation" - "github.com/compose-spec/compose-go/v3/override" - "github.com/compose-spec/compose-go/v3/tree" - "github.com/compose-spec/compose-go/v3/types" -) - -// loadIncludeConfig parse the required config from raw yaml -func loadIncludeConfig(source any) ([]types.IncludeConfig, error) { - if source == nil { - return nil, nil - } - configs, ok := source.([]any) - if !ok { - return nil, fmt.Errorf("`include` must be a list, got %s", source) - } - for i, config := range configs { - if v, ok := config.(string); ok { - configs[i] = map[string]any{ - "path": v, - } - } - } - var requires []types.IncludeConfig - err := Transform(source, &requires) - return requires, err -} - -func ApplyInclude(ctx context.Context, workingDir string, environment types.Mapping, model map[string]any, options *Options, included []string, processor PostProcessor) error { - includeConfig, err := loadIncludeConfig(model["include"]) - if err != nil { - return err - } - - for _, r := range includeConfig { - for _, listener := range options.Listeners { - listener("include", map[string]any{ - "path": r.Path, - "workingdir": workingDir, - }) - } - - var relworkingdir string - for i, p := range r.Path { - for _, loader := range options.ResourceLoaders { - if !loader.Accept(p) { - continue - } - path, err := loader.Load(ctx, p) - if err != nil { - return err - } - p = path - - if i == 0 { // This is the "main" file, used to define project-directory. Others are overrides - - switch { - case r.ProjectDirectory == "": - relworkingdir = loader.Dir(path) - r.ProjectDirectory = filepath.Dir(path) - case !filepath.IsAbs(r.ProjectDirectory): - relworkingdir = loader.Dir(r.ProjectDirectory) - r.ProjectDirectory = filepath.Join(workingDir, r.ProjectDirectory) - - default: - relworkingdir = r.ProjectDirectory - - } - for _, f := range included { - if f == path { - included = append(included, path) - return fmt.Errorf("include cycle detected:\n%s\n include %s", included[0], strings.Join(included[1:], "\n include ")) - } - } - } - } - r.Path[i] = p - } - - loadOptions := options.clone() - loadOptions.ResolvePaths = true - loadOptions.SkipNormalization = true - loadOptions.SkipConsistencyCheck = true - loadOptions.ResourceLoaders = append(loadOptions.RemoteResourceLoaders(), localResourceLoader{ - WorkingDir: r.ProjectDirectory, - }) - - if len(r.EnvFile) == 0 { - f := filepath.Join(r.ProjectDirectory, ".env") - if s, err := os.Stat(f); err == nil && !s.IsDir() { - r.EnvFile = types.StringList{f} - } - } else { - envFile := []string{} - for _, f := range r.EnvFile { - if f == "/dev/null" { - continue - } - if !filepath.IsAbs(f) { - f = filepath.Join(workingDir, f) - s, err := os.Stat(f) - if err != nil { - return err - } - if s.IsDir() { - return fmt.Errorf("%s is not a file", f) - } - } - envFile = append(envFile, f) - } - r.EnvFile = envFile - } - - envFromFile, err := dotenv.GetEnvFromFile(environment, r.EnvFile) - if err != nil { - return err - } - - config := types.ConfigDetails{ - WorkingDir: relworkingdir, - ConfigFiles: types.ToConfigFiles(r.Path), - Environment: environment.Clone().Merge(envFromFile), - } - loadOptions.Interpolate = &interp.Options{ - Substitute: options.Interpolate.Substitute, - LookupValue: config.LookupEnv, - TypeCastMapping: options.Interpolate.TypeCastMapping, - } - imported, err := loadYamlModel(ctx, config, loadOptions, &cycleTracker{}, included) - if err != nil { - return err - } - err = importResources(imported, model, processor) - if err != nil { - return err - } - } - delete(model, "include") - return nil -} - -// importResources import into model all resources defined by imported, and report error on conflict -func importResources(source map[string]any, target map[string]any, processor PostProcessor) error { - if err := importResource(source, target, "services", processor); err != nil { - return err - } - if err := importResource(source, target, "volumes", processor); err != nil { - return err - } - if err := importResource(source, target, "networks", processor); err != nil { - return err - } - if err := importResource(source, target, "secrets", processor); err != nil { - return err - } - if err := importResource(source, target, "configs", processor); err != nil { - return err - } - if err := importResource(source, target, "models", processor); err != nil { - return err - } - return nil -} - -func importResource(source map[string]any, target map[string]any, key string, processor PostProcessor) error { - from := source[key] - if from != nil { - var to map[string]any - if v, ok := target[key]; ok { - to = v.(map[string]any) - } else { - to = map[string]any{} - } - for name, a := range from.(map[string]any) { - conflict, ok := to[name] - if !ok { - to[name] = a - continue - } - err := processor.Apply(map[string]any{ - key: map[string]any{ - name: a, - }, - }) - if err != nil { - return err - } - - merged, err := override.MergeYaml(a, conflict, tree.NewPath(key, name)) - if err != nil { - return err - } - to[name] = merged - } - target[key] = to - } - return nil -} diff --git a/loader/include_test.go b/loader/include_test.go index 217cea627..04e238dc3 100644 --- a/loader/include_test.go +++ b/loader/include_test.go @@ -243,3 +243,70 @@ func createFileSubDir(t *testing.T, rootDir, subDir, content, fileName string) { path := filepath.Join(subDirPath, fileName) assert.NilError(t, os.WriteFile(path, []byte(content), 0o600)) } + +// TestInclude_EnvFile_ProvidesContextToServiceEnvFile asserts that each +// env_file entry is interpolated with the environment of the file that +// declared it: +// +// - extra.env is declared inside the included sub/compose.yaml; its content +// `FOO=$BAR` resolves against include.env_file (BAR=bar), yielding FOO=bar. +// - override.env is declared in the top-level compose.yaml as an override of +// the included `app` service; its content `OVR=${BAR:-fallback}` is +// interpolated in the top-level scope, where BAR is not defined, so the +// default value is selected (OVR=fallback). +func TestInclude_EnvFile_ProvidesContextToServiceEnvFile(t *testing.T) { + workdir, err := filepath.Abs("testdata/include/env_file") + assert.NilError(t, err) + topPath := filepath.Join(workdir, "compose.yaml") + + p, err := LoadWithContext(context.TODO(), types.ConfigDetails{ + WorkingDir: workdir, + ConfigFiles: []types.ConfigFile{{Filename: topPath}}, + Environment: map[string]string{}, + }, withProjectName("test-include-envfile-context", true)) + assert.NilError(t, err) + + resolved, err := p.WithServicesEnvironmentResolved(false) + assert.NilError(t, err) + + app := resolved.Services["app"] + + foo, ok := app.Environment["FOO"] + assert.Check(t, ok, "FOO should be present in resolved environment") + if ok && foo != nil { + assert.Check(t, *foo == "bar", "FOO should be 'bar' (from include.env_file BAR), got %q", *foo) + } + + ovr, ok := app.Environment["OVR"] + assert.Check(t, ok, "OVR should be present in resolved environment") + if ok && ovr != nil { + assert.Check(t, *ovr == "fallback", "OVR should be 'fallback' (BAR is not visible in top-level scope), got %q", *ovr) + } +} + +// TestInclude_SecretEnvironment_ProvidesContextToSecret asserts that a +// secret declared inside an included file resolves its `environment:` +// variable against the env_file declared on the include block, not the +// parent project environment. Fix for the v2 limitation where +// resolveSecretsEnvironment only looked at the project-wide environment +// and therefore could not see a variable that an include env_file +// introduced inside the include scope. +func TestInclude_SecretEnvironment_ProvidesContextToSecret(t *testing.T) { + workdir, err := filepath.Abs("testdata/include/secret_env") + assert.NilError(t, err) + topPath := filepath.Join(workdir, "compose.yaml") + + p, err := LoadWithContext(context.TODO(), types.ConfigDetails{ + WorkingDir: workdir, + ConfigFiles: []types.ConfigFile{{Filename: topPath}}, + Environment: map[string]string{}, + }, withProjectName("test-include-secret-env", true)) + assert.NilError(t, err) + + secret, ok := p.Secrets["scoped"] + assert.Assert(t, ok, "secret 'scoped' should be present") + assert.Equal(t, secret.Environment, "MY_SECRET", + "secret keeps the environment variable name it was declared with") + assert.Equal(t, secret.Content, "shadoks", + "secret content resolves against include env_file MY_SECRET, not parent env") +} diff --git a/loader/load.go b/loader/load.go new file mode 100644 index 000000000..38db4bd68 --- /dev/null +++ b/loader/load.go @@ -0,0 +1,1062 @@ +/* + Copyright 2020 The Compose Specification Authors. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +*/ + +package loader + +import ( + "context" + "errors" + "fmt" + "path/filepath" + "strings" + + "go.yaml.in/yaml/v4" + + "github.com/compose-spec/compose-go/v3/errdefs" + "github.com/compose-spec/compose-go/v3/internal/node" + interp "github.com/compose-spec/compose-go/v3/interpolation" + "github.com/compose-spec/compose-go/v3/override" + "github.com/compose-spec/compose-go/v3/paths" + "github.com/compose-spec/compose-go/v3/schema" + "github.com/compose-spec/compose-go/v3/template" + "github.com/compose-spec/compose-go/v3/transform" + "github.com/compose-spec/compose-go/v3/tree" + "github.com/compose-spec/compose-go/v3/types" + "github.com/compose-spec/compose-go/v3/validation" +) + +// nodePosition is the pre-canonical position snapshot for a single path. +type nodePosition struct { + file string + line int + column int +} + +// diagnoseValidation wraps a *validation.Error with the source location +// of the offending node so the user-facing error reads as +// "file:line:col: path: cause". When the node has lost its origin +// attribution to CanonicalNode's encode/decode round-trip (post-canonical +// nodes are fresh pointers absent from the origins map and carry +// Line / Column = 0), positions provides the pre-canonical snapshot +// keyed by tree.Path. fallbackFile is the absolute path of the first +// ConfigFile, used when neither side knows better. +func diagnoseValidation(err error, origins map[*yaml.Node]*node.SourceContext, positions map[string]nodePosition, fallbackFile string) error { + var ve *validation.Error + if !errors.As(err, &ve) { + return err + } + file, line, column := fallbackFile, 0, 0 + if ve.Node != nil { + line, column = ve.Node.Line, ve.Node.Column + if ctx := origins[ve.Node]; ctx != nil && ctx.File != "" { + file = ctx.File + } + } + if pos, ok := positions[ve.Path.String()]; ok { + if line == 0 { + line = pos.line + } + if column == 0 { + column = pos.column + } + if pos.file != "" { + file = pos.file + } + } + return &errdefs.Diagnostic{ + File: file, + Line: line, + Column: column, + Path: ve.Path.String(), + Cause: ve.Cause, + } +} + +// buildPathPositions snapshots every reachable path -> (file, line, +// column) so diagnostics survive CanonicalNode invalidating the origins +// pointer map. Walks mappings only (sequences expose their elements as +// `[]` segments that are unstable across canonical re-encoding). +func buildPathPositions(root *yaml.Node, origins map[*yaml.Node]*node.SourceContext) map[string]nodePosition { + out := map[string]nodePosition{} + target := root + if target == nil { + return out + } + if target.Kind == yaml.DocumentNode && len(target.Content) == 1 { + target = target.Content[0] + } + var visit func(n *yaml.Node, p tree.Path) + visit = func(n *yaml.Node, p tree.Path) { + if n == nil { + return + } + if n.Kind == yaml.MappingNode { + for i := 0; i+1 < len(n.Content); i += 2 { + k, v := n.Content[i], n.Content[i+1] + child := p.Next(k.Value) + pos := nodePosition{line: v.Line, column: v.Column} + if ctx := origins[v]; ctx != nil { + pos.file = ctx.File + } else if ctx := origins[k]; ctx != nil { + pos.file = ctx.File + } + out[child.String()] = pos + visit(v, child) + } + } + } + visit(target, tree.NewPath()) + return out +} + +// firstConfigFile returns the absolute path of the first ConfigFile +// when one is set, or node.SourceInline as a default. Used as the +// diagnostics fallback when a per-node origin is not available. +func firstConfigFile(cd types.ConfigDetails) string { + for _, f := range cd.ConfigFiles { + if f.Filename != "" { + return f.Filename + } + } + return node.SourceInline +} + +// load runs the full yaml.Node-centric pipeline over the input +// ConfigDetails and returns the merged compose tree as a canonical +// *yaml.Node. The pointer on cd lets the projectName side effect on +// cd.Environment (COMPOSE_PROJECT_NAME) propagate back to the caller +// and reach nodeToProject through the same Environment map. +// +// The pipeline goes: +// +// 1. parse every ConfigFile into one or more Layer values +// (LoadLayer + recursive CollectIncludeLayers); +// 2. apply extends inside each layer (ApplyExtendsToLayer); +// 3. populate per-scalar origins so each scalar can be looked up against +// the SourceContext of the layer that produced it (lazy interpolation); +// 4. merge layers left-to-right via override.MergeNode at the root path +// (ConfigFiles[0] is base, later files override); +// 5. apply !reset / !override paths collected from each layer; +// 6. interpolate every scalar with its own SourceContext.Environment; +// 7. canonicalize short-form syntax via transform.CanonicalNode; +// 8. resolve relative paths per-scalar via paths.ResolveRelativePathsNode; +// 9. validate via validation.ValidateNode; +// 10. normalize defaults via NormalizeNode. +// +// Entry points are LoadWithContext (returns *types.Project) and +// LoadModelWithContext (returns map[string]any). +func load(ctx context.Context, cd *types.ConfigDetails, opts *Options) (*yaml.Node, error) { + opts = ensureLoadOptions(opts, *cd) + // Extract the project name from the first config file (or its `name:` + // field) before the pipeline runs. Errors from explicit-name + // validation (NormalizeProjectName) propagate; an empty result is + // rejected after schema validation below. + if err := projectName(cd, opts); err != nil { + return nil, err + } + + rootCtx := &node.SourceContext{ + WorkingDir: cd.WorkingDir, + Environment: cd.Environment, + } + + allLayers, err := collectAllLayers(ctx, *cd, rootCtx, opts) + if err != nil { + return nil, err + } + if len(allLayers) == 0 { + return nil, errors.New("empty compose file") + } + + // Lazy env_file interpolation: capture each env_file entry's + // declaring-layer environment so nodeToProject can attach it to the + // Project EnvFileScopes side-table. WithServicesEnvironmentResolved + // then prefers that scope when interpolating the env_file content. + if opts.envFileScopes == nil { + opts.envFileScopes = map[string]types.Mapping{} + } + for _, layer := range allLayers { + captureEnvFileScopes(layer, opts.envFileScopes) + } + + if !opts.SkipExtends { + if err := applyExtendsPerLayer(ctx, allLayers, opts); err != nil { + return nil, err + } + } + + origins := map[*yaml.Node]*node.SourceContext{} + for _, layer := range allLayers { + populateOrigins(origins, layer.Node, layer.Context) + } + + merged, resetPaths, err := mergeLayers(allLayers) + if err != nil { + return nil, err + } + node.ApplyResetPaths(merged.Node, resetPaths) + + // Remove the include directive from the final tree (it has been + // consumed by collectAllLayers). + deleteMappingKey(merged.Node, "include") + + if !opts.SkipInterpolation { + if err := interpolateMerged(merged, origins, opts); err != nil { + return nil, diagnoseInterpolation(err, origins, firstConfigFile(*cd)) + } + } + + // Snapshot every reachable path -> (file, line, column) up front so + // schema validation diagnostics can resolve the offending location + // (the schema validator only returns the dotted path) and the + // post-canonical compose-rule validator can fall back to it after + // CanonicalNode wipes Line / Column on every fresh node. The + // snapshot is also exposed on Project.Sources when the caller + // opted in via Options.Diagnostics. + positions := buildPathPositions(merged.Node, origins) + stashPositionsForDiagnostics(opts, positions) + + // JSON Schema validation runs early — before canonicalization and + // transform — so structural errors (top-level not a mapping, services + // declared as a list, ...) are caught with a clear message rather + // than panicking inside a downstream transformer that assumes a + // canonical shape. + if err := validateAndStripVersion(merged.Node, *cd, opts, positions); err != nil { + return nil, err + } + + // Lazy bare-key environment resolution: services.*.environment entries + // that are just `KEY` (no `=`) get rewritten to `KEY=value` using each + // scalar own SourceContext.Environment. Mirrors v2 ResolveEnvironment + // but operates per-scalar so an env_file scoped to an include block is + // visible to services declared inside that include — and not leaked to + // the surrounding project environment. + ResolveEnvironmentNode(merged.Node, origins) + + // Capture per-scalar secret/config `environment: NAME` resolutions + // BEFORE CanonicalNode re-encodes subtrees and invalidates origin + // pointers. The captured map[name]value is replayed onto the tree + // AFTER the compose-rule validator so the synthesized `content` + // scalar does not trip the content+environment mutual-exclusivity + // check. + secretContents, configContents := CaptureSecretConfigContent(merged.Node, origins) + + // Path resolution runs first on the pre-canonical tree so that + // pointer identity is preserved for every scalar whose origin is + // tracked in the side-table. The CanonicalNode bridge currently + // rebuilds affected subtrees via map[string]any, which would lose + // origin pointers — Phase B follow-ups will port individual + // transformers to operate on yaml.Node and remove this constraint. + if opts.ResolvePaths { + var remotes []paths.RemoteResource + for _, loader := range opts.RemoteResourceLoaders() { + remotes = append(remotes, loader.Accept) + } + if err := paths.ResolveRelativePathsNode(merged.Node, paths.NodeResolverOptions{ + WorkingDirFor: workingDirLookup(origins, merged.Context.WorkingDir), + Remotes: remotes, + }); err != nil { + return nil, err + } + } + + // Snapshot a service-name → SourceContext map BEFORE Canonical to + // survive the bridge: Canonical re-encodes the merged tree and loses + // the origin pointer identity for every Node, so post-canonical + // passes that need per-service context (default build.context + // resolution) consult this name-keyed map instead of the pointer map. + serviceContexts := buildServiceContexts(merged.Node, origins) + + if _, err := transform.CanonicalNode(merged.Node, opts.SkipInterpolation); err != nil { + return nil, err + } + + // SetDefaultValues fills in canonical defaults (DeviceCount(-1) for + // unspecified GPU count, default network configuration, default build + // context ".", ...). Runs after Canonical through a temporary map + // roundtrip until per-rule Node ports land. Path resolution + // intentionally runs *before* + // SetDefaultValues so the per-scalar origins side-table can still drive + // the WorkingDir lookup. Defaults that are themselves path-shaped + // (build.context ".") are resolved by a targeted helper below rather + // than by a second full sweep, which would double-resolve every + // already-handled relative path. + if !opts.SkipDefaultValues { + if err := setDefaultValuesNode(merged.Node); err != nil { + return nil, err + } + if opts.ResolvePaths { + resolveDefaultBuildContext(merged.Node, cd.WorkingDir, serviceContexts) + } + } + + // Post-canonical path resolution for entries whose short form bypassed + // the pre-canonical sweep (volumes:./host:/container yields canonical + // nodes with no recorded origin). v2 ran a second paths.ResolveRelative + // Paths after Canonical in loadYamlModel; mirror that here, but use the + // per-service serviceContexts so an included service still picks up + // the include project_directory rather than the project root. + if opts.ResolvePaths { + resolveServiceVolumeSources(merged.Node, cd.WorkingDir, serviceContexts) + } + + if !opts.SkipValidation { + if err := validation.ValidateNode(merged.Node); err != nil { + return nil, diagnoseValidation(err, origins, positions, firstConfigFile(*cd)) + } + // Reject a load whose project name is still empty at this + // point. The check is gated on SkipValidation to keep the + // orchestrator usable from tests that skip validation outright. + if opts.projectName == "" { + return nil, errors.New("project name must not be empty") + } + } + + if !opts.SkipNormalization { + // Stamp the resolved project name onto the tree so + // setNameFromKeyNode can build the implicit + // "_" names with the right prefix. Mirrors + // the v2 `dict["name"] = opts.projectName` step that ran just + // before Normalize. + setMappingValue(merged.Node, "name", &yaml.Node{ + Kind: yaml.ScalarNode, Tag: "!!str", Value: opts.projectName, + }) + if _, err := NormalizeNode(merged.Node, cd.Environment); err != nil { + return nil, err + } + } + + root := merged.Node + if root.Kind == yaml.DocumentNode && len(root.Content) == 1 { + root = root.Content[0] + } + if root.Kind != yaml.MappingNode || len(root.Content) == 0 { + return nil, errors.New("empty compose file") + } + + // Drop empty attributes that resulted from interpolation of unset + // variables (e.g. `dns: ${UNSET}` -> `dns: ""` collapses to absent). + // Equivalent of v2 loadYamlModel's post-Canonical OmitEmpty pass, + // applied at the node level so both nodeToModel and nodeToProject + // observe the same shape. + omitEmptyNode(root, tree.NewPath()) + + // Replay the per-scalar secret/config Content resolution captured + // before CanonicalNode invalidated the origin pointer map. Runs + // after the validator so the synthesized `content` scalar does not + // trip the content+environment mutual-exclusivity check. + ApplySecretConfigContent(root, secretContents, configContents) + + return root, nil +} + +// omitEmptyNode walks the tree and drops entries whose value is empty +// (nil / empty string) when their path matches one of the omitempty +// patterns. Mirrors what v2 OmitEmpty used to do on the map representation. +var omitEmptyPatterns = []tree.Path{ + "services.*.dns", +} + +func mustOmit(p tree.Path) bool { + for _, pattern := range omitEmptyPatterns { + if p.Matches(pattern) { + return true + } + } + return false +} + +func omitEmptyNode(n *yaml.Node, p tree.Path) { + if n == nil { + return + } + switch n.Kind { + case yaml.MappingNode: + filtered := n.Content[:0] + for i := 0; i+1 < len(n.Content); i += 2 { + k, v := n.Content[i], n.Content[i+1] + child := p.Next(k.Value) + if isEmptyNode(v) && mustOmit(child) { + continue + } + omitEmptyNode(v, child) + filtered = append(filtered, k, v) + } + n.Content = filtered + case yaml.SequenceNode: + // The map-based OmitEmpty passes the parent path to mustOmit (not + // path.Next("[]")) so a pattern like `services.*.dns` filters + // scalar items inside the dns sequence. Mirror that here. + filtered := n.Content[:0] + for _, item := range n.Content { + if isEmptyNode(item) && mustOmit(p) { + continue + } + omitEmptyNode(item, p.Next("[]")) + filtered = append(filtered, item) + } + n.Content = filtered + } +} + +func isEmptyNode(n *yaml.Node) bool { + if n == nil || n.Tag == "!!null" { + return true + } + return n.Kind == yaml.ScalarNode && n.Value == "" +} + +// ensureLoadOptions applies the same defaults as ToOptions for callers +// that pass a bare *Options (most production callers go through +// ToOptions; this covers tests that build the struct directly). +func ensureLoadOptions(opts *Options, cd types.ConfigDetails) *Options { + if opts == nil { + opts = &Options{} + } + if !hasLocalLoader(opts.ResourceLoaders) { + opts.ResourceLoaders = append(opts.ResourceLoaders, localResourceLoader{WorkingDir: cd.WorkingDir}) + } + if opts.Interpolate == nil { + opts.Interpolate = &interp.Options{ + Substitute: template.Substitute, + LookupValue: cd.LookupEnv, + TypeCastMapping: interpolateTypeCastMapping, + } + } + return opts +} + +// nodeToModel projects the merged tree into the legacy map[string]any +// shape consumed by LoadModelWithContext. OmitEmpty and the per-scalar +// secrets / configs environment resolution have already run on the node +// (Load calls them); the map is only the decoded view. +func nodeToModel(root *yaml.Node) (map[string]any, error) { + var dict map[string]any + if err := root.Decode(&dict); err != nil { + return nil, fmt.Errorf("load: decode merged tree: %w", err) + } + return dict, nil +} + +// validateAndStripVersion runs the JSON Schema validator on a decoded +// view of the merged tree and, on success, strips the obsolete top-level +// `version` attribute with the deprecation warning. Carved out of load +// to keep its cyclomatic complexity in check. The positions snapshot +// turns each schema failure into an *errdefs.Diagnostic that points at +// the line and column the user wrote. +func validateAndStripVersion(root *yaml.Node, cd types.ConfigDetails, opts *Options, positions map[string]nodePosition) error { + if opts.SkipValidation { + return nil + } + var schemaDict map[string]any + if err := root.Decode(&schemaDict); err != nil { + return fmt.Errorf("load: decode for schema validation: %w", err) + } + if err := schema.Validate(schemaDict); err != nil { + return diagnoseSchema(err, positions, firstConfigFile(cd)) + } + if hasMappingKey(root, "version") { + for _, f := range cd.ConfigFiles { + opts.warnObsoleteVersion(f.Filename) + } + deleteMappingKey(root, "version") + } + return nil +} + +// stashPositionsForDiagnostics records the per-path positions snapshot +// onto opts when the caller opted in via WithDiagnostics. nodeToProject +// reads the same snapshot back when populating Project.Sources. +func stashPositionsForDiagnostics(opts *Options, positions map[string]nodePosition) { + if !opts.Diagnostics { + return + } + opts.pathPositions = positions +} + +// diagnoseAt wraps cause as an errdefs.Diagnostic that points at the +// given file and node. Used by the orchestrator to enrich the errors +// returned by helper functions that do not themselves know the source +// file. A nil cause yields nil so callers can pass through. +func diagnoseAt(cause error, file string, n *yaml.Node, path string) error { + if cause == nil { + return nil + } + var diag *errdefs.Diagnostic + if errors.As(cause, &diag) { + // Cause already carries position metadata; preserve it. + return cause + } + d := &errdefs.Diagnostic{ + File: file, + Path: path, + Cause: errString(cause.Error()), + } + if n != nil { + d.Line = n.Line + d.Column = n.Column + } + return d +} + +// diagnoseInterpolation wraps an *interpolation.Error with the source +// location of the offending scalar so a strict-mode substitution +// failure (unset `${VAR:?msg}`, invalid template, ...) surfaces as +// "file:line:col: path: cause" instead of a bare interpolation +// message. +func diagnoseInterpolation(err error, origins map[*yaml.Node]*node.SourceContext, fallbackFile string) error { + var ie *interp.Error + if !errors.As(err, &ie) || ie.Node == nil { + return err + } + file := fallbackFile + if ctx := origins[ie.Node]; ctx != nil && ctx.File != "" { + file = ctx.File + } + return &errdefs.Diagnostic{ + File: file, + Line: ie.Node.Line, + Column: ie.Node.Column, + Path: ie.Path.String(), + Cause: errString(ie.Cause.Error()), + } +} + +// diagnoseSchema wraps a schema validation failure with the source +// location of the offending value. The schema package exposes a *Error +// whose Path() returns the dotted compose path; we look that path up in +// the pre-canonical positions snapshot to pull the file, line and +// column the user wrote. +func diagnoseSchema(err error, positions map[string]nodePosition, fallbackFile string) error { + var se *schema.Error + if !errors.As(err, &se) { + return err + } + path := se.Path() + pos := positions[path] + file := pos.file + if file == "" { + file = fallbackFile + } + return &errdefs.Diagnostic{ + File: file, + Line: pos.line, + Column: pos.column, + Path: path, + Cause: errString(se.Error()), + } +} + +// errString is a tiny error type used to attach a plain string body to +// a Diagnostic without re-prepending the path that the wrapped Cause +// already embeds. Keeps the rendered diagnostic free of duplicated +// prefixes when the wrapped error is one of the package-specific +// *Error types. +type errString string + +func (e errString) Error() string { return string(e) } + +// setDefaultValuesNode applies the v2 transform.SetDefaultValues defaults +// to the merged tree via a temporary map roundtrip. Sets DeviceCount(-1) +// for unspecified GPU count and similar defaults that exist outside the +// per-path Canonical transformers. The Node-typed port lives in transform/ +// and replaces the bridge in a follow-up. +func setDefaultValuesNode(root *yaml.Node) error { + target := root + if target.Kind == yaml.DocumentNode && len(target.Content) == 1 { + target = target.Content[0] + } + var data map[string]any + if err := target.Decode(&data); err != nil { + return fmt.Errorf("load: decode for SetDefaultValues: %w", err) + } + defaulted, err := transform.SetDefaultValues(data) + if err != nil { + return err + } + var rebuilt yaml.Node + if err := rebuilt.Encode(defaulted); err != nil { + return fmt.Errorf("load: re-encode after SetDefaultValues: %w", err) + } + *target = rebuilt + return nil +} + +// resolveDefaultBuildContext walks services.*.build.context entries and, +// for each one whose value is "." or empty (i.e. the default produced by +// SetDefaultValues for builds that did not declare a context), joins it +// with the appropriate working directory. The service node's origin is +// consulted first so an included service whose build had no context picks +// up the include's project_directory; falls back to projectWD for services +// whose origin is unknown (e.g. synthesized by SetDefaultValues itself). +// +// Tightly scoped to avoid the double-resolution problem that a generic +// post-defaults sweep would introduce on relative paths already resolved +// by the earlier pass. +func resolveDefaultBuildContext(root *yaml.Node, projectWD string, serviceContexts map[string]string) { + target := root + if target.Kind == yaml.DocumentNode && len(target.Content) == 1 { + target = target.Content[0] + } + services := mappingValueByKey(target, "services") + if services == nil || services.Kind != yaml.MappingNode { + return + } + for i := 0; i+1 < len(services.Content); i += 2 { + name := services.Content[i].Value + svc := services.Content[i+1] + build := mappingValueByKey(svc, "build") + if build == nil || build.Kind != yaml.MappingNode { + continue + } + ctx := mappingValueByKey(build, "context") + if ctx == nil || ctx.Kind != yaml.ScalarNode { + continue + } + if ctx.Value != "." && ctx.Value != "" { + continue + } + wd := projectWD + if origin, ok := serviceContexts[name]; ok && origin != "" { + wd = origin + } + ctx.Value = wd + } +} + +// resolveServiceVolumeSources walks the canonical services.*.volumes +// sequence and joins each relative bind-mount source with the service's +// recorded WorkingDir. Sources that the pre-canonical sweep already +// absolutized (because they were declared in long form to begin with) +// are skipped via filepath.IsAbs. Volume entries whose type is not bind +// (named volumes) are left untouched. +func resolveServiceVolumeSources(root *yaml.Node, projectWD string, serviceContexts map[string]string) { + target := root + if target.Kind == yaml.DocumentNode && len(target.Content) == 1 { + target = target.Content[0] + } + services := mappingValueByKey(target, "services") + if services == nil || services.Kind != yaml.MappingNode { + return + } + for i := 0; i+1 < len(services.Content); i += 2 { + name := services.Content[i].Value + svc := services.Content[i+1] + volumes := mappingValueByKey(svc, "volumes") + if volumes == nil || volumes.Kind != yaml.SequenceNode { + continue + } + wd := projectWD + if origin, ok := serviceContexts[name]; ok && origin != "" { + wd = origin + } + if wd == "" { + continue + } + for _, item := range volumes.Content { + if item == nil || item.Kind != yaml.MappingNode { + continue + } + if mappingValueByKey(item, "type") == nil || mappingValueByKey(item, "type").Value != "bind" { + continue + } + source := mappingValueByKey(item, "source") + if source == nil || source.Kind != yaml.ScalarNode || source.Value == "" { + continue + } + if filepath.IsAbs(source.Value) || paths.IsWindowsAbs(source.Value) { + continue + } + // Only resolve sources that still carry the relative-dot + // indicator that format.ParseVolume preserved from the + // short form. A value like "testdata/subdir/foo" comes + // from a long-form mapping the pre-canonical sweep + // already absolutized against its layer WorkingDir; + // re-joining it here would double the relative prefix. + if !strings.HasPrefix(source.Value, ".") { + continue + } + source.Value = filepath.Join(wd, source.Value) + } + } +} + +// buildServiceContexts inspects the merged tree's `services` mapping and +// records, for each service name, the WorkingDir of the SourceContext that +// produced it. The map survives the CanonicalNode bridge because it is +// keyed by name (a stable identifier) rather than by Node pointer. Used by +// resolveDefaultBuildContext to give an included service whose build had +// no context the include's project_directory as the resolved default. +func buildServiceContexts(root *yaml.Node, origins map[*yaml.Node]*node.SourceContext) map[string]string { + out := map[string]string{} + target := root + if target.Kind == yaml.DocumentNode && len(target.Content) == 1 { + target = target.Content[0] + } + services := mappingValueByKey(target, "services") + if services == nil || services.Kind != yaml.MappingNode { + return out + } + for i := 0; i+1 < len(services.Content); i += 2 { + name := services.Content[i].Value + svc := services.Content[i+1] + if wd := serviceOriginWorkingDir(svc, origins); wd != "" { + out[name] = wd + } + } + return out +} + +func serviceOriginWorkingDir(svc *yaml.Node, origins map[*yaml.Node]*node.SourceContext) string { + if ctx, ok := origins[svc]; ok && ctx != nil { + return ctx.WorkingDir + } + for _, c := range svc.Content { + if c == nil { + continue + } + if ctx, ok := origins[c]; ok && ctx != nil { + return ctx.WorkingDir + } + } + return "" +} + +// hasMappingKey reports whether n is a MappingNode containing key. +func hasMappingKey(n *yaml.Node, key string) bool { + if n == nil || n.Kind != yaml.MappingNode { + return false + } + for i := 0; i+1 < len(n.Content); i += 2 { + if n.Content[i].Value == key { + return true + } + } + return false +} + +// captureEnvFileScopes walks a layer's services and records, for each +// env_file entry it carries, the layer environment in effect when the +// entry was declared. Keyed by the resolved env_file path (absolute when +// CollectIncludeLayers has pre-resolved it, raw otherwise) so the +// downstream ModelToProject step can attach Mapping to the corresponding +// types.EnvFile.Env field. +func captureEnvFileScopes(layer *node.Layer, scopes map[string]types.Mapping) { + if layer == nil || layer.Context == nil || layer.Context.Parent == nil || len(layer.Context.Environment) == 0 { + return + } + target := layer.Node + if target == nil { + return + } + if target.Kind == yaml.DocumentNode && len(target.Content) == 1 { + target = target.Content[0] + } + services := mappingValueByKey(target, "services") + if services == nil || services.Kind != yaml.MappingNode { + return + } + for i := 1; i < len(services.Content); i += 2 { + svc := services.Content[i] + if svc == nil || svc.Kind != yaml.MappingNode { + continue + } + envFile := mappingValueByKey(svc, "env_file") + if envFile == nil { + continue + } + switch envFile.Kind { + case yaml.ScalarNode: + scopes[envFile.Value] = layer.Context.Environment + case yaml.SequenceNode: + for _, item := range envFile.Content { + switch item.Kind { + case yaml.ScalarNode: + scopes[item.Value] = layer.Context.Environment + case yaml.MappingNode: + if p := mappingValueByKey(item, "path"); p != nil && p.Kind == yaml.ScalarNode { + scopes[p.Value] = layer.Context.Environment + } + } + } + } + } +} + +// applyExtendsPerLayer iterates layers and applies extends to each with a +// child-scoped Options whose localResourceLoader points at the layer's own +// WorkingDir. Mirrors v2 ApplyExtends running per-file inside the recursive +// loadYamlModel of an include, so a relative extends.file declared in an +// included file resolves against the include project_directory. +func applyExtendsPerLayer(ctx context.Context, allLayers []*node.Layer, opts *Options) error { + tracker := &cycleTracker{} + for _, layer := range allLayers { + layerOpts := opts + if layer.Context != nil && layer.Context.WorkingDir != "" && layer.Context.WorkingDir != opts.workingDirOfFirstLoader() { + layerOpts = opts.clone() + layerOpts.ResourceLoaders = append(opts.RemoteResourceLoaders(), localResourceLoader{WorkingDir: layer.Context.WorkingDir}) + } + if err := ApplyExtendsToLayer(ctx, layer, layerOpts, tracker); err != nil { + return err + } + } + return nil +} + +// collectAllLayers parses each ConfigFile and recursively folds in every +// include directive it carries. The returned slice is ordered so that +// included files appear before their parent, which matches the v2 +// importResources convention where the parent overrides the include. +func collectAllLayers(ctx context.Context, cd types.ConfigDetails, root *node.SourceContext, opts *Options) ([]*node.Layer, error) { + var all []*node.Layer + seen := map[string]bool{} + chain := []string{} + for _, file := range cd.ConfigFiles { + sc := *root + sc.File = file.Filename + layers, err := LoadLayer(ctx, file, &sc, opts) + if err != nil { + return nil, err + } + for _, layer := range layers { + expanded, err := expandIncludes(ctx, layer, opts, seen, chain) + if err != nil { + return nil, err + } + all = append(all, expanded...) + } + } + return all, nil +} + +// expandIncludes returns layer prefixed by every include layer reachable +// from it (recursive traversal). Cycle protection comes from the cycle +// tracker maintained by CollectIncludeLayers; an explicit visited set at +// this level guards against fixture-induced infinite loops in the +// orchestrator itself. +// +// Each child include is processed recursively with opts re-rooted at the +// child's WorkingDir so its own include directives resolve relative paths +// against the include's project_directory, not the outer project root. +// Matches v2 ApplyInclude which similarly replaces ResourceLoaders on the +// recursive load. +func expandIncludes(ctx context.Context, layer *node.Layer, opts *Options, seen map[string]bool, chain []string) ([]*node.Layer, error) { + if opts.SkipInclude { + return []*node.Layer{layer}, nil + } + // Cycle detection: track the absolute filename chain. A file that + // appears as its own ancestor (directly or transitively) means an + // include directive eventually points back to a file already being + // expanded; return an "include cycle detected" diagnostic pointing + // at the offending file rather than recursing forever. + if layer.Context != nil && layer.Context.File != "" { + file := layer.Context.File + if seen[file] { + cause := errString(fmt.Sprintf("include cycle detected:\n%s\n include %s", chain[0], strings.Join(append(chain[1:], file), "\n include "))) + return nil, &errdefs.Diagnostic{ + File: file, + Cause: cause, + } + } + seen[file] = true + chain = append(chain, file) + defer func() { + delete(seen, file) + }() + } + children, err := CollectIncludeLayers(ctx, layer, opts) + if err != nil { + return nil, err + } + var out []*node.Layer + for _, child := range children { + childOpts := opts + if child.Context != nil && child.Context.WorkingDir != "" && child.Context.WorkingDir != opts.workingDirOfFirstLoader() { + childOpts = opts.clone() + childOpts.ResourceLoaders = append(opts.RemoteResourceLoaders(), localResourceLoader{WorkingDir: child.Context.WorkingDir}) + } + grandchildren, err := expandIncludes(ctx, child, childOpts, seen, chain) + if err != nil { + return nil, err + } + out = append(out, grandchildren...) + } + out = append(out, layer) + return out, nil +} + +// populateOrigins records the SourceContext for every node reachable from +// root in m, so the merge phase can later look up which layer a scalar +// originated from. Mappings, sequences and scalars are all recorded; +// downstream phases query the map per scalar. Existing entries are +// preserved, so a sub-load (extends merging a cloned base service from a +// different SourceContext) that pre-stamped its clones can override the +// parent layer attribution that would otherwise win here. +func populateOrigins(m map[*yaml.Node]*node.SourceContext, root *yaml.Node, ctx *node.SourceContext) { + if root == nil || ctx == nil { + return + } + target := root + if target.Kind == yaml.DocumentNode && len(target.Content) == 1 { + target = target.Content[0] + } + var visit func(n *yaml.Node) + visit = func(n *yaml.Node) { + if n == nil { + return + } + if _, exists := m[n]; !exists { + m[n] = ctx + } + for _, c := range n.Content { + visit(c) + } + } + visit(target) +} + +// mergeLayers folds layers[1:] into layers[0] using override.MergeNode at +// the root path. Before each merge, the right-hand layer's recorded +// !reset / !override paths are applied to the accumulator so the override +// value replaces (rather than merges with) the base; the same paths are +// then dropped from the returned list so the orchestrator post-merge +// ApplyResetPaths does not delete the value it was meant to preserve. +func mergeLayers(layers []*node.Layer) (*node.Layer, []tree.Path, error) { + acc := layers[0] + resetPaths := append([]tree.Path(nil), acc.ResetPaths()...) + for _, layer := range layers[1:] { + if len(layer.ResetPaths()) > 0 { + node.ApplyResetPaths(acc.Node, layer.ResetPaths()) + } + out, err := override.MergeNode(acc.Node, layer.Node, tree.NewPath()) + if err != nil { + return nil, nil, err + } + acc.Node = out + // Do not re-record paths consumed during merge; they have served + // their purpose by clearing the base value, and re-applying them + // post-merge would delete the override value the user wants kept. + } + if _, err := override.EnforceUnicityNode(acc.Node); err != nil { + return nil, nil, err + } + return acc, resetPaths, nil +} + +// interpolateMerged runs lazy per-scalar interpolation across the merged +// tree, using the origins map to pick the right SourceContext for each +// scalar. The fall-back is the merged layer Context, which applies to +// synthetic nodes injected by canonicalization / merge promotion. +func interpolateMerged(merged *node.Layer, origins map[*yaml.Node]*node.SourceContext, opts *Options) error { + substitute := template.Substitute + if opts.Interpolate != nil && opts.Interpolate.Substitute != nil { + substitute = opts.Interpolate.Substitute + } + lookupFor := func(n *yaml.Node) interp.LookupValue { + ctx := origins[n] + if ctx == nil { + ctx = merged.Context + } + env := ctx.Environment + return func(k string) (string, bool) { + v, ok := env[k] + return v, ok + } + } + return interp.InterpolateNode(merged.Node, interp.NodeOptions{ + LookupValueFor: lookupFor, + Substitute: substitute, + Tags: tagsForCasts(), + }) +} + +// workingDirLookup returns a function that picks the working directory to +// use when resolving a relative path scalar. Each scalar consults the +// origins map for its SourceContext; nodes that have no recorded origin +// (synthesized during merge) fall back to fallback. +func workingDirLookup(origins map[*yaml.Node]*node.SourceContext, fallback string) func(*yaml.Node) string { + return func(n *yaml.Node) string { + if ctx := origins[n]; ctx != nil { + // Skip scalars whose layer already went through the include + // sub-load path resolution: re-resolving them at this level + // would double-join when the include project_directory was + // relative. + if ctx.PathsPreResolved { + return "" + } + if ctx.WorkingDir != "" { + return ctx.WorkingDir + } + } + return fallback + } +} + +// tagsForCasts maps tree.Path patterns to YAML tags so the interpolation +// phase can rewrite scalar.Tag in place after substitution, letting yaml.v4 +// perform the type conversion natively at decode time. Mirrors the cast +// targets registered in interpolateTypeCastMapping. +func tagsForCasts() map[tree.Path]string { + out := map[tree.Path]string{} + for path, caster := range interpolateTypeCastMapping { + out[path] = tagForCast(caster) + } + return out +} + +// hasLocalLoader reports whether the slice already contains a +// localResourceLoader. Order-insensitive helper for the defensive +// initialization in Load. +func hasLocalLoader(loaders []ResourceLoader) bool { + for _, l := range loaders { + if _, ok := l.(localResourceLoader); ok { + return true + } + } + return false +} + +// workingDirOfFirstLoader returns the WorkingDir of the first +// localResourceLoader in opts.ResourceLoaders, or empty when none is +// present. Used to detect when expandIncludes should clone Options to +// re-root the resource lookup at a child's project_directory. +func (o Options) workingDirOfFirstLoader() string { + for _, l := range o.ResourceLoaders { + if local, ok := l.(localResourceLoader); ok { + return local.WorkingDir + } + } + return "" +} + +func tagForCast(c interp.Cast) string { + if c == nil { + return "" + } + v, err := c("0") + if err != nil { + return "" + } + switch v.(type) { + case bool: + return "!!bool" + case int, int32, int64: + return "!!int" + case float32, float64: + return "!!float" + } + return "" +} diff --git a/loader/load_bench_test.go b/loader/load_bench_test.go new file mode 100644 index 000000000..b3ac5a2d8 --- /dev/null +++ b/loader/load_bench_test.go @@ -0,0 +1,132 @@ +/* + Copyright 2020 The Compose Specification Authors. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +*/ + +package loader + +import ( + "context" + "fmt" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/compose-spec/compose-go/v3/types" +) + +// BenchmarkLoadSmall measures the loader cost on a small project +// (one service). Establishes the per-invocation overhead floor: +// every Load goes through the full pipeline (parse, reset, alias +// normalize, merge, interpolate, canonical, paths, validate, +// normalize, decode) regardless of the project size. +func BenchmarkLoadSmall(b *testing.B) { + dir := b.TempDir() + writeFileBench(b, dir, "compose.yaml", ` +services: + web: + image: nginx + ports: + - "80:80" +`) + cd := types.ConfigDetails{ + WorkingDir: dir, + ConfigFiles: []types.ConfigFile{{Filename: filepath.Join(dir, "compose.yaml")}}, + Environment: map[string]string{}, + } + b.ResetTimer() + b.ReportAllocs() + for i := 0; i < b.N; i++ { + _, err := LoadWithContext(context.TODO(), cd, withProjectName("bench-small", true)) + if err != nil { + b.Fatal(err) + } + } +} + +// BenchmarkLoadMedium exercises a project with 50 services that share +// a YAML anchor for build defaults. Useful as a comparison point for +// the small benchmark to estimate scaling under realistic input. +func BenchmarkLoadMedium(b *testing.B) { + dir := b.TempDir() + var sb strings.Builder + sb.WriteString(`x-build: &build + context: . + dockerfile: Dockerfile + +services: +`) + for i := 0; i < 50; i++ { + fmt.Fprintf(&sb, ` svc%d: + image: alpine:3.${TAG:-19} + build: *build + environment: + INDEX: %d + ports: + - "%d:%d" +`, i, i, 8000+i, 80) + } + writeFileBench(b, dir, "compose.yaml", sb.String()) + cd := types.ConfigDetails{ + WorkingDir: dir, + ConfigFiles: []types.ConfigFile{{Filename: filepath.Join(dir, "compose.yaml")}}, + Environment: map[string]string{"TAG": "20"}, + } + b.ResetTimer() + b.ReportAllocs() + for i := 0; i < b.N; i++ { + _, err := LoadWithContext(context.TODO(), cd, withProjectName("bench-medium", true)) + if err != nil { + b.Fatal(err) + } + } +} + +// BenchmarkLoadWithDiagnostics measures the additional cost of the +// WithDiagnostics opt-in (the buildPathPositions snapshot must be +// retained across the whole pipeline and attached to Project.Sources). +func BenchmarkLoadWithDiagnostics(b *testing.B) { + dir := b.TempDir() + writeFileBench(b, dir, "compose.yaml", ` +services: + web: + image: nginx + ports: + - "80:80" +`) + cd := types.ConfigDetails{ + WorkingDir: dir, + ConfigFiles: []types.ConfigFile{{Filename: filepath.Join(dir, "compose.yaml")}}, + Environment: map[string]string{}, + } + b.ResetTimer() + b.ReportAllocs() + for i := 0; i < b.N; i++ { + _, err := LoadWithContext(context.TODO(), cd, func(o *Options) { + o.SetProjectName("bench-diag", true) + WithDiagnostics(o) + }) + if err != nil { + b.Fatal(err) + } + } +} + +func writeFileBench(b *testing.B, dir, name, content string) { + b.Helper() + if err := os.WriteFile(filepath.Join(dir, name), []byte(content), 0o600); err != nil { + b.Fatal(err) + } +} diff --git a/loader/load_extends.go b/loader/load_extends.go new file mode 100644 index 000000000..69fe98337 --- /dev/null +++ b/loader/load_extends.go @@ -0,0 +1,547 @@ +/* + Copyright 2020 The Compose Specification Authors. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +*/ + +package loader + +import ( + "context" + "fmt" + "path/filepath" + "strings" + + "go.yaml.in/yaml/v4" + + "github.com/compose-spec/compose-go/v3/errdefs" + "github.com/compose-spec/compose-go/v3/internal/node" + "github.com/compose-spec/compose-go/v3/override" + "github.com/compose-spec/compose-go/v3/paths" + "github.com/compose-spec/compose-go/v3/tree" + "github.com/compose-spec/compose-go/v3/types" +) + +// ApplyExtendsToLayer resolves every `extends` directive in layer's services +// block by walking the inheritance chain, merging base + derived service +// nodes through override.MergeNode at path "services.x", and stripping the +// `extends` field from the result. +// +// extends.file referencing another compose file produces an on-the-fly +// Layer for that file (parse + reset/override resolution + alias +// normalization) so the base service is available for merging. Those +// sub-layers are discarded after the merge; only the resulting merged +// service node is grafted back into the original layer's tree. +// +// Cycles are detected via the standard cycleTracker keyed by (file, name); +// the same value used by the v2 ApplyExtends so existing fixtures keep +// triggering the same diagnostics. +// +// ApplyExtendsToLayer mutates layer in place. It does not perform cross- +// file merge, interpolation, or path resolution; those run in subsequent +// phases of the orchestrator. +func ApplyExtendsToLayer(ctx context.Context, layer *node.Layer, opts *Options, tracker *cycleTracker) error { + services := layerMappingField(layer.Node, "services") + if services == nil || services.Kind != yaml.MappingNode { + return nil + } + for i := 0; i+1 < len(services.Content); i += 2 { + name := services.Content[i].Value + merged, err := applyServiceExtendsNode(ctx, layer, name, services, opts, tracker) + if err != nil { + return err + } + if merged != nil { + services.Content[i+1] = merged + } + } + return nil +} + +// applyServiceExtendsNode resolves the extends chain for a single service +// in siblingServices. It returns the merged service node or the original +// node when no extends directive is present. +// +// The base service is located either in siblingServices (same layer) or in +// a freshly loaded Layer for the file referenced by extends.file. extends +// is applied recursively to the base so a chain of N levels resolves +// before the final merge fires. +func applyServiceExtendsNode( + ctx context.Context, + layer *node.Layer, + name string, + siblingServices *yaml.Node, + opts *Options, + tracker *cycleTracker, +) (*yaml.Node, error) { + service := mappingValueByKey(siblingServices, name) + if service == nil { + return nil, nil + } + // A YAML null value (`name:` with no body) is treated as an empty + // service — same as v2, where the empty mapping contributes no fields + // to a downstream extends merge but is otherwise valid. + if service.Kind == yaml.ScalarNode && service.Tag == "!!null" { + return service, nil + } + if service.Kind != yaml.MappingNode { + return nil, &errdefs.Diagnostic{ + File: layer.Context.File, + Line: service.Line, + Column: service.Column, + Path: fmt.Sprintf("services.%s", name), + Cause: errString(fmt.Sprintf("services.%s must be a mapping", name)), + } + } + extendsNode := mappingValueByKey(service, "extends") + if extendsNode == nil { + return service, nil + } + + ref, file, err := parseExtendsRef(name, extendsNode, opts) + if err != nil { + return nil, &errdefs.Diagnostic{ + File: layer.Context.File, + Line: extendsNode.Line, + Column: extendsNode.Column, + Path: fmt.Sprintf("services.%s.extends", name), + Cause: errString(err.Error()), + } + } + + currentFile := layer.Context.File + baseSiblings := siblingServices + childOpts := opts + originalLayer := layer + if file != "" { + baseLayer, childOptsLoaded, err := loadExtendsBaseLayer(ctx, layer, file, opts) + if err != nil { + return nil, err + } + baseSiblings = layerMappingField(baseLayer.Node, "services") + if baseSiblings == nil { + return nil, &errdefs.Diagnostic{ + File: layer.Context.File, + Line: extendsNode.Line, + Column: extendsNode.Column, + Path: fmt.Sprintf("services.%s.extends", name), + Cause: errString(fmt.Sprintf("cannot extend service %q in %s: no services section", name, file)), + } + } + currentFile = baseLayer.Context.File + // Reuse layer so the recursion sees the base layer's tree, but + // keep the child-scoped opts so further extends.file references + // resolve against the extended file's directory rather than the + // project root. + layer = baseLayer + childOpts = childOptsLoaded + } + + if mappingValueByKey(baseSiblings, ref) == nil { + diag := &errdefs.Diagnostic{ + Path: fmt.Sprintf("services.%s.extends", name), + Cause: errString(fmt.Sprintf("cannot extend service %q in %s: service %q not found", name, layer.Context.File, ref)), + } + if extendsNode != nil { + diag.File = originalLayer.Context.File + diag.Line = extendsNode.Line + diag.Column = extendsNode.Column + } + return nil, diag + } + + tracker, err = tracker.Add(currentFile, name) + if err != nil { + return nil, err + } + + // Recurse into the base to resolve its own extends chain first. + base, err := applyServiceExtendsNode(ctx, layer, ref, baseSiblings, childOpts, tracker) + if err != nil { + return nil, err + } + if base == nil { + return service, nil + } + // Mutate the sibling services mapping so the resolved base replaces + // its original entry. Subsequent top-level iterations over the same + // services mapping see the already-resolved base and skip re-entering + // the extends chain — mirrors the v2 `services[name] = merged` side + // effect that keeps Listener event counts deterministic. + setMappingValue(baseSiblings, ref, base) + + // Apply the parent layer's recorded !reset / !override paths to the + // cloned base BEFORE merging it with the derived service. Mirrors v2 + // applyServiceExtends, which calls processor.Apply on the wrapped base + // to drop any path that the derived service marked with !reset or + // !override — so the override entry from the derived service wins + // outright once mergeSpecials kicks in. The consumed paths are then + // removed from the layer's resetPaths so the orchestrator post-merge + // ApplyResetPaths does not delete them again from the final tree. + clonedBase := deepCloneNode(base) + consumed := resetParentPaths(clonedBase, name, originalLayer.ResetPaths()) + if len(consumed) > 0 { + originalLayer.SetResetPaths(diffPaths(originalLayer.ResetPaths(), consumed)) + } + + // Merge base + service through the standard service-level rules. The + // canonical merge path is "services.x" — same key used by the v2 + // override.ExtendService. + merged, err := override.MergeNode(clonedBase, service, tree.NewPath("services", "x")) + if err != nil { + return nil, err + } + deleteMappingKey(merged, "extends") + // When extends went through an extends.file (loaded a sub-layer), + // rewrite relative paths in the merged service against the sub-file's + // working directory. + if file != "" { + extendsWD := childOpts.extendsRelativeDir + if extendsWD == "" { + extendsWD = layer.Context.WorkingDir + } + if err := resolveExtendedServicePaths(merged, extendsWD, childOpts); err != nil { + return nil, err + } + // Also rewrite short-form `services.*.volumes.*` host paths + // (`./host:/container[:opts]`). absVolumeMount only handles + // the canonical long form, so the relative dot prefix on a + // short-form entry would otherwise reach the outer pipeline + // unresolved -- format.ParseVolume then fails to detect it as + // a bind path because the path looks like a named volume + // (the joined result drops the leading "."). Use the absolute + // base WorkingDir here so the produced value starts with "/" + // (or a drive letter on Windows) and ParseVolume reliably + // detects it as a bind path. + resolveShortFormVolumeSources(merged, layer.Context.WorkingDir) + } + return merged, nil +} + +// resolveShortFormVolumeSources walks every short-form +// `services.*.volumes.*` scalar in the merged service body and joins +// its host portion (`./host`) with extendsWD so the canonical pass +// later detects it as a bind. Skips named volumes (no leading `.` or +// `~`) and interpolation placeholders (`${...}`) which the outer +// pipeline resolves first. +func resolveShortFormVolumeSources(merged *yaml.Node, extendsWD string) { + if merged == nil || merged.Kind != yaml.MappingNode { + return + } + volumes := mappingValueByKey(merged, "volumes") + if volumes == nil || volumes.Kind != yaml.SequenceNode { + return + } + for _, item := range volumes.Content { + if item == nil || item.Kind != yaml.ScalarNode || item.Value == "" { + continue + } + parts := strings.SplitN(item.Value, ":", 3) + if len(parts) < 2 { + continue + } + src := parts[0] + if !strings.HasPrefix(src, ".") && !strings.HasPrefix(src, "~") { + continue + } + parts[0] = filepath.Join(extendsWD, src) + item.Value = strings.Join(parts, ":") + } +} + +// parseExtendsRef extracts the (service, file) tuple from an extends value +// and fires the "extends" Listener event with a v2-compatible payload so +// downstream consumers (telemetry, dependency analysis) keep observing the +// same callback signature as before the refactor. The short form (a bare +// scalar) names a sibling service; the long form is a mapping with +// required `service` and optional `file`. +func parseExtendsRef(name string, extendsNode *yaml.Node, opts *Options) (string, string, error) { + switch extendsNode.Kind { + case yaml.ScalarNode: + opts.ProcessEvent("extends", map[string]any{"service": extendsNode.Value}) + return extendsNode.Value, "", nil + case yaml.MappingNode: + var ref, file string + payload := map[string]any{} + if r := mappingValueByKey(extendsNode, "service"); r != nil && r.Kind == yaml.ScalarNode { + ref = r.Value + payload["service"] = r.Value + } + if f := mappingValueByKey(extendsNode, "file"); f != nil && f.Kind == yaml.ScalarNode { + file = f.Value + payload["file"] = f.Value + } + opts.ProcessEvent("extends", payload) + if ref == "" { + return "", "", fmt.Errorf("extends.%s.service is required", name) + } + return ref, file, nil + } + return "", "", fmt.Errorf("services.%s.extends must be a string or a mapping", name) +} + +// loadExtendsBaseLayer loads the file referenced by extends.file into a +// stand-alone Layer that carries the file's own SourceContext (working dir, +// environment). The returned layer is meant for a single read of one +// service definition and is discarded after the merge. +// +// Relative paths are resolved through the configured ResourceLoaders, so +// remote loaders (oci://, https://, ...) registered on opts also work for +// extends.file references. +// +// The function also returns child-scoped Options whose ResourceLoaders are +// re-rooted at the extended file's directory. Recursive extends inside the +// loaded layer (extends.file pointing at a sibling file) are then resolved +// against the file's own directory rather than the project root, matching +// v2 getExtendsBaseFromFile behavior. +func loadExtendsBaseLayer(ctx context.Context, parent *node.Layer, file string, opts *Options) (*node.Layer, *Options, error) { + // Resolve extends.file against the *parent layer* working directory so + // extends declared inside an included file pick up the include's own + // project_directory rather than the outer project root. This makes + // nested `include -> extends -> extends.file` work the same way v2 + // does, where each recursive load uses ResourceLoaders pinned to the + // current file's directory. + parentOpts := opts + if parent.Context.WorkingDir != "" { + parentOpts = opts.clone() + parentOpts.ResourceLoaders = append(opts.RemoteResourceLoaders(), localResourceLoader{WorkingDir: parent.Context.WorkingDir}) + } + loader, fullPath, err := resolveResourceWithLoader(ctx, parentOpts, file) + if err != nil { + return nil, nil, err + } + // absLocalDir is the directory of the extended file (always absolute). + // localDir is the relative form returned by loader.Dir, used as the + // base for in-tree path resolution so the resulting paths match the + // v2 relative form (paths look like "testdata/subdir/extra.env" + // rather than absolute paths until the outer pass absolutizes them). + // We store absLocalDir on the SourceContext so chained extends / + // include extends always find files relative to a real directory, + // and keep localDir as a side-table on Options for the per-merge + // path resolution call below. + localDir := loader.Dir(file) + absLocalDir := filepath.Dir(fullPath) + sc := &node.SourceContext{ + File: fullPath, + WorkingDir: absLocalDir, + Environment: parent.Context.Environment, + Parent: parent.Context, + } + childOpts := opts.clone() + childOpts.ResourceLoaders = append(opts.RemoteResourceLoaders(), localResourceLoader{WorkingDir: absLocalDir}) + childOpts.extendsRelativeDir = localDir + layers, err := LoadLayer(ctx, types.ConfigFile{Filename: fullPath}, sc, childOpts) + if err != nil { + return nil, nil, err + } + if len(layers) == 0 { + return nil, nil, fmt.Errorf("extends.file %s yields no document", fullPath) + } + return layers[0], childOpts, nil +} + +// resolveExtendedServicePaths runs path resolution on the merged service +// node using workingDir as the base, mimicking the v2 paths.ResolveRelative +// Paths call inside getExtendsBaseFromFile. Each extends.file level rewrites +// the paths against its own relative dir, so nested extends accumulate the +// expected relative form (sibling.yaml's `.` becomes `testdata/extends` +// when extended from base.yaml which lives there). +func resolveExtendedServicePaths(merged *yaml.Node, workingDir string, opts *Options) error { + if workingDir == "" { + return nil + } + var remotes []paths.RemoteResource + for _, loader := range opts.RemoteResourceLoaders() { + remotes = append(remotes, loader.Accept) + } + // Wrap the merged service node in a synthetic "services.x" mapping so + // the path patterns (which all start at the root) match against it. + wrapper := &yaml.Node{ + Kind: yaml.MappingNode, + Tag: "!!map", + Content: []*yaml.Node{ + {Kind: yaml.ScalarNode, Tag: "!!str", Value: "services"}, + { + Kind: yaml.MappingNode, + Tag: "!!map", + Content: []*yaml.Node{ + {Kind: yaml.ScalarNode, Tag: "!!str", Value: "x"}, + merged, + }, + }, + }, + } + return paths.ResolveRelativePathsNode(wrapper, paths.NodeResolverOptions{ + WorkingDir: workingDir, + Remotes: remotes, + }) +} + +// resetParentPaths removes mapping keys in serviceNode that match a recorded +// !reset / !override path under services.. Mirrors the +// applyNullOverrides traversal v2 does on processor.Apply, but scoped to a +// single service's body so it can run on the cloned base before extends +// merge fires. Returns the list of paths that were consumed so the caller +// can clear them from the layer's master list to avoid double-application +// during the orchestrator post-merge sweep. +func resetParentPaths(serviceNode *yaml.Node, serviceName string, resetPaths []tree.Path) []tree.Path { + if serviceNode == nil || serviceNode.Kind != yaml.MappingNode || len(resetPaths) == 0 { + return nil + } + prefix := tree.NewPath("services", serviceName) + var consumed []tree.Path + for _, p := range resetPaths { + rel := relativePath(p, prefix) + if rel == "" { + continue + } + deleteAtPath(serviceNode, rel) + consumed = append(consumed, p) + } + return consumed +} + +// diffPaths returns the elements of all not present in remove, preserving +// the original order. Used to drop !override paths that have already been +// honored by extends so they don't get re-applied by ApplyResetPaths on the +// merged tree. +func diffPaths(all []tree.Path, remove []tree.Path) []tree.Path { + if len(all) == 0 || len(remove) == 0 { + return all + } + removed := make(map[tree.Path]bool, len(remove)) + for _, r := range remove { + removed[r] = true + } + out := all[:0] + for _, p := range all { + if removed[p] { + continue + } + out = append(out, p) + } + return out +} + +// relativePath returns the portion of p that follows prefix, or "" when p +// is not rooted at prefix. Comparison treats prefix parts as literal (no +// wildcard expansion). +func relativePath(p, prefix tree.Path) tree.Path { + pParts := p.Parts() + prefixParts := prefix.Parts() + if len(pParts) <= len(prefixParts) { + return "" + } + for i, part := range prefixParts { + if pParts[i] != part { + return "" + } + } + return tree.NewPath(pParts[len(prefixParts):]...) +} + +// deleteAtPath removes the entry at a relative path inside n (a Mapping +// Node). Only the first segment is followed at each step; intermediate +// segments must reference Mapping keys, otherwise the function is a no-op. +func deleteAtPath(n *yaml.Node, p tree.Path) { + parts := p.Parts() + if len(parts) == 0 || n == nil { + return + } + if len(parts) == 1 { + deleteMappingKey(n, parts[0]) + return + } + child := mappingValueByKey(n, parts[0]) + deleteAtPath(child, tree.NewPath(parts[1:]...)) +} + +// setMappingValue replaces (or appends) the entry whose key matches in a +// MappingNode. Used by applyServiceExtendsNode to commit the resolved base +// service back into the siblings mapping so further iterations observe the +// updated tree. +func setMappingValue(n *yaml.Node, key string, value *yaml.Node) { + if n == nil || n.Kind != yaml.MappingNode { + return + } + for i := 0; i+1 < len(n.Content); i += 2 { + if n.Content[i].Value == key { + n.Content[i+1] = value + return + } + } + n.Content = append(n.Content, + &yaml.Node{Kind: yaml.ScalarNode, Tag: "!!str", Value: key}, + value, + ) +} + +// mappingValueByKey returns the value Node for a key inside a MappingNode, +// or nil when absent. Shared by the include and extends paths because both +// need to look up service entries inside the services mapping. +func mappingValueByKey(n *yaml.Node, key string) *yaml.Node { + if n == nil || n.Kind != yaml.MappingNode { + return nil + } + for i := 0; i+1 < len(n.Content); i += 2 { + if n.Content[i].Value == key { + return n.Content[i+1] + } + } + return nil +} + +// deleteMappingKey removes the (key, value) pair whose key matches the +// given string from a MappingNode. No-op when the node is not a mapping or +// the key is absent. +func deleteMappingKey(n *yaml.Node, key string) { + if n == nil || n.Kind != yaml.MappingNode { + return + } + for i := 0; i+1 < len(n.Content); i += 2 { + if n.Content[i].Value == key { + n.Content = append(n.Content[:i], n.Content[i+2:]...) + return + } + } +} + +// deepCloneNode returns a structural copy of n with nested Content cloned. +// Used to avoid mutating the base service node while merging it into a +// derived service (the same base may be reused by other extends chains in +// the same load). +func deepCloneNode(n *yaml.Node) *yaml.Node { + if n == nil { + return nil + } + clone := &yaml.Node{ + Kind: n.Kind, + Tag: n.Tag, + Value: n.Value, + Style: n.Style, + Anchor: n.Anchor, + Alias: n.Alias, + Line: n.Line, + Column: n.Column, + HeadComment: n.HeadComment, + LineComment: n.LineComment, + FootComment: n.FootComment, + } + if len(n.Content) > 0 { + clone.Content = make([]*yaml.Node, len(n.Content)) + for i, c := range n.Content { + clone.Content[i] = deepCloneNode(c) + } + } + return clone +} diff --git a/loader/load_extends_test.go b/loader/load_extends_test.go new file mode 100644 index 000000000..867510d7e --- /dev/null +++ b/loader/load_extends_test.go @@ -0,0 +1,186 @@ +/* + Copyright 2020 The Compose Specification Authors. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +*/ + +package loader + +import ( + "context" + "testing" + + "gotest.tools/v3/assert" + + "github.com/compose-spec/compose-go/v3/internal/node" + "github.com/compose-spec/compose-go/v3/types" +) + +func loadParentLayer(t *testing.T, dir, content string) (*node.Layer, *Options) { + t.Helper() + return buildParent(t, dir, types.Mapping{}, content) +} + +func TestApplyExtendsToLayer_SameFile(t *testing.T) { + dir := t.TempDir() + parent, opts := loadParentLayer(t, dir, ` +services: + base: + image: nginx + restart: always + web: + extends: base + command: ["nginx", "-g", "daemon off;"] +`) + assert.NilError(t, ApplyExtendsToLayer(context.TODO(), parent, opts, &cycleTracker{})) + + var m map[string]any + assert.NilError(t, parent.Node.Decode(&m)) + web := m["services"].(map[string]any)["web"].(map[string]any) + // inherited from base + assert.Equal(t, web["image"], "nginx") + assert.Equal(t, web["restart"], "always") + // own override survives + assert.DeepEqual(t, web["command"], []any{"nginx", "-g", "daemon off;"}) + // extends key stripped from result + _, hasExtends := web["extends"] + assert.Assert(t, !hasExtends, "extends key must be removed after merge") +} + +func TestApplyExtendsToLayer_LongFormService(t *testing.T) { + dir := t.TempDir() + parent, opts := loadParentLayer(t, dir, ` +services: + base: + image: nginx + web: + extends: + service: base + restart: always +`) + assert.NilError(t, ApplyExtendsToLayer(context.TODO(), parent, opts, &cycleTracker{})) + + var m map[string]any + assert.NilError(t, parent.Node.Decode(&m)) + web := m["services"].(map[string]any)["web"].(map[string]any) + assert.Equal(t, web["image"], "nginx") + assert.Equal(t, web["restart"], "always") +} + +func TestApplyExtendsToLayer_FromOtherFile(t *testing.T) { + dir := t.TempDir() + writeFile(t, dir, "base.yaml", ` +services: + base: + image: nginx + restart: always +`) + parent, opts := loadParentLayer(t, dir, ` +services: + web: + extends: + file: base.yaml + service: base + command: ["echo", "ok"] +`) + assert.NilError(t, ApplyExtendsToLayer(context.TODO(), parent, opts, &cycleTracker{})) + + var m map[string]any + assert.NilError(t, parent.Node.Decode(&m)) + web := m["services"].(map[string]any)["web"].(map[string]any) + assert.Equal(t, web["image"], "nginx") + assert.Equal(t, web["restart"], "always") + assert.DeepEqual(t, web["command"], []any{"echo", "ok"}) +} + +func TestApplyExtendsToLayer_ChainedExtends(t *testing.T) { + dir := t.TempDir() + parent, opts := loadParentLayer(t, dir, ` +services: + grandparent: + image: nginx + restart: always + parent: + extends: grandparent + environment: + LEVEL: parent + child: + extends: parent + environment: + OWN: child +`) + assert.NilError(t, ApplyExtendsToLayer(context.TODO(), parent, opts, &cycleTracker{})) + + var m map[string]any + assert.NilError(t, parent.Node.Decode(&m)) + child := m["services"].(map[string]any)["child"].(map[string]any) + assert.Equal(t, child["image"], "nginx") + assert.Equal(t, child["restart"], "always") + envSeq := child["environment"].([]any) + // environment is merged as a sequence; both parent and child entries + // must be present (EnforceUnicity would dedupe later if needed). + have := map[string]bool{} + for _, e := range envSeq { + have[e.(string)] = true + } + assert.Assert(t, have["LEVEL=parent"]) + assert.Assert(t, have["OWN=child"]) +} + +func TestApplyExtendsToLayer_DetectsCycle(t *testing.T) { + dir := t.TempDir() + parent, opts := loadParentLayer(t, dir, ` +services: + a: + extends: b + b: + extends: a +`) + err := ApplyExtendsToLayer(context.TODO(), parent, opts, &cycleTracker{}) + assert.ErrorContains(t, err, "Circular reference") +} + +func TestApplyExtendsToLayer_MissingServiceErrors(t *testing.T) { + dir := t.TempDir() + parent, opts := loadParentLayer(t, dir, ` +services: + web: + extends: missing +`) + err := ApplyExtendsToLayer(context.TODO(), parent, opts, &cycleTracker{}) + assert.ErrorContains(t, err, "service \"missing\" not found") +} + +func TestApplyExtendsToLayer_NoServicesBlockIsNoop(t *testing.T) { + dir := t.TempDir() + parent, opts := loadParentLayer(t, dir, ` +networks: + default: + driver: bridge +`) + assert.NilError(t, ApplyExtendsToLayer(context.TODO(), parent, opts, &cycleTracker{})) +} + +func TestApplyExtendsToLayer_NoExtendsLeavesServicesAlone(t *testing.T) { + dir := t.TempDir() + parent, opts := loadParentLayer(t, dir, ` +services: + web: + image: nginx +`) + assert.NilError(t, ApplyExtendsToLayer(context.TODO(), parent, opts, &cycleTracker{})) + var m map[string]any + assert.NilError(t, parent.Node.Decode(&m)) + web := m["services"].(map[string]any)["web"].(map[string]any) + assert.Equal(t, web["image"], "nginx") +} diff --git a/loader/load_include.go b/loader/load_include.go new file mode 100644 index 000000000..95a5f7ef1 --- /dev/null +++ b/loader/load_include.go @@ -0,0 +1,379 @@ +/* + Copyright 2020 The Compose Specification Authors. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +*/ + +package loader + +import ( + "context" + "fmt" + "os" + "path/filepath" + + "go.yaml.in/yaml/v4" + + "github.com/compose-spec/compose-go/v3/dotenv" + "github.com/compose-spec/compose-go/v3/errdefs" + "github.com/compose-spec/compose-go/v3/internal/node" + interp "github.com/compose-spec/compose-go/v3/interpolation" + "github.com/compose-spec/compose-go/v3/paths" + "github.com/compose-spec/compose-go/v3/template" + "github.com/compose-spec/compose-go/v3/types" +) + +// CollectIncludeLayers reads the top-level `include` block from a parent +// layer and returns the list of direct child layers it materializes. Each +// child carries its own SourceContext, capturing the include block's +// `project_directory` and the environment resolved from its `env_file` +// entries — which is what allows the merge / interpolate phases downstream +// to honor per-include context lazily. +// +// The include block is interpolated in the parent's SourceContext before +// any path is resolved, because the path / project_directory / env_file +// scalars themselves may contain ${VAR} references that must be substituted +// in the *parent* environment. This is the one point in the pipeline +// where interpolation is performed eagerly; everywhere else, scalars are +// interpolated after merge in their own SourceContext. +// +// The function only produces *direct* children. The orchestrator is +// responsible for recursing into each child to process its own include +// block. CollectIncludeLayers leaves the parent's `include` mapping entry +// in place; the orchestrator removes it once all children have been +// collected. +// +// CollectIncludeLayers does not perform cross-file merging; it only loads +// included files into stand-alone Layers. Cycle detection is delegated to +// the orchestrator, which keeps a global set of resolved filenames. +func CollectIncludeLayers(ctx context.Context, parent *node.Layer, opts *Options) ([]*node.Layer, error) { + includeNode := layerMappingField(parent.Node, "include") + if includeNode == nil { + return nil, nil + } + if includeNode.Kind != yaml.SequenceNode { + return nil, &errdefs.Diagnostic{ + File: parent.Context.File, + Line: includeNode.Line, + Column: includeNode.Column, + Path: "include", + Cause: errString(fmt.Sprintf("`include` must be a list, got %s", kindName(includeNode.Kind))), + } + } + + if err := interpolateIncludeBlock(includeNode, parent.Context, opts); err != nil { + return nil, err + } + + var layers []*node.Layer + for _, entry := range includeNode.Content { + entryLayers, err := collectOneInclude(ctx, parent, entry, opts) + if err != nil { + return nil, err + } + layers = append(layers, entryLayers...) + } + return layers, nil +} + +// collectOneInclude turns a single include entry (string short form or +// mapping long form) into its corresponding child Layers. Multi-path entries +// produce one Layer per resolved path, in declaration order, the same way +// the v2 ApplyInclude does. +func collectOneInclude(ctx context.Context, parent *node.Layer, entry *yaml.Node, opts *Options) ([]*node.Layer, error) { + cfg, err := readIncludeEntry(entry) + if err != nil { + return nil, diagnoseAt(err, parent.Context.File, entry, "include") + } + + parentWD := parent.Context.WorkingDir + + // Notify listeners that an include directive is being processed, + // matching the v2 ApplyInclude event payload (path is the raw list + // declared by the user, workingdir is the parent layer's working + // dir). Consumers such as the Docker Compose CLI rely on this event + // to count local includes and decide whether a project can be + // published as a self-contained OCI artifact. + opts.ProcessEvent("include", map[string]any{ + "path": cfg.Path, + "workingdir": parentWD, + }) + resolvedPaths, projectDir, err := resolveIncludePaths(ctx, cfg, parentWD, opts) + if err != nil { + return nil, err + } + + envFiles, env, err := resolveIncludeEnvironment(cfg, projectDir, parentWD, parent.Context.Environment) + if err != nil { + return nil, err + } + + childCtx := &node.SourceContext{ + WorkingDir: projectDir, + Environment: env, + EnvFiles: envFiles, + Parent: parent.Context, + } + + var layers []*node.Layer + for _, p := range resolvedPaths { + layerCtx := *childCtx + layerCtx.File = p + fileLayers, err := LoadLayer(ctx, types.ConfigFile{Filename: p}, &layerCtx, opts) + if err != nil { + return nil, err + } + // v2 ApplyInclude always forces ResolvePaths=true for the include + // sub-load. matches that here; subsequent passes in the + // orchestrator short-circuit on already-absolute paths via + // filepath.IsAbs in absScalar, so this single resolution does not + // double up with the outer pass when ResolvePaths is also true. + // + // extends.file is deliberately left untouched: the orchestrator + // extends pass needs the original relative reference so it can + // re-resolve through the loaded layer's ResourceLoader (re-rooted + // at the include working directory in Load), exactly as v2 + // ApplyExtends does inside the recursive loadYamlModel of an + // include. Resolving it here would lead to double-joining when + // the orchestrator runs loader.Load on the already-absolutized + // path. + // v2 ApplyInclude force-runs ResolvePaths=true on the include + // sub-load even when the outer load opted out, so include paths + // become absolute and the outer pass never has to touch them + // again. We only run the sub-resolve when the outer load opted + // in: otherwise leave the include's relative paths untouched so + // `build: .` declared next to the include stays "." after the + // merge (TestIncludeRelative). When skipping, run a lightweight + // cleaning pass so cosmetic forms (`./`, `./foo`) collapse to + // their canonical relative spelling (`.`, `foo`) the same way + // filepath.Join in the v2 sub-resolve would have. + if opts.ResolvePaths { + var remotes []paths.RemoteResource + for _, loader := range opts.RemoteResourceLoaders() { + remotes = append(remotes, loader.Accept) + } + for _, layer := range fileLayers { + if err := paths.ResolveRelativePathsNode(layer.Node, paths.NodeResolverOptions{ + WorkingDirFor: func(_ *yaml.Node) string { + return projectDir + }, + Remotes: remotes, + ExcludePaths: []string{ + "services.*.extends.file", + }, + }); err != nil { + return nil, err + } + if layer.Context != nil { + layer.Context.PathsPreResolved = true + } + } + } else { + for _, layer := range fileLayers { + if err := paths.ResolveRelativePathsNode(layer.Node, paths.NodeResolverOptions{ + WorkingDirFor: func(_ *yaml.Node) string { + return "." + }, + ExcludePaths: []string{ + "services.*.extends.file", + }, + }); err != nil { + return nil, err + } + if layer.Context != nil { + layer.Context.PathsPreResolved = true + } + } + } + layers = append(layers, fileLayers...) + } + return layers, nil +} + +// readIncludeEntry normalizes a single include sequence entry. A bare +// scalar is promoted to a single-path long form; a mapping is decoded +// natively into IncludeConfig via yaml.v4 (StringList now implements +// UnmarshalYAML so short-form path / env_file values are accepted). +func readIncludeEntry(entry *yaml.Node) (types.IncludeConfig, error) { + if entry == nil { + return types.IncludeConfig{}, fmt.Errorf("empty include entry") + } + switch entry.Kind { + case yaml.ScalarNode: + return types.IncludeConfig{Path: types.StringList{entry.Value}}, nil + case yaml.MappingNode: + var cfg types.IncludeConfig + if err := entry.Decode(&cfg); err != nil { + return types.IncludeConfig{}, fmt.Errorf("invalid include entry: %w", err) + } + return cfg, nil + } + return types.IncludeConfig{}, fmt.Errorf("include entry must be a string or a mapping, got %s", kindName(entry.Kind)) +} + +// resolveIncludePaths walks each entry in cfg.Path through the configured +// ResourceLoaders and returns the absolute local paths plus the +// project_directory that applies to the included files. The first path +// defines the project_directory when none is declared; later paths in the +// same entry are treated as overrides loaded from the same directory. +// +// The returned project_directory is the absolute path to the include's +// project root. The per-include path resolution pass uses it as the +// WorkingDir to absolutize relative paths inside the included tree, and +// the PathsPreResolved flag set on the layer's SourceContext prevents the +// orchestrator outer pass from re-resolving them. +func resolveIncludePaths(ctx context.Context, cfg types.IncludeConfig, parentWD string, opts *Options) ([]string, string, error) { + var resolved []string + projectDir := cfg.ProjectDirectory + for i, p := range cfg.Path { + _, fullPath, err := resolveResourceWithLoader(ctx, opts, p) + if err != nil { + return nil, "", err + } + if i == 0 { + switch { + case projectDir == "": + projectDir = filepath.Dir(fullPath) + case !filepath.IsAbs(projectDir): + projectDir = filepath.Join(parentWD, projectDir) + } + } + resolved = append(resolved, fullPath) + } + return resolved, projectDir, nil +} + +// resolveIncludeEnvironment loads the env_file(s) declared on the include +// block and merges them on top of the parent environment. Relative env_file +// paths are resolved against parentWD (matching v2 behavior); a single +// `/dev/null` entry disables environment inheritance for that file. +// +// When cfg.EnvFile is empty, an implicit `/.env` is used +// if it exists — same convention as v2. +func resolveIncludeEnvironment(cfg types.IncludeConfig, projectDir, parentWD string, parentEnv types.Mapping) ([]string, types.Mapping, error) { + envFiles := []string{} + if len(cfg.EnvFile) == 0 { + f := filepath.Join(projectDir, ".env") + if s, err := os.Stat(f); err == nil && !s.IsDir() { + envFiles = []string{f} + } + } else { + for _, f := range cfg.EnvFile { + if f == "/dev/null" { + continue + } + if !filepath.IsAbs(f) { + f = filepath.Join(parentWD, f) + } + s, err := os.Stat(f) + if err != nil { + return nil, nil, err + } + if s.IsDir() { + return nil, nil, fmt.Errorf("%s is not a file", f) + } + envFiles = append(envFiles, f) + } + } + + envFromFile, err := dotenv.GetEnvFromFile(parentEnv, envFiles) + if err != nil { + return nil, nil, err + } + merged := parentEnv.Clone().Merge(envFromFile) + return envFiles, merged, nil +} + +// resolveResourceWithLoader finds the ResourceLoader in opts that accepts +// p and returns it together with the resolved absolute path produced by +// its Load method. Mirrors the v2 dispatch logic inside ApplyInclude and +// is the only resource-lookup helper kept because every caller needs +// the loader handle for follow-up loader.Dir computations. +func resolveResourceWithLoader(ctx context.Context, opts *Options, p string) (ResourceLoader, string, error) { + for _, loader := range opts.ResourceLoaders { + if !loader.Accept(p) { + continue + } + full, err := loader.Load(ctx, p) + if err != nil { + return nil, "", err + } + return loader, full, nil + } + return nil, "", fmt.Errorf("no ResourceLoader accepted %q", p) +} + +// interpolateIncludeBlock runs InterpolateNode on the include sub-tree with +// the parent SourceContext. This is the one place in the pipeline where +// interpolation is eager: the include path / project_directory / env_file +// scalars must be substituted before paths are resolved, otherwise the +// loader has no way to find the referenced files. +func interpolateIncludeBlock(includeNode *yaml.Node, sc *node.SourceContext, opts *Options) error { + if opts != nil && opts.SkipInterpolation { + return nil + } + lookup := func(key string) (string, bool) { + if sc == nil { + return "", false + } + v, ok := sc.Environment[key] + return v, ok + } + substitute := template.Substitute + if opts != nil && opts.Interpolate != nil && opts.Interpolate.Substitute != nil { + substitute = opts.Interpolate.Substitute + } + return interp.InterpolateNode(includeNode, interp.NodeOptions{ + LookupValue: lookup, + Substitute: substitute, + }) +} + +// layerMappingField returns the value Node for key inside a Layer's root +// mapping, or nil when absent / not a mapping. +func layerMappingField(root *yaml.Node, key string) *yaml.Node { + if root == nil { + return nil + } + r := root + if r.Kind == yaml.DocumentNode && len(r.Content) == 1 { + r = r.Content[0] + } + if r.Kind != yaml.MappingNode { + return nil + } + for i := 0; i+1 < len(r.Content); i += 2 { + if r.Content[i].Value == key { + return r.Content[i+1] + } + } + return nil +} + +// kindName returns a human-readable label for a yaml.Kind, used in error +// messages. yaml.v4 exposes the constants but no String() helper. +func kindName(k yaml.Kind) string { + switch k { + case yaml.DocumentNode: + return "document" + case yaml.MappingNode: + return "mapping" + case yaml.SequenceNode: + return "sequence" + case yaml.ScalarNode: + return "scalar" + case yaml.AliasNode: + return "alias" + } + return "unknown" +} diff --git a/loader/load_include_test.go b/loader/load_include_test.go new file mode 100644 index 000000000..a436957fd --- /dev/null +++ b/loader/load_include_test.go @@ -0,0 +1,190 @@ +/* + Copyright 2020 The Compose Specification Authors. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +*/ + +package loader + +import ( + "context" + "os" + "path/filepath" + "testing" + + "gotest.tools/v3/assert" + + "github.com/compose-spec/compose-go/v3/internal/node" + "github.com/compose-spec/compose-go/v3/types" +) + +func writeFile(t *testing.T, dir, name, content string) string { + t.Helper() + path := filepath.Join(dir, name) + assert.NilError(t, os.MkdirAll(filepath.Dir(path), 0o755)) + assert.NilError(t, os.WriteFile(path, []byte(content), 0o644)) + return path +} + +func buildParent(t *testing.T, workingDir string, env types.Mapping, content string) (*node.Layer, *Options) { + t.Helper() + parentPath := writeFile(t, workingDir, "compose.yaml", content) + sc := &node.SourceContext{ + File: parentPath, + WorkingDir: workingDir, + Environment: env, + } + opts := &Options{ + ResourceLoaders: []ResourceLoader{localResourceLoader{WorkingDir: workingDir}}, + } + layers, err := LoadLayer(context.TODO(), types.ConfigFile{Filename: parentPath}, sc, opts) + assert.NilError(t, err) + assert.Equal(t, len(layers), 1) + return layers[0], opts +} + +func TestCollectIncludeLayers_NoBlockYieldsEmpty(t *testing.T) { + dir := t.TempDir() + parent, opts := buildParent(t, dir, types.Mapping{}, ` +services: + web: + image: nginx +`) + got, err := CollectIncludeLayers(context.TODO(), parent, opts) + assert.NilError(t, err) + assert.Equal(t, len(got), 0) +} + +func TestCollectIncludeLayers_ShortFormString(t *testing.T) { + dir := t.TempDir() + writeFile(t, dir, "included.yaml", ` +services: + api: + image: caddy +`) + parent, opts := buildParent(t, dir, types.Mapping{}, ` +include: + - included.yaml +services: + web: + image: nginx +`) + got, err := CollectIncludeLayers(context.TODO(), parent, opts) + assert.NilError(t, err) + assert.Equal(t, len(got), 1) + + // The included layer's WorkingDir defaults to the absolute directory + // of the included file. prefers absolute paths + // form so the per-scalar path resolution in the orchestrator can fall + // back to the layer working directory without an extra rebasing step. + assert.Equal(t, got[0].Context.WorkingDir, dir) + assert.Equal(t, got[0].Context.File, filepath.Join(dir, "included.yaml")) + // Parent chain is preserved for diagnostics. + assert.Equal(t, got[0].Context.Parent, parent.Context) + + var m map[string]any + assert.NilError(t, got[0].Node.Decode(&m)) + assert.Equal(t, m["services"].(map[string]any)["api"].(map[string]any)["image"], "caddy") +} + +func TestCollectIncludeLayers_ProjectDirectoryRedefined(t *testing.T) { + root := t.TempDir() + subdir := filepath.Join(root, "sub") + writeFile(t, subdir, "compose.yaml", ` +services: + api: + image: caddy +`) + parent, opts := buildParent(t, root, types.Mapping{}, ` +include: + - path: sub/compose.yaml + project_directory: sub +`) + got, err := CollectIncludeLayers(context.TODO(), parent, opts) + assert.NilError(t, err) + assert.Equal(t, len(got), 1) + assert.Equal(t, got[0].Context.WorkingDir, subdir, + "project_directory under sub/ resolved against parent working dir") +} + +func TestCollectIncludeLayers_EnvFileLoaded(t *testing.T) { + root := t.TempDir() + writeFile(t, root, ".env.parent", "TAG=2.0\n") + writeFile(t, root, "included.yaml", ` +services: + api: + image: caddy:${TAG} +`) + parent, opts := buildParent(t, root, types.Mapping{}, ` +include: + - path: included.yaml + env_file: + - .env.parent +`) + got, err := CollectIncludeLayers(context.TODO(), parent, opts) + assert.NilError(t, err) + assert.Equal(t, len(got), 1) + assert.Equal(t, got[0].Context.Environment["TAG"], "2.0", + "env_file scoped to the include is merged into the child SourceContext") + // The included layer's image scalar still carries ${TAG} — interpolation + // will fire later on the merged tree, using this SourceContext. + var m map[string]any + assert.NilError(t, got[0].Node.Decode(&m)) + assert.Equal(t, m["services"].(map[string]any)["api"].(map[string]any)["image"], "caddy:${TAG}", + "included layer is not eagerly interpolated; substitution defers to the merge phase") +} + +func TestCollectIncludeLayers_InterpolatesIncludeBlockPaths(t *testing.T) { + root := t.TempDir() + writeFile(t, root, "included.yaml", ` +services: + api: + image: caddy +`) + env := types.Mapping{"FILE": "included.yaml"} + parent, opts := buildParent(t, root, env, ` +include: + - ${FILE} +`) + got, err := CollectIncludeLayers(context.TODO(), parent, opts) + assert.NilError(t, err) + assert.Equal(t, len(got), 1) + assert.Equal(t, got[0].Context.File, filepath.Join(root, "included.yaml"), + "include path scalar interpolated in parent context before loading") +} + +func TestCollectIncludeLayers_RejectsNonListIncludeBlock(t *testing.T) { + dir := t.TempDir() + parent, opts := buildParent(t, dir, types.Mapping{}, ` +include: included.yaml +`) + _, err := CollectIncludeLayers(context.TODO(), parent, opts) + assert.ErrorContains(t, err, "`include` must be a list") +} + +func TestCollectIncludeLayers_DevNullDisablesEnvInheritance(t *testing.T) { + root := t.TempDir() + writeFile(t, root, ".env.parent", "TAG=2.0\n") + writeFile(t, root, "included.yaml", `services: {api: {image: caddy}}`) + parent, opts := buildParent(t, root, types.Mapping{"TAG": "1.0"}, ` +include: + - path: included.yaml + env_file: + - /dev/null +`) + got, err := CollectIncludeLayers(context.TODO(), parent, opts) + assert.NilError(t, err) + assert.Equal(t, len(got), 1) + // No env_file actually loaded; child env is just the parent env clone. + assert.Equal(t, got[0].Context.Environment["TAG"], "1.0") +} diff --git a/loader/load_layer.go b/loader/load_layer.go new file mode 100644 index 000000000..a35bbc0e6 --- /dev/null +++ b/loader/load_layer.go @@ -0,0 +1,182 @@ +/* + Copyright 2020 The Compose Specification Authors. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +*/ + +package loader + +import ( + "bytes" + "context" + "errors" + "fmt" + "io" + "os" + + "go.yaml.in/yaml/v4" + + "github.com/compose-spec/compose-go/v3/consts" + "github.com/compose-spec/compose-go/v3/internal/node" + "github.com/compose-spec/compose-go/v3/types" +) + +// LoadLayer parses a single ConfigFile into one or more node.Layer values, +// each carrying a *yaml.Node tree and the SourceContext that produced it. +// +// The function is the per-file parse stage. +// It performs only the steps that turn raw YAML bytes into a clean, +// alias-free Node tree: +// +// 1. read file content (or use file.Content / file.Node when provided); +// 2. decode each YAML document into a *yaml.Node (multi-document files +// produce one Layer per document, in source order); +// 3. resolve !reset and !override tags via node.ResolveResetOverride, +// recording their paths on the Layer for later replay; +// 4. unfold YAML aliases and fold `<<` merge keys via node.NormalizeAliases +// so the resulting tree is self-contained and safe to merge across files. +// +// Cross-file merge, include/extends resolution, interpolation, transform, +// path resolution, validation and decoding to types.Project are performed +// by the orchestrator in subsequent commits and are out of scope here. +// +// LoadLayer does not touch the network or load any included file; it +// operates on a single ConfigFile in isolation. +func LoadLayer(ctx context.Context, file types.ConfigFile, sc *node.SourceContext, opts *Options) ([]*node.Layer, error) { + // ctx is reserved for orchestrator commits that will wire cancellation + // through ResourceLoaders and remote include / extends fetches. + _ = ctx + // consts.ComposeFileKey is referenced so future orchestrator commits can + // re-introduce ctx telemetry without adding a fresh import. + _ = consts.ComposeFileKey{} + + content, err := readConfigFileContent(file) + if err != nil { + return nil, err + } + + maxVisits := 0 + if opts != nil { + maxVisits = opts.MaxNodeVisits + } + + if file.Node != nil { + // Caller already produced the parsed Node; honor it as a single + // "document" layer without re-parsing the bytes. + return processLayer(file.Node, sc, maxVisits) + } + + dec := yaml.NewDecoder(bytes.NewReader(content)) + var layers []*node.Layer + for { + var doc yaml.Node + if err := dec.Decode(&doc); err != nil { + if errors.Is(err, io.EOF) { + break + } + return nil, fmt.Errorf("failed to parse %s: %w", file.Filename, err) + } + ls, err := processLayer(&doc, sc, maxVisits) + if err != nil { + return nil, err + } + layers = append(layers, ls...) + } + return layers, nil +} + +// processLayer applies the per-document Node transformations (reset/override +// resolution and alias normalization) and wraps the result in a Layer. +// A single yaml.Document may produce zero or one Layer depending on whether +// the document body resolves to a non-nil tree. +func processLayer(doc *yaml.Node, sc *node.SourceContext, maxVisits int) ([]*node.Layer, error) { + resolved, resetPaths, err := node.ResolveResetOverride(doc, maxVisits) + if err != nil { + return nil, err + } + if resolved == nil { + return nil, nil + } + // Reject documents whose top-level is not a mapping so the v2-compatible + // error message surfaces before the downstream pipeline tries to decode + // the tree into a map[string]any and panics with a generic yaml error. + if resolved.Kind != yaml.MappingNode { + return nil, errors.New("top-level object must be a mapping") + } + if err := node.NormalizeAliases(resolved); err != nil { + return nil, err + } + // Reject non-string keys throughout the tree: yaml.v4 accepts + // non-string scalar keys (e.g. integers), but every downstream + // consumer assumes string keys. Runs after NormalizeAliases so the + // merge-key marker (`<<`) used by anchor expansion is folded away + // before we walk for the diagnostic. + if err := checkStringKeys(resolved, "top level"); err != nil { + return nil, err + } + layer := node.NewLayer(resolved, sc) + layer.SetResetPaths(resetPaths) + return []*node.Layer{layer}, nil +} + +// checkStringKeys walks a yaml.Node tree depth-first and returns the first +// non-string mapping key it encounters. The path string mirrors the v2 +// diagnostic format ("services", "networks.default.ipam.config[0]", ...) +// so existing fixture tests keep their error-content assertions stable. +func checkStringKeys(n *yaml.Node, currentPath string) error { + if n == nil { + return nil + } + switch n.Kind { + case yaml.MappingNode: + for i := 0; i+1 < len(n.Content); i += 2 { + key := n.Content[i] + value := n.Content[i+1] + if key.Kind != yaml.ScalarNode || (key.Tag != "" && key.Tag != "!!str") { + preposition := "in" + if currentPath == "top level" { + preposition = "at" + } + return fmt.Errorf("non-string key %s %s: %s", preposition, currentPath, key.Value) + } + var next string + if currentPath == "top level" { + next = key.Value + } else { + next = currentPath + "." + key.Value + } + if err := checkStringKeys(value, next); err != nil { + return err + } + } + case yaml.SequenceNode: + for i, c := range n.Content { + if err := checkStringKeys(c, fmt.Sprintf("%s[%d]", currentPath, i)); err != nil { + return err + } + } + } + return nil +} + +// readConfigFileContent returns the raw YAML bytes for a ConfigFile, +// reading from disk when neither Content nor a pre-parsed Node is provided. +func readConfigFileContent(file types.ConfigFile) ([]byte, error) { + if file.Node != nil || file.Content != nil { + return file.Content, nil + } + if file.Filename == "" { + return nil, errors.New("ConfigFile has neither Filename nor Content nor Node") + } + return os.ReadFile(file.Filename) +} diff --git a/loader/load_layer_test.go b/loader/load_layer_test.go new file mode 100644 index 000000000..4489150c1 --- /dev/null +++ b/loader/load_layer_test.go @@ -0,0 +1,193 @@ +/* + Copyright 2020 The Compose Specification Authors. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +*/ + +package loader + +import ( + "context" + "os" + "path/filepath" + "testing" + + "go.yaml.in/yaml/v4" + "gotest.tools/v3/assert" + + "github.com/compose-spec/compose-go/v3/internal/node" + "github.com/compose-spec/compose-go/v3/types" +) + +func sourceCtx(workingDir string) *node.SourceContext { + return &node.SourceContext{ + File: "test.yaml", + WorkingDir: workingDir, + Environment: types.Mapping{}, + } +} + +func TestLoadLayer_FromContent(t *testing.T) { + file := types.ConfigFile{ + Filename: "(inline)", + Content: []byte(` +services: + web: + image: nginx +`), + } + layers, err := LoadLayer(context.TODO(), file, sourceCtx("/work"), &Options{}) + assert.NilError(t, err) + assert.Equal(t, len(layers), 1) + assert.Equal(t, layers[0].Context.WorkingDir, "/work") + assert.Equal(t, layers[0].Node.Kind, yaml.MappingNode) + + var m map[string]any + assert.NilError(t, layers[0].Node.Decode(&m)) + assert.Equal(t, m["services"].(map[string]any)["web"].(map[string]any)["image"], "nginx") +} + +func TestLoadLayer_FromFile(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, "compose.yaml") + assert.NilError(t, os.WriteFile(path, []byte(`name: from-file +services: + api: + image: alpine +`), 0o644)) + + layers, err := LoadLayer(context.TODO(), types.ConfigFile{Filename: path}, sourceCtx(dir), &Options{}) + assert.NilError(t, err) + assert.Equal(t, len(layers), 1) + var m map[string]any + assert.NilError(t, layers[0].Node.Decode(&m)) + assert.Equal(t, m["name"], "from-file") +} + +func TestLoadLayer_UnfoldsAliasesAndMergeKeys(t *testing.T) { + file := types.ConfigFile{ + Filename: "(inline)", + Content: []byte(` +defaults: &defaults + image: nginx + restart: always +services: + web: + <<: *defaults + image: caddy +`), + } + layers, err := LoadLayer(context.TODO(), file, sourceCtx("/work"), &Options{}) + assert.NilError(t, err) + assert.Equal(t, len(layers), 1) + + // After NormalizeAliases: no AliasNode, no `<<` key. Surrounding wins. + var sawAlias, sawMergeKey bool + var visit func(*yaml.Node) + visit = func(n *yaml.Node) { + if n == nil { + return + } + if n.Kind == yaml.AliasNode { + sawAlias = true + } + if n.Kind == yaml.MappingNode { + for i := 0; i+1 < len(n.Content); i += 2 { + if n.Content[i].Value == "<<" { + sawMergeKey = true + } + visit(n.Content[i+1]) + } + return + } + for _, c := range n.Content { + visit(c) + } + } + visit(layers[0].Node) + assert.Assert(t, !sawAlias, "no alias should remain") + assert.Assert(t, !sawMergeKey, "no merge key should remain") + + var m map[string]any + assert.NilError(t, layers[0].Node.Decode(&m)) + web := m["services"].(map[string]any)["web"].(map[string]any) + assert.Equal(t, web["image"], "caddy") + assert.Equal(t, web["restart"], "always") +} + +func TestLoadLayer_CollectsResetPaths(t *testing.T) { + file := types.ConfigFile{ + Filename: "(inline)", + Content: []byte(` +services: + web: + image: nginx + command: !reset null +`), + } + layers, err := LoadLayer(context.TODO(), file, sourceCtx("/work"), &Options{}) + assert.NilError(t, err) + paths := layers[0].ResetPaths() + assert.Equal(t, len(paths), 1) + assert.Equal(t, paths[0].String(), "services.web.command") +} + +func TestLoadLayer_MultiDocument(t *testing.T) { + file := types.ConfigFile{ + Filename: "(inline)", + Content: []byte(` +name: first +--- +name: second +`), + } + layers, err := LoadLayer(context.TODO(), file, sourceCtx("/work"), &Options{}) + assert.NilError(t, err) + assert.Equal(t, len(layers), 2) + + var m1, m2 map[string]any + assert.NilError(t, layers[0].Node.Decode(&m1)) + assert.NilError(t, layers[1].Node.Decode(&m2)) + assert.Equal(t, m1["name"], "first") + assert.Equal(t, m2["name"], "second") +} + +func TestLoadLayer_FromPrebuiltNode(t *testing.T) { + // Build a yaml.Node by parsing then passing it as ConfigFile.Node. + var doc yaml.Node + assert.NilError(t, yaml.Unmarshal([]byte("name: pre-parsed\n"), &doc)) + + layers, err := LoadLayer(context.TODO(), types.ConfigFile{Node: &doc}, sourceCtx("/work"), &Options{}) + assert.NilError(t, err) + assert.Equal(t, len(layers), 1) + var m map[string]any + assert.NilError(t, layers[0].Node.Decode(&m)) + assert.Equal(t, m["name"], "pre-parsed") +} + +func TestLoadLayer_RejectsAliasBomb(t *testing.T) { + // Document under MaxNodeVisits cap to verify the resolver propagates + // its error through LoadLayer. + file := types.ConfigFile{ + Filename: "(inline)", + Content: []byte(` +services: + web: + extends: &self {service: web} + <<: *self +`), + } + // Lower the cap so the resolver fires. + _, err := LoadLayer(context.TODO(), file, sourceCtx("/work"), &Options{MaxNodeVisits: 3}) + assert.ErrorContains(t, err, "exceeds maximum node visit limit") +} diff --git a/loader/load_test.go b/loader/load_test.go new file mode 100644 index 000000000..e2ccd3b05 --- /dev/null +++ b/loader/load_test.go @@ -0,0 +1,243 @@ +/* + Copyright 2020 The Compose Specification Authors. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +*/ + +package loader + +import ( + "context" + "path/filepath" + "testing" + + "gotest.tools/v3/assert" + + "github.com/compose-spec/compose-go/v3/types" +) + +// loadMap runs load and decodes the returned tree into a +// map[string]any so the existing test assertions can keep navigating it +// in dict form. +func loadMap(t *testing.T, cd types.ConfigDetails, opts *Options) (map[string]any, error) { + t.Helper() + root, err := load(context.TODO(), &cd, opts) + if err != nil { + return nil, err + } + var dict map[string]any + if err := root.Decode(&dict); err != nil { + return nil, err + } + return dict, nil +} + +func loadConfig(t *testing.T, dir string, files ...string) types.ConfigDetails { + t.Helper() + cfgFiles := make([]types.ConfigFile, len(files)) + for i, name := range files { + cfgFiles[i] = types.ConfigFile{Filename: filepath.Join(dir, name)} + } + return types.ConfigDetails{ + WorkingDir: dir, + ConfigFiles: cfgFiles, + Environment: types.Mapping{}, + } +} + +func TestLoad_SingleFileBasic(t *testing.T) { + dir := t.TempDir() + writeFile(t, dir, "compose.yaml", ` +services: + web: + image: nginx +`) + dict, err := loadMap(t, loadConfig(t, dir, "compose.yaml"), &Options{ + SkipNormalization: true, + SkipValidation: true, + SkipConsistencyCheck: true, + }) + assert.NilError(t, err) + web := dict["services"].(map[string]any)["web"].(map[string]any) + assert.Equal(t, web["image"], "nginx") +} + +func TestLoad_MultiFileMergeLeftToRight(t *testing.T) { + dir := t.TempDir() + writeFile(t, dir, "base.yaml", ` +services: + web: + image: nginx + restart: always +`) + writeFile(t, dir, "override.yaml", ` +services: + web: + image: caddy +`) + dict, err := loadMap(t, loadConfig(t, dir, "base.yaml", "override.yaml"), &Options{ + SkipNormalization: true, + SkipValidation: true, + SkipConsistencyCheck: true, + }) + assert.NilError(t, err) + web := dict["services"].(map[string]any)["web"].(map[string]any) + assert.Equal(t, web["image"], "caddy", "later file overrides base") + assert.Equal(t, web["restart"], "always", "base value preserved") +} + +func TestLoad_LazyInterpolationAcrossInclude(t *testing.T) { + // The headline demonstration: an env_file declared on the include + // block introduces variables that are only visible to scalars from the + // included file. The parent file keeps the variables of its own shell + // environment. Same merged tree, two scopes. + // + // The semantics match v2's Mapping.Merge: existing keys (from the + // shell environment) win over env_file entries, so this test relies on + // API_TAG being defined ONLY in the env_file and WEB_TAG being defined + // ONLY in the shell environment. + root := t.TempDir() + writeFile(t, root, ".env.parent", "API_TAG=2.0\n") + writeFile(t, root, "included.yaml", ` +services: + api: + image: caddy:${API_TAG} +`) + writeFile(t, root, "compose.yaml", ` +include: + - path: included.yaml + env_file: + - .env.parent +services: + web: + image: nginx:${WEB_TAG} +`) + cd := loadConfig(t, root, "compose.yaml") + cd.Environment = types.Mapping{"WEB_TAG": "root-1.0"} + dict, err := loadMap(t, cd, &Options{ + SkipNormalization: true, + SkipValidation: true, + SkipConsistencyCheck: true, + }) + assert.NilError(t, err) + // api inherits API_TAG from the include block env_file. + api := dict["services"].(map[string]any)["api"].(map[string]any) + assert.Equal(t, api["image"], "caddy:2.0", + "included scalar interpolated in include SourceContext") + // web uses WEB_TAG from the shell environment (parent context). + web := dict["services"].(map[string]any)["web"].(map[string]any) + assert.Equal(t, web["image"], "nginx:root-1.0", + "parent scalar interpolated in parent SourceContext") +} + +func TestLoad_ExtendsSameFile(t *testing.T) { + dir := t.TempDir() + writeFile(t, dir, "compose.yaml", ` +services: + base: + image: nginx + restart: always + web: + extends: base +`) + dict, err := loadMap(t, loadConfig(t, dir, "compose.yaml"), &Options{ + SkipNormalization: true, + SkipValidation: true, + SkipConsistencyCheck: true, + }) + assert.NilError(t, err) + web := dict["services"].(map[string]any)["web"].(map[string]any) + assert.Equal(t, web["image"], "nginx") + assert.Equal(t, web["restart"], "always") + _, hasExtends := web["extends"] + assert.Assert(t, !hasExtends, "extends key stripped after merge") +} + +func TestLoad_ResetTagApplied(t *testing.T) { + dir := t.TempDir() + writeFile(t, dir, "base.yaml", ` +services: + web: + image: nginx + command: ["nginx"] +`) + writeFile(t, dir, "override.yaml", ` +services: + web: + command: !reset null +`) + dict, err := loadMap(t, loadConfig(t, dir, "base.yaml", "override.yaml"), &Options{ + SkipNormalization: true, + SkipValidation: true, + SkipConsistencyCheck: true, + }) + assert.NilError(t, err) + web := dict["services"].(map[string]any)["web"].(map[string]any) + _, hasCommand := web["command"] + assert.Assert(t, !hasCommand, "command stripped by !reset") + assert.Equal(t, web["image"], "nginx") +} + +func TestLoad_PathResolutionPerInclude(t *testing.T) { + // Different relative paths in parent vs included file must resolve + // against their own working dirs. + root := t.TempDir() + subdir := filepath.Join(root, "sub") + writeFile(t, subdir, "compose.yaml", ` +services: + api: + build: + context: ./local-app +`) + writeFile(t, root, "compose.yaml", ` +include: + - path: sub/compose.yaml + project_directory: sub +services: + web: + build: + context: ./root-app +`) + dict, err := loadMap(t, loadConfig(t, root, "compose.yaml"), &Options{ + SkipNormalization: true, + SkipValidation: true, + SkipConsistencyCheck: true, + ResolvePaths: true, + }) + assert.NilError(t, err) + web := dict["services"].(map[string]any)["web"].(map[string]any) + api := dict["services"].(map[string]any)["api"].(map[string]any) + assert.Equal(t, + web["build"].(map[string]any)["context"], + filepath.Join(root, "root-app"), + "parent scalar resolved against project root") + assert.Equal(t, + api["build"].(map[string]any)["context"], + filepath.Join(subdir, "local-app"), + "included scalar resolved against include project_directory") +} + +func TestLoad_EmptyConfigRejected(t *testing.T) { + // load rejects an empty input rather than silently producing an + // empty tree. + cd := types.ConfigDetails{ + WorkingDir: "/work", + Environment: types.Mapping{}, + } + _, err := load(context.TODO(), &cd, &Options{ + SkipNormalization: true, + SkipValidation: true, + SkipConsistencyCheck: true, + }) + assert.ErrorContains(t, err, "empty compose file") +} diff --git a/loader/loader.go b/loader/loader.go index 3ab4fbc65..792dc0592 100644 --- a/loader/loader.go +++ b/loader/loader.go @@ -27,21 +27,13 @@ import ( "reflect" "regexp" "slices" - "strconv" "strings" "github.com/compose-spec/compose-go/v3/consts" "github.com/compose-spec/compose-go/v3/errdefs" interp "github.com/compose-spec/compose-go/v3/interpolation" - "github.com/compose-spec/compose-go/v3/override" - "github.com/compose-spec/compose-go/v3/paths" - "github.com/compose-spec/compose-go/v3/schema" "github.com/compose-spec/compose-go/v3/template" - "github.com/compose-spec/compose-go/v3/transform" - "github.com/compose-spec/compose-go/v3/tree" "github.com/compose-spec/compose-go/v3/types" - "github.com/compose-spec/compose-go/v3/validation" - "github.com/go-viper/mapstructure/v2" "github.com/sirupsen/logrus" "go.yaml.in/yaml/v4" ) @@ -95,6 +87,35 @@ type Options struct { // MaxNodeVisits caps total YAML node visits during reset/override resolution. // Zero means use the default. Useful for very large compose files that exceed the default cap. MaxNodeVisits int + + // Diagnostics opts in to per-path source position tracking. When + // true, the resulting *types.Project carries a populated Sources + // map (path -> file:line:column) so tooling can surface the source + // location of any compose value (validation errors, schema misses, + // dependency warnings, ...). Defaults to off so the project shape + // stays the same for callers that did not opt in. + Diagnostics bool + + // pathPositions is the snapshot of dotted compose path -> source + // position captured pre-canonical by load() when Diagnostics is on. + // nodeToProject converts it into Project.Sources at the end of the + // pipeline. Unexported so callers cannot mutate it directly. + pathPositions map[string]nodePosition + + // envFileScopes captures, during Load, the layer Environment in + // effect when each env_file entry was declared. The map is keyed by + // the resolved absolute env_file path and consumed by ModelToProject + // to populate EnvFile.Env, which WithServicesEnvironmentResolved + // then uses as the preferred interpolation scope. + envFileScopes map[string]types.Mapping + + // extendsRelativeDir carries the v2-compatible relative project + // directory recorded by loadExtendsBaseLayer for the path resolution + // that runs on the merged service body. SourceContext.WorkingDir + // remains absolute (chained extends.file lookups require it), so + // this side-table keeps the v2 relative form available without + // regressing the absolute lookup path. + extendsRelativeDir string } var versionWarning []string @@ -263,6 +284,14 @@ func WithSkipValidation(opts *Options) { opts.SkipValidation = true } +// WithDiagnostics turns per-path source position tracking on. The +// returned *types.Project will carry a populated Sources map keyed by +// dotted compose path. Tooling that wants to surface "this error +// happened at file:line:col" needs this opt-in. +func WithDiagnostics(opts *Options) { + opts.Diagnostics = true +} + // WithProfiles sets profiles to be activated func WithProfiles(profiles []string) func(*Options) { return func(opts *Options) { @@ -352,31 +381,32 @@ func LoadConfigFiles(ctx context.Context, configFiles []string, workingDir strin // LoadWithContext reads a ConfigDetails and returns a fully loaded configuration as a compose-go Project func LoadWithContext(ctx context.Context, configDetails types.ConfigDetails, options ...func(*Options)) (*types.Project, error) { opts := ToOptions(&configDetails, options) - dict, err := loadModelWithContext(ctx, &configDetails, opts) + if len(configDetails.ConfigFiles) < 1 { + return nil, errors.New("no compose file specified") + } + // Capture Load's mutation of cd.Environment (COMPOSE_PROJECT_NAME) + // so nodeToProject sees the same environment that scalar + // interpolation observed during the pipeline. + cd := configDetails + root, err := load(ctx, &cd, opts) if err != nil { return nil, err } - return ModelToProject(dict, opts, configDetails) + return nodeToProject(root, opts, cd) } // LoadModelWithContext reads a ConfigDetails and returns a fully loaded configuration as a yaml dictionary func LoadModelWithContext(ctx context.Context, configDetails types.ConfigDetails, options ...func(*Options)) (map[string]any, error) { opts := ToOptions(&configDetails, options) - return loadModelWithContext(ctx, &configDetails, opts) -} - -// LoadModelWithContext reads a ConfigDetails and returns a fully loaded configuration as a yaml dictionary -func loadModelWithContext(ctx context.Context, configDetails *types.ConfigDetails, opts *Options) (map[string]any, error) { if len(configDetails.ConfigFiles) < 1 { return nil, errors.New("no compose file specified") } - - err := projectName(configDetails, opts) + cd := configDetails + root, err := load(ctx, &cd, opts) if err != nil { return nil, err } - - return load(ctx, *configDetails, opts, nil) + return nodeToModel(root) } func ToOptions(configDetails *types.ConfigDetails, options []func(*Options)) *Options { @@ -396,217 +426,50 @@ func ToOptions(configDetails *types.ConfigDetails, options []func(*Options)) *Op return opts } -func loadYamlModel(ctx context.Context, config types.ConfigDetails, opts *Options, ct *cycleTracker, included []string) (map[string]interface{}, error) { - var ( - dict = map[string]interface{}{} - err error - ) - workingDir, environment := config.WorkingDir, config.Environment - - for _, file := range config.ConfigFiles { - dict, _, err = loadYamlFile(ctx, file, opts, workingDir, environment, ct, dict, included) - if err != nil { - return nil, err - } - } - - if !opts.SkipDefaultValues { - dict, err = transform.SetDefaultValues(dict) - if err != nil { - return nil, err - } - } - - if !opts.SkipValidation { - if err := validation.Validate(dict); err != nil { - return nil, err - } - } - - if opts.ResolvePaths { - var remotes []paths.RemoteResource - for _, loader := range opts.RemoteResourceLoaders() { - remotes = append(remotes, loader.Accept) - } - err = paths.ResolveRelativePaths(dict, config.WorkingDir, remotes) - if err != nil { - return nil, err - } - } - ResolveEnvironment(dict, config.Environment) - - return dict, nil -} - -func loadYamlFile(ctx context.Context, - file types.ConfigFile, - opts *Options, - workingDir string, - environment types.Mapping, - ct *cycleTracker, - dict map[string]interface{}, - included []string, -) (map[string]interface{}, PostProcessor, error) { - ctx = context.WithValue(ctx, consts.ComposeFileKey{}, file.Filename) - if file.Content == nil && file.Config == nil { - content, err := os.ReadFile(file.Filename) - if err != nil { - return nil, nil, err - } - file.Content = content +// nodeToProject decodes the canonical merged yaml.Node directly into a +// *types.Project (no intermediate map[string]any) and applies the +// project-level post-decode passes: env_file declaring-scope side-table +// for lazy interpolation, Windows path conversion, profile / service +// selection, services environment + label resolution. Runs the +// equivalent of v2 ModelToProject without the map -> mapstructure +// detour. +func nodeToProject(root *yaml.Node, opts *Options, configDetails types.ConfigDetails) (*types.Project, error) { + project := &types.Project{ + WorkingDir: configDetails.WorkingDir, + Environment: configDetails.Environment, } - processRawYaml := func(raw interface{}, processor PostProcessor) error { - converted, err := convertToStringKeysRecursive(raw, "") - if err != nil { - return err - } - cfg, ok := converted.(map[string]interface{}) - if !ok { - return errors.New("top-level object must be a mapping") - } - - if opts.Interpolate != nil && !opts.SkipInterpolation { - cfg, err = interp.Interpolate(cfg, *opts.Interpolate) - if err != nil { - return err - } - } - - fixEmptyNotNull(cfg) - - // Process includes first so that extended services have all merged attributes - if !opts.SkipInclude { - included = append(included, file.Filename) - err = ApplyInclude(ctx, workingDir, environment, cfg, opts, included, processor) - if err != nil { - return err - } - } - - if err := processor.Apply(dict); err != nil { - return err - } - - // Process extends after includes so base services are fully merged - if !opts.SkipExtends { - err = ApplyExtends(ctx, cfg, opts, ct, processor) - if err != nil { - return err - } - - } - - dict, err = override.Merge(dict, cfg) - if err != nil { - return err - } - - dict, err = override.EnforceUnicity(dict) - if err != nil { - return err - } - - if !opts.SkipValidation { - if err := schema.Validate(dict); err != nil { - return fmt.Errorf("validating %s: %w", file.Filename, err) - } - if _, ok := dict["version"]; ok { - opts.warnObsoleteVersion(file.Filename) - delete(dict, "version") - } - } - - dict, err = transform.Canonical(dict, opts.SkipInterpolation) - if err != nil { - return err - } - - dict = OmitEmpty(dict) - - // Canonical transformation can reveal duplicates, typically as ports can be a range and conflict with an override - dict, err = override.EnforceUnicity(dict) - return err + // The project name has been stamped onto the merged tree by load() + // just before NormalizeNode, so the Decode below picks it up via + // the regular `name:` field. No special handling needed here. + if err := root.Decode(project); err != nil { + return nil, fmt.Errorf("decode project: %w", err) } - var processor PostProcessor - if file.Config == nil { - r := bytes.NewReader(file.Content) - decoder := yaml.NewDecoder(r) - for { - var raw interface{} - reset := &ResetProcessor{target: &raw, maxNodeVisits: opts.MaxNodeVisits} - err := decoder.Decode(reset) - if err != nil && errors.Is(err, io.EOF) { - break - } - if err != nil { - return nil, nil, fmt.Errorf("failed to parse %s: %w", file.Filename, err) - } - processor = reset - if err := processRawYaml(raw, processor); err != nil { - return nil, nil, err + // Attach the pre-canonical path positions snapshot to the project + // when the caller opted in via WithDiagnostics. Tooling can then + // resolve any compose path to its source file + line + column. + if opts.Diagnostics && len(opts.pathPositions) > 0 { + project.Sources = make(types.Sources, len(opts.pathPositions)) + for p, pos := range opts.pathPositions { + project.Sources[p] = types.Location{ + File: pos.file, + Line: pos.line, + Column: pos.column, } } - } else { - if err := processRawYaml(file.Config, NoopPostProcessor{}); err != nil { - return nil, nil, err - } - } - return dict, processor, nil -} - -func load(ctx context.Context, configDetails types.ConfigDetails, opts *Options, loaded []string) (map[string]interface{}, error) { - mainFile := configDetails.ConfigFiles[0].Filename - for _, f := range loaded { - if f == mainFile { - loaded = append(loaded, mainFile) - return nil, fmt.Errorf("include cycle detected:\n%s\n include %s", loaded[0], strings.Join(loaded[1:], "\n include ")) - } } - dict, err := loadYamlModel(ctx, configDetails, opts, &cycleTracker{}, nil) - if err != nil { + // Decode KnownExtensions into their declared target types. The yaml + // inline tag has parked them as map[string]any under Extensions; this + // pass swaps each known x-* entry for the typed value the caller + // registered. + if err := decodeKnownExtensions(project, opts.KnownExtensions); err != nil { return nil, err } - if len(dict) == 0 { - return nil, errors.New("empty compose file") - } - - if !opts.SkipValidation && opts.projectName == "" { - return nil, errors.New("project name must not be empty") - } - - if !opts.SkipNormalization { - dict["name"] = opts.projectName - dict, err = Normalize(dict, configDetails.Environment) - if err != nil { - return nil, err - } - } - - return dict, nil -} - -// ModelToProject binds a canonical yaml dict into compose-go structs -func ModelToProject(dict map[string]interface{}, opts *Options, configDetails types.ConfigDetails) (*types.Project, error) { - project := &types.Project{ - Name: opts.projectName, - WorkingDir: configDetails.WorkingDir, - Environment: configDetails.Environment, - } - delete(dict, "name") // project name set by yaml must be identified by caller as opts.projectName - - var err error - dict, err = processExtensions(dict, tree.NewPath(), opts.KnownExtensions) - if err != nil { - return nil, err - } - - err = Transform(dict, project) - if err != nil { - return nil, err + for path, env := range opts.envFileScopes { + project.SetEnvFileScope(path, env) } if opts.ConvertWindowsPaths { @@ -618,13 +481,13 @@ func ModelToProject(dict map[string]interface{}, opts *Options, configDetails ty } } + var err error if project, err = project.WithProfiles(opts.Profiles); err != nil { return nil, err } if !opts.SkipConsistencyCheck { - err := checkConsistency(project) - if err != nil { + if err := checkConsistency(project); err != nil { return nil, err } } @@ -659,6 +522,51 @@ func ModelToProject(dict map[string]interface{}, opts *Options, configDetails ty return project, nil } +// decodeKnownExtensions walks Project.Extensions and every typed +// container's Extensions map looking for keys the caller registered via +// Options.KnownExtensions. Each match has its raw map[string]any value +// re-decoded into the declared target type via a yaml round-trip so the +// caller gets the strongly-typed value back at p.Extensions[name]. +func decodeKnownExtensions(project *types.Project, known map[string]any) error { + if len(known) == 0 { + return nil + } + maps := []types.Extensions{project.Extensions} + for _, s := range project.Services { + maps = append(maps, s.Extensions) + } + for _, n := range project.Networks { + maps = append(maps, n.Extensions) + } + for _, v := range project.Volumes { + maps = append(maps, v.Extensions) + } + for _, c := range project.Configs { + maps = append(maps, c.Extensions) + } + for _, s := range project.Secrets { + maps = append(maps, s.Extensions) + } + for _, m := range maps { + for name, typ := range known { + raw, ok := m[name] + if !ok { + continue + } + target := reflect.New(reflect.TypeOf(typ)).Interface() + buf, err := yaml.Marshal(raw) + if err != nil { + return err + } + if err := yaml.Unmarshal(buf, target); err != nil { + return err + } + m[name] = reflect.ValueOf(target).Elem().Interface() + } + } + return nil +} + func InvalidProjectNameErr(v string) error { return fmt.Errorf( "invalid project name %q: must consist only of lowercase alphanumeric characters, hyphens, and underscores as well as start with a letter or number", @@ -750,186 +658,6 @@ func NormalizeProjectName(s string) string { return strings.TrimLeft(s, "_-") } -var userDefinedKeys = []tree.Path{ - "services", - "services.*.depends_on", - "volumes", - "networks", - "secrets", - "configs", -} - -func processExtensions(dict map[string]any, p tree.Path, extensions map[string]any) (map[string]interface{}, error) { - extras := map[string]any{} - var err error - for key, value := range dict { - skip := false - for _, uk := range userDefinedKeys { - if p.Matches(uk) { - skip = true - break - } - } - if !skip && strings.HasPrefix(key, "x-") { - extras[key] = value - delete(dict, key) - continue - } - switch v := value.(type) { - case map[string]interface{}: - dict[key], err = processExtensions(v, p.Next(key), extensions) - if err != nil { - return nil, err - } - case []interface{}: - for i, e := range v { - if m, ok := e.(map[string]interface{}); ok { - v[i], err = processExtensions(m, p.Next(strconv.Itoa(i)), extensions) - if err != nil { - return nil, err - } - } - } - } - } - for name, val := range extras { - if typ, ok := extensions[name]; ok { - target := reflect.New(reflect.TypeOf(typ)).Elem().Interface() - err = Transform(val, &target) - if err != nil { - return nil, err - } - extras[name] = target - } - } - if len(extras) > 0 { - dict[consts.Extensions] = extras - } - return dict, nil -} - -// Transform converts the source into the target struct with compose types transformer -// and the specified transformers if any. -func Transform(source interface{}, target interface{}) error { - data := mapstructure.Metadata{} - config := &mapstructure.DecoderConfig{ - DecodeHook: mapstructure.ComposeDecodeHookFunc( - nameServices, - decoderHook, - cast, - secretConfigDecoderHook, - ), - Result: target, - TagName: "yaml", - Metadata: &data, - } - decoder, err := mapstructure.NewDecoder(config) - if err != nil { - return err - } - return decoder.Decode(source) -} - -// nameServices create implicit `name` key for convenience accessing service -func nameServices(from reflect.Value, to reflect.Value) (interface{}, error) { - if to.Type() == reflect.TypeOf(types.Services{}) { - nameK := reflect.ValueOf("name") - iter := from.MapRange() - for iter.Next() { - name := iter.Key() - elem := iter.Value() - elem.Elem().SetMapIndex(nameK, name) - } - } - return from.Interface(), nil -} - -func secretConfigDecoderHook(from, to reflect.Type, data interface{}) (interface{}, error) { - // Check if the input is a map and we're decoding into a SecretConfig - if from.Kind() == reflect.Map && to == reflect.TypeOf(types.SecretConfig{}) { - if v, ok := data.(map[string]interface{}); ok { - if ext, ok := v[consts.Extensions].(map[string]interface{}); ok { - if val, ok := ext[types.SecretConfigXValue].(string); ok { - // Return a map with the Content field populated - v["Content"] = val - delete(ext, types.SecretConfigXValue) - - if len(ext) == 0 { - delete(v, consts.Extensions) - } - } - } - } - } - - // Return the original data so the rest is handled by default mapstructure logic - return data, nil -} - -// keys need to be converted to strings for jsonschema -func convertToStringKeysRecursive(value interface{}, keyPrefix string) (interface{}, error) { - if mapping, ok := value.(map[string]interface{}); ok { - for key, entry := range mapping { - var newKeyPrefix string - if keyPrefix == "" { - newKeyPrefix = key - } else { - newKeyPrefix = fmt.Sprintf("%s.%s", keyPrefix, key) - } - convertedEntry, err := convertToStringKeysRecursive(entry, newKeyPrefix) - if err != nil { - return nil, err - } - mapping[key] = convertedEntry - } - return mapping, nil - } - if mapping, ok := value.(map[interface{}]interface{}); ok { - dict := make(map[string]interface{}) - for key, entry := range mapping { - str, ok := key.(string) - if !ok { - return nil, formatInvalidKeyError(keyPrefix, key) - } - var newKeyPrefix string - if keyPrefix == "" { - newKeyPrefix = str - } else { - newKeyPrefix = fmt.Sprintf("%s.%s", keyPrefix, str) - } - convertedEntry, err := convertToStringKeysRecursive(entry, newKeyPrefix) - if err != nil { - return nil, err - } - dict[str] = convertedEntry - } - return dict, nil - } - if list, ok := value.([]interface{}); ok { - var convertedList []interface{} - for index, entry := range list { - newKeyPrefix := fmt.Sprintf("%s[%d]", keyPrefix, index) - convertedEntry, err := convertToStringKeysRecursive(entry, newKeyPrefix) - if err != nil { - return nil, err - } - convertedList = append(convertedList, convertedEntry) - } - return convertedList, nil - } - return value, nil -} - -func formatInvalidKeyError(keyPrefix string, key interface{}) error { - var location string - if keyPrefix == "" { - location = "at top level" - } else { - location = fmt.Sprintf("in %s", keyPrefix) - } - return fmt.Errorf("non-string key %s: %#v", location, key) -} - // Windows path, c:\\my\\path\\shiny, need to be changed to be compatible with // the Engine. Volume path are expected to be linux style /c/my/path/shiny/ func convertVolumePath(volume types.ServiceVolumeConfig) types.ServiceVolumeConfig { diff --git a/loader/loader_test.go b/loader/loader_test.go index e1e782c4d..ac1e07bd3 100644 --- a/loader/loader_test.go +++ b/loader/loader_test.go @@ -2149,7 +2149,7 @@ services: func TestInvalidProjectNameType(t *testing.T) { p, err := loadYAML(`name: 123`) - assert.Error(t, err, "validating filename0.yml: name must be a string") + assert.Error(t, err, "filename0.yml:1:7: name: name must be a string") assert.Assert(t, is.Nil(p)) } @@ -2309,7 +2309,7 @@ func TestLoadWithIncludeCycle(t *testing.T) { }, }, }) - assert.Check(t, strings.HasPrefix(err.Error(), "include cycle detected")) + assert.Check(t, strings.Contains(err.Error(), "include cycle detected")) } func TestLoadWithIncludeOverride(t *testing.T) { diff --git a/loader/loader_yaml_test.go b/loader/loader_yaml_test.go deleted file mode 100644 index 728a00871..000000000 --- a/loader/loader_yaml_test.go +++ /dev/null @@ -1,124 +0,0 @@ -/* - Copyright 2020 The Compose Specification Authors. - - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. -*/ - -package loader - -import ( - "context" - "testing" - - "github.com/compose-spec/compose-go/v3/types" - "gotest.tools/v3/assert" -) - -func TestParseYAMLFiles(t *testing.T) { - model, err := loadYamlModel(context.TODO(), types.ConfigDetails{ - ConfigFiles: []types.ConfigFile{ - { - Filename: "test.yaml", - Content: []byte(` -x-extension: - test1: first - -services: - test: - image: foo - command: echo hello - init: true -`), - }, - { - Filename: "override.yaml", - Content: []byte(` -x-extension: - test2: second - -services: - test: - image: bar - command: echo world - init: false -`), - }, - }, - }, &Options{}, &cycleTracker{}, nil) - assert.NilError(t, err) - assert.DeepEqual(t, model, map[string]interface{}{ - "services": map[string]interface{}{ - "test": map[string]interface{}{ - "image": "bar", - "command": "echo world", - "init": false, - }, - }, - "x-extension": map[string]interface{}{ - "test1": "first", - "test2": "second", - }, - }) -} - -func TestParseYAMLFilesMergeOverride(t *testing.T) { - model, err := loadYamlModel(context.TODO(), types.ConfigDetails{ - ConfigFiles: []types.ConfigFile{ - { - Filename: "override.yaml", - Content: []byte(` -services: - base: - configs: - - source: credentials - target: /credentials/file1 - x: &x - extends: - base - configs: !override - - source: credentials - target: /literally-anywhere-else - - y: - <<: *x - -configs: - credentials: - content: | - dummy value -`), - }, - }, - }, &Options{}, &cycleTracker{}, nil) - assert.NilError(t, err) - assert.DeepEqual(t, model, map[string]interface{}{ - "configs": map[string]interface{}{"credentials": map[string]interface{}{"content": string("dummy value\n")}}, - "services": map[string]interface{}{ - "base": map[string]interface{}{ - "configs": []interface{}{ - map[string]interface{}{"source": string("credentials"), "target": string("/credentials/file1")}, - }, - }, - "x": map[string]interface{}{ - "configs": []interface{}{ - map[string]interface{}{"source": string("credentials"), "target": string("/literally-anywhere-else")}, - }, - }, - "y": map[string]interface{}{ - "configs": []interface{}{ - map[string]interface{}{"source": string("credentials"), "target": string("/literally-anywhere-else")}, - }, - }, - }, - }) -} diff --git a/loader/mapstructure.go b/loader/mapstructure.go deleted file mode 100644 index e5b902ab2..000000000 --- a/loader/mapstructure.go +++ /dev/null @@ -1,79 +0,0 @@ -/* - Copyright 2020 The Compose Specification Authors. - - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. -*/ - -package loader - -import ( - "reflect" - "strconv" -) - -// comparable to yaml.Unmarshaler, decoder allow a type to define it's own custom logic to convert value -// see https://github.com/mitchellh/mapstructure/pull/294 -type decoder interface { - DecodeMapstructure(interface{}) error -} - -// see https://github.com/mitchellh/mapstructure/issues/115#issuecomment-735287466 -// adapted to support types derived from built-in types, as DecodeMapstructure would not be able to mutate internal -// value, so need to invoke DecodeMapstructure defined by pointer to type -func decoderHook(from reflect.Value, to reflect.Value) (interface{}, error) { - // If the destination implements the decoder interface - u, ok := to.Interface().(decoder) - if !ok { - // for non-struct types we need to invoke func (*type) DecodeMapstructure() - if to.CanAddr() { - pto := to.Addr() - u, ok = pto.Interface().(decoder) - } - if !ok { - return from.Interface(), nil - } - } - // If it is nil and a pointer, create and assign the target value first - if to.Type().Kind() == reflect.Ptr && to.IsNil() { - to.Set(reflect.New(to.Type().Elem())) - u = to.Interface().(decoder) - } - // Call the custom DecodeMapstructure method - if err := u.DecodeMapstructure(from.Interface()); err != nil { - return to.Interface(), err - } - return to.Interface(), nil -} - -func cast(from reflect.Value, to reflect.Value) (interface{}, error) { - switch from.Type().Kind() { - case reflect.String: - switch to.Kind() { - case reflect.Bool: - return toBoolean(from.String()) - case reflect.Int: - return toInt(from.String()) - case reflect.Int64: - return toInt64(from.String()) - case reflect.Float32: - return toFloat32(from.String()) - case reflect.Float64: - return toFloat(from.String()) - } - case reflect.Int: - if to.Kind() == reflect.String { - return strconv.FormatInt(from.Int(), 10), nil - } - } - return from.Interface(), nil -} diff --git a/loader/mapstructure_test.go b/loader/mapstructure_test.go deleted file mode 100644 index 787739b03..000000000 --- a/loader/mapstructure_test.go +++ /dev/null @@ -1,65 +0,0 @@ -/* - Copyright 2020 The Compose Specification Authors. - - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. -*/ - -package loader - -import ( - "testing" - - "github.com/compose-spec/compose-go/v3/types" - "github.com/go-viper/mapstructure/v2" - "gotest.tools/v3/assert" -) - -func TestDecodeMapStructure(t *testing.T) { - var target types.ServiceConfig - data := mapstructure.Metadata{} - config := &mapstructure.DecoderConfig{ - Result: &target, - TagName: "yaml", - Metadata: &data, - DecodeHook: mapstructure.ComposeDecodeHookFunc(decoderHook), - } - decoder, err := mapstructure.NewDecoder(config) - assert.NilError(t, err) - err = decoder.Decode(map[string]interface{}{ - "mem_limit": "640k", - "command": "echo hello", - "stop_grace_period": "60s", - "labels": []interface{}{ - "FOO=BAR", - }, - "deploy": map[string]interface{}{ - "labels": map[string]interface{}{ - "FOO": "BAR", - "BAZ": nil, - "QIX": 2, - "ZOT": true, - }, - }, - }) - assert.NilError(t, err) - assert.Equal(t, target.MemLimit, types.UnitBytes(640*1024)) - assert.DeepEqual(t, target.Command, types.ShellCommand{"echo", "hello"}) - assert.Equal(t, *target.StopGracePeriod, types.Duration(60_000_000_000)) - assert.DeepEqual(t, target.Labels, types.Labels{"FOO": "BAR"}) - assert.DeepEqual(t, target.Deploy.Labels, types.Labels{ - "FOO": "BAR", - "BAZ": "", - "QIX": "2", - "ZOT": "true", - }) -} diff --git a/loader/normalize.go b/loader/normalize.go index 1ba0f63a2..ba66d129f 100644 --- a/loader/normalize.go +++ b/loader/normalize.go @@ -18,249 +18,29 @@ package loader import ( "fmt" - "path" - "strconv" - "strings" + + "go.yaml.in/yaml/v4" "github.com/compose-spec/compose-go/v3/types" ) -// Normalize compose project by moving deprecated attributes to their canonical position and injecting implicit defaults +// Normalize injects implicit defaults (default networks, derived +// service dependencies, build defaults, implicit `name`, ...) into a +// map-shaped compose model. The function is a thin wrapper around +// NormalizeNode -- the canonical logic lives on the yaml.Node side +// where source positions are preserved -- and round-trips through +// yaml so callers that hold a map[string]any keep working. func Normalize(dict map[string]any, env types.Mapping) (map[string]any, error) { - normalizeNetworks(dict) - - if d, ok := dict["services"]; ok { - services := d.(map[string]any) - for name, s := range services { - service := s.(map[string]any) - - if service["pull_policy"] == types.PullPolicyIfNotPresent { - service["pull_policy"] = types.PullPolicyMissing - } - - fn := func(s string) (string, bool) { - v, ok := env[s] - return v, ok - } - - if b, ok := service["build"]; ok { - build := b.(map[string]any) - if build["context"] == nil { - build["context"] = "." - } - if build["dockerfile"] == nil && build["dockerfile_inline"] == nil { - build["dockerfile"] = "Dockerfile" - } - - if a, ok := build["args"]; ok { - build["args"], _ = resolve(a, fn, false) - } - - service["build"] = build - } - - if e, ok := service["environment"]; ok { - service["environment"], _ = resolve(e, fn, true) - } - - var dependsOn map[string]any - if d, ok := service["depends_on"]; ok { - dependsOn = d.(map[string]any) - } else { - dependsOn = map[string]any{} - } - if l, ok := service["links"]; ok { - links := l.([]any) - for _, e := range links { - link := e.(string) - parts := strings.Split(link, ":") - if len(parts) == 2 { - link = parts[0] - } - if _, ok := dependsOn[link]; !ok { - dependsOn[link] = map[string]any{ - "condition": types.ServiceConditionStarted, - "restart": true, - "required": true, - } - } - } - } - - for _, namespace := range []string{"network_mode", "ipc", "pid", "uts", "cgroup"} { - if n, ok := service[namespace]; ok { - ref := n.(string) - if strings.HasPrefix(ref, types.ServicePrefix) { - shared := ref[len(types.ServicePrefix):] - if _, ok := dependsOn[shared]; !ok { - dependsOn[shared] = map[string]any{ - "condition": types.ServiceConditionStarted, - "restart": true, - "required": true, - } - } - } - } - } - - if v, ok := service["volumes"]; ok { - volumes := v.([]any) - for i, volume := range volumes { - vol := volume.(map[string]any) - target := vol["target"].(string) - vol["target"] = path.Clean(target) - volumes[i] = vol - } - service["volumes"] = volumes - } - - if n, ok := service["volumes_from"]; ok { - volumesFrom := n.([]any) - for _, v := range volumesFrom { - vol := v.(string) - if !strings.HasPrefix(vol, types.ContainerPrefix) { - spec := strings.Split(vol, ":") - if _, ok := dependsOn[spec[0]]; !ok { - dependsOn[spec[0]] = map[string]any{ - "condition": types.ServiceConditionStarted, - "restart": false, - "required": true, - } - } - } - } - } - if len(dependsOn) > 0 { - service["depends_on"] = dependsOn - } - services[name] = service - } - - dict["services"] = services - } - setNameFromKey(dict) - - return dict, nil -} - -func normalizeNetworks(dict map[string]any) { - var networks map[string]any - if n, ok := dict["networks"]; ok { - networks = n.(map[string]any) - } else { - networks = map[string]any{} - } - - // implicit `default` network must be introduced only if actually used by some service - usesDefaultNetwork := false - - if s, ok := dict["services"]; ok { - services := s.(map[string]any) - for name, se := range services { - service := se.(map[string]any) - if _, ok := service["provider"]; ok { - continue - } - if _, ok := service["network_mode"]; ok { - continue - } - if n, ok := service["networks"]; !ok { - // If none explicitly declared, service is connected to default network - service["networks"] = map[string]any{"default": nil} - usesDefaultNetwork = true - } else { - net := n.(map[string]any) - if len(net) == 0 { - // networks section declared but empty (corner case) - service["networks"] = map[string]any{"default": nil} - usesDefaultNetwork = true - } else if _, ok := net["default"]; ok { - usesDefaultNetwork = true - } - } - services[name] = service - } - dict["services"] = services + var n yaml.Node + if err := n.Encode(dict); err != nil { + return nil, fmt.Errorf("normalize: encode map: %w", err) } - - if _, ok := networks["default"]; !ok && usesDefaultNetwork { - // If not declared explicitly, Compose model involves an implicit "default" network - networks["default"] = nil + if _, err := NormalizeNode(&n, env); err != nil { + return nil, err } - - if len(networks) > 0 { - dict["networks"] = networks + var out map[string]any + if err := n.Decode(&out); err != nil { + return nil, fmt.Errorf("normalize: decode after node normalize: %w", err) } -} - -func resolve(a any, fn func(s string) (string, bool), keepEmpty bool) (any, bool) { - switch v := a.(type) { - case []any: - var resolved []any - for _, val := range v { - if r, ok := resolve(val, fn, keepEmpty); ok { - resolved = append(resolved, r) - } - } - return resolved, true - case map[string]any: - resolved := map[string]any{} - for key, val := range v { - if val != nil { - resolved[key] = val - continue - } - if s, ok := fn(key); ok { - resolved[key] = s - } else if keepEmpty { - resolved[key] = nil - } - } - return resolved, true - case string: - if !strings.Contains(v, "=") { - if val, ok := fn(v); ok { - return fmt.Sprintf("%s=%s", v, val), true - } - if keepEmpty { - return v, true - } - return "", false - } - return v, true - default: - return v, false - } -} - -// Resources with no explicit name are actually named by their key in map -func setNameFromKey(dict map[string]any) { - for _, r := range []string{"networks", "volumes", "configs", "secrets"} { - a, ok := dict[r] - if !ok { - continue - } - toplevel := a.(map[string]any) - for key, r := range toplevel { - var resource map[string]any - if r != nil { - resource = r.(map[string]any) - } else { - resource = map[string]any{} - } - if resource["name"] == nil { - if x, ok := resource["external"]; ok && isTrue(x) { - resource["name"] = key - } else { - resource["name"] = fmt.Sprintf("%s_%s", dict["name"], key) - } - } - toplevel[key] = resource - } - } -} - -func isTrue(x any) bool { - parseBool, _ := strconv.ParseBool(fmt.Sprint(x)) - return parseBool + return out, nil } diff --git a/loader/normalize_node.go b/loader/normalize_node.go new file mode 100644 index 000000000..4af85178a --- /dev/null +++ b/loader/normalize_node.go @@ -0,0 +1,407 @@ +/* + Copyright 2020 The Compose Specification Authors. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +*/ + +package loader + +import ( + "fmt" + "path" + "strconv" + "strings" + + "go.yaml.in/yaml/v4" + + "github.com/compose-spec/compose-go/v3/types" +) + +// NormalizeNode injects implicit defaults (default networks, derived +// service dependencies, build defaults, implicit `name`, ...) into the +// merged yaml.Node tree. +// +// The walker keeps the root mapping pointer stable and operates section +// by section so untouched siblings (and untouched scalars inside +// touched sections) keep the Line / Column the YAML parser recorded. +// Downstream diagnostics still hit the right source location after +// normalize runs. +func NormalizeNode(root *yaml.Node, env types.Mapping) (*yaml.Node, error) { + if root == nil { + return nil, nil + } + target := root + if target.Kind == yaml.DocumentNode && len(target.Content) == 1 { + target = target.Content[0] + } + if target.Kind != yaml.MappingNode { + return root, nil + } + + normalizeNetworksNode(target) + if err := normalizeServicesNode(target, env); err != nil { + return nil, err + } + setNameFromKeyNode(target) + return root, nil +} + +// normalizeNetworksNode injects the implicit `default` network when any +// service does not opt out (network_mode, provider, explicit networks) +// and ensures the top-level `networks` mapping carries the entry. +func normalizeNetworksNode(root *yaml.Node) { + networks := mappingValueByKey(root, "networks") + if networks == nil { + networks = &yaml.Node{Kind: yaml.MappingNode, Tag: "!!map"} + } + + usesDefault := false + services := mappingValueByKey(root, "services") + if services != nil && services.Kind == yaml.MappingNode { + for i := 0; i+1 < len(services.Content); i += 2 { + svc := services.Content[i+1] + if svc == nil || svc.Kind != yaml.MappingNode { + continue + } + if mappingValueByKey(svc, "provider") != nil { + continue + } + if mappingValueByKey(svc, "network_mode") != nil { + continue + } + netsKey := mappingFieldNode(svc, "networks") + if netsKey == nil { + setMappingValue(svc, "networks", defaultNetworkOnly()) + usesDefault = true + continue + } + if netsKey.Kind != yaml.MappingNode || len(netsKey.Content) == 0 { + setMappingValue(svc, "networks", defaultNetworkOnly()) + usesDefault = true + continue + } + if mappingValueByKey(netsKey, "default") != nil { + usesDefault = true + } + } + } + + if usesDefault && mappingValueByKey(networks, "default") == nil { + setMappingValue(networks, "default", &yaml.Node{Kind: yaml.ScalarNode, Tag: "!!null"}) + } + if networks.Kind == yaml.MappingNode && len(networks.Content) > 0 { + setMappingValue(root, "networks", networks) + } +} + +// defaultNetworkOnly returns the canonical `{default: null}` mapping +// used as the `networks` value on services that did not declare any. +func defaultNetworkOnly() *yaml.Node { + return &yaml.Node{ + Kind: yaml.MappingNode, + Tag: "!!map", + Content: []*yaml.Node{ + {Kind: yaml.ScalarNode, Tag: "!!str", Value: "default"}, + {Kind: yaml.ScalarNode, Tag: "!!null"}, + }, + } +} + +// normalizeServicesNode walks each service and applies the per-service +// normalizations (pull_policy alias, build defaults, environment +// resolution, derived depends_on, volume target cleanup). +func normalizeServicesNode(root *yaml.Node, env types.Mapping) error { + services := mappingValueByKey(root, "services") + if services == nil || services.Kind != yaml.MappingNode { + return nil + } + for i := 0; i+1 < len(services.Content); i += 2 { + svc := services.Content[i+1] + if svc == nil || svc.Kind != yaml.MappingNode { + continue + } + normalizePullPolicy(svc) + if err := normalizeBuild(svc, env); err != nil { + return err + } + normalizeServiceEnvironment(svc, env) + normalizeServiceDependsOn(svc) + normalizeVolumeTargets(svc) + } + return nil +} + +func normalizePullPolicy(svc *yaml.Node) { + pp := mappingValueByKey(svc, "pull_policy") + if pp == nil || pp.Kind != yaml.ScalarNode { + return + } + if pp.Value == types.PullPolicyIfNotPresent { + pp.Value = types.PullPolicyMissing + } +} + +func normalizeBuild(svc *yaml.Node, env types.Mapping) error { + build := mappingValueByKey(svc, "build") + if build == nil || build.Kind != yaml.MappingNode { + return nil + } + if mappingValueByKey(build, "context") == nil { + setMappingValue(build, "context", &yaml.Node{ + Kind: yaml.ScalarNode, Tag: "!!str", Value: ".", + }) + } + if mappingValueByKey(build, "dockerfile") == nil && mappingValueByKey(build, "dockerfile_inline") == nil { + setMappingValue(build, "dockerfile", &yaml.Node{ + Kind: yaml.ScalarNode, Tag: "!!str", Value: "Dockerfile", + }) + } + if args := mappingValueByKey(build, "args"); args != nil { + resolveSequenceOrMapping(args, env, false) + } + return nil +} + +func normalizeServiceEnvironment(svc *yaml.Node, env types.Mapping) { + e := mappingValueByKey(svc, "environment") + if e == nil { + return + } + resolveSequenceOrMapping(e, env, true) +} + +// resolveSequenceOrMapping rewrites bare `KEY` entries to `KEY=value` +// when the variable is set in env. Operates on both sequence form (list +// of strings) and mapping form (with null values). When keepEmpty is +// false, unset entries in mapping form are dropped. +func resolveSequenceOrMapping(n *yaml.Node, env types.Mapping, keepEmpty bool) { + switch n.Kind { + case yaml.SequenceNode: + filtered := n.Content[:0] + for _, item := range n.Content { + if item == nil || item.Kind != yaml.ScalarNode { + filtered = append(filtered, item) + continue + } + if strings.Contains(item.Value, "=") { + filtered = append(filtered, item) + continue + } + if v, ok := env[item.Value]; ok { + item.Value = fmt.Sprintf("%s=%s", item.Value, v) + filtered = append(filtered, item) + continue + } + if keepEmpty { + filtered = append(filtered, item) + } + } + n.Content = filtered + case yaml.MappingNode: + filtered := n.Content[:0] + for i := 0; i+1 < len(n.Content); i += 2 { + k, v := n.Content[i], n.Content[i+1] + if v != nil && v.Kind == yaml.ScalarNode && v.Tag != "!!null" { + filtered = append(filtered, k, v) + continue + } + if val, ok := env[k.Value]; ok { + filtered = append(filtered, k, &yaml.Node{Kind: yaml.ScalarNode, Tag: "!!str", Value: val}) + continue + } + if keepEmpty { + filtered = append(filtered, k, v) + } + } + n.Content = filtered + } +} + +// normalizeServiceDependsOn derives implicit depends_on entries from +// links, namespace references (network_mode/ipc/pid/uts/cgroup with the +// "service:" prefix) and volumes_from. The existing depends_on mapping +// is mutated in place; an empty section is left untouched. +func normalizeServiceDependsOn(svc *yaml.Node) { + dependsOn := mappingValueByKey(svc, "depends_on") + created := false + if dependsOn == nil || dependsOn.Kind != yaml.MappingNode { + dependsOn = &yaml.Node{Kind: yaml.MappingNode, Tag: "!!map"} + created = true + } + + addDep := func(name string, restart bool) { + if name == "" { + return + } + if mappingValueByKey(dependsOn, name) != nil { + return + } + setMappingValue(dependsOn, name, dependsOnEntry(restart)) + } + + if links := mappingValueByKey(svc, "links"); links != nil && links.Kind == yaml.SequenceNode { + for _, item := range links.Content { + if item == nil || item.Kind != yaml.ScalarNode { + continue + } + link := item.Value + parts := strings.Split(link, ":") + if len(parts) == 2 { + link = parts[0] + } + addDep(link, true) + } + } + + for _, namespace := range []string{"network_mode", "ipc", "pid", "uts", "cgroup"} { + ref := mappingValueByKey(svc, namespace) + if ref == nil || ref.Kind != yaml.ScalarNode { + continue + } + if !strings.HasPrefix(ref.Value, types.ServicePrefix) { + continue + } + addDep(ref.Value[len(types.ServicePrefix):], true) + } + + if vf := mappingValueByKey(svc, "volumes_from"); vf != nil && vf.Kind == yaml.SequenceNode { + for _, item := range vf.Content { + if item == nil || item.Kind != yaml.ScalarNode { + continue + } + vol := item.Value + if strings.HasPrefix(vol, types.ContainerPrefix) { + continue + } + spec := strings.Split(vol, ":") + addDep(spec[0], false) + } + } + + if len(dependsOn.Content) == 0 { + return + } + if created { + setMappingValue(svc, "depends_on", dependsOn) + } +} + +func dependsOnEntry(restart bool) *yaml.Node { + restartScalar := &yaml.Node{Kind: yaml.ScalarNode, Tag: "!!bool", Value: "false"} + if restart { + restartScalar.Value = "true" + } + return &yaml.Node{ + Kind: yaml.MappingNode, + Tag: "!!map", + Content: []*yaml.Node{ + {Kind: yaml.ScalarNode, Tag: "!!str", Value: "condition"}, + {Kind: yaml.ScalarNode, Tag: "!!str", Value: types.ServiceConditionStarted}, + {Kind: yaml.ScalarNode, Tag: "!!str", Value: "restart"}, + restartScalar, + {Kind: yaml.ScalarNode, Tag: "!!str", Value: "required"}, + {Kind: yaml.ScalarNode, Tag: "!!bool", Value: "true"}, + }, + } +} + +// normalizeVolumeTargets cleans every `services.*.volumes.*.target` +// path with path.Clean so the canonical form matches what v2 produced. +func normalizeVolumeTargets(svc *yaml.Node) { + volumes := mappingValueByKey(svc, "volumes") + if volumes == nil || volumes.Kind != yaml.SequenceNode { + return + } + for _, item := range volumes.Content { + if item == nil || item.Kind != yaml.MappingNode { + continue + } + target := mappingValueByKey(item, "target") + if target == nil || target.Kind != yaml.ScalarNode || target.Value == "" { + continue + } + target.Value = path.Clean(target.Value) + } +} + +// setNameFromKeyNode assigns the implicit `_` name (or +// the bare key for `external: true` entries) to networks / volumes / +// configs / secrets entries that did not declare one explicitly. +func setNameFromKeyNode(root *yaml.Node) { + projectName := scalarValueByKey(root, "name") + for _, section := range []string{"networks", "volumes", "configs", "secrets"} { + topLevel := mappingValueByKey(root, section) + if topLevel == nil || topLevel.Kind != yaml.MappingNode { + continue + } + for i := 0; i+1 < len(topLevel.Content); i += 2 { + key := topLevel.Content[i] + resource := topLevel.Content[i+1] + if resource == nil || resource.Kind != yaml.MappingNode { + resource = &yaml.Node{Kind: yaml.MappingNode, Tag: "!!map"} + topLevel.Content[i+1] = resource + } + if mappingValueByKey(resource, "name") != nil { + continue + } + ext := mappingValueByKey(resource, "external") + if ext != nil && isTrueNode(ext) { + setMappingValue(resource, "name", &yaml.Node{Kind: yaml.ScalarNode, Tag: "!!str", Value: key.Value}) + continue + } + setMappingValue(resource, "name", &yaml.Node{ + Kind: yaml.ScalarNode, + Tag: "!!str", + Value: fmt.Sprintf("%s_%s", projectName, key.Value), + }) + } + } +} + +func scalarValueByKey(n *yaml.Node, key string) string { + v := mappingValueByKey(n, key) + if v == nil || v.Kind != yaml.ScalarNode { + return "" + } + return v.Value +} + +func isTrueNode(n *yaml.Node) bool { + if n == nil { + return false + } + if n.Kind == yaml.MappingNode { + // `external: { name: ... }` shorthand is treated as truthy. + return true + } + if n.Kind != yaml.ScalarNode { + return false + } + parsed, _ := strconv.ParseBool(n.Value) + return parsed +} + +// mappingFieldNode returns the value node for key in n, or nil. Unlike +// mappingValueByKey, the function returns nil even for null values so +// callers can distinguish "key absent" from "key present with null". +func mappingFieldNode(n *yaml.Node, key string) *yaml.Node { + if n == nil || n.Kind != yaml.MappingNode { + return nil + } + for i := 0; i+1 < len(n.Content); i += 2 { + if n.Content[i].Value == key { + return n.Content[i+1] + } + } + return nil +} diff --git a/loader/normalize_node_test.go b/loader/normalize_node_test.go new file mode 100644 index 000000000..15c70451b --- /dev/null +++ b/loader/normalize_node_test.go @@ -0,0 +1,77 @@ +/* + Copyright 2020 The Compose Specification Authors. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +*/ + +package loader + +import ( + "testing" + + "go.yaml.in/yaml/v4" + "gotest.tools/v3/assert" + + "github.com/compose-spec/compose-go/v3/types" +) + +func parseNormalizeNode(t *testing.T, src string) *yaml.Node { + t.Helper() + var doc yaml.Node + assert.NilError(t, yaml.Unmarshal([]byte(src), &doc)) + return &doc +} + +func decodeNormalize(t *testing.T, n *yaml.Node) map[string]any { + t.Helper() + var m map[string]any + assert.NilError(t, n.Decode(&m)) + return m +} + +func TestNormalizeNode_InjectsDefaultNetwork(t *testing.T) { + root := parseNormalizeNode(t, ` +name: app +services: + web: + image: nginx +`) + out, err := NormalizeNode(root, types.Mapping{}) + assert.NilError(t, err) + m := decodeNormalize(t, out) + nets, ok := m["networks"].(map[string]any) + assert.Assert(t, ok, "default network injected: %v", m["networks"]) + _, hasDefault := nets["default"] + assert.Assert(t, hasDefault, "networks.default should be created") +} + +func TestNormalizeNode_BuildContextDefaultsToDot(t *testing.T) { + root := parseNormalizeNode(t, ` +name: app +services: + web: + build: + dockerfile: Dockerfile +`) + out, err := NormalizeNode(root, types.Mapping{}) + assert.NilError(t, err) + m := decodeNormalize(t, out) + build := m["services"].(map[string]any)["web"].(map[string]any)["build"].(map[string]any) + assert.Equal(t, build["context"], ".") +} + +func TestNormalizeNode_NilSafe(t *testing.T) { + out, err := NormalizeNode(nil, types.Mapping{}) + assert.NilError(t, err) + assert.Assert(t, out == nil) +} diff --git a/loader/omitEmpty.go b/loader/omitEmpty.go deleted file mode 100644 index c88057a3b..000000000 --- a/loader/omitEmpty.go +++ /dev/null @@ -1,75 +0,0 @@ -/* - Copyright 2020 The Compose Specification Authors. - - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. -*/ - -package loader - -import "github.com/compose-spec/compose-go/v3/tree" - -var omitempty = []tree.Path{ - "services.*.dns", -} - -// OmitEmpty removes empty attributes which are irrelevant when unset -func OmitEmpty(yaml map[string]any) map[string]any { - cleaned := omitEmpty(yaml, tree.NewPath()) - return cleaned.(map[string]any) -} - -func omitEmpty(data any, p tree.Path) any { - switch v := data.(type) { - case map[string]any: - for k, e := range v { - if isEmpty(e) && mustOmit(p) { - delete(v, k) - continue - } - - v[k] = omitEmpty(e, p.Next(k)) - } - return v - case []any: - c := make([]any, 0, len(v)) - for _, e := range v { - if isEmpty(e) && mustOmit(p) { - continue - } - - c = append(c, omitEmpty(e, p.Next("[]"))) - } - return c - default: - return data - } -} - -func mustOmit(p tree.Path) bool { - for _, pattern := range omitempty { - if p.Matches(pattern) { - return true - } - } - return false -} - -func isEmpty(e any) bool { - if e == nil { - return true - } - if v, ok := e.(string); ok && v == "" { - return true - } - return false -} diff --git a/loader/omitEmpty_test.go b/loader/omitEmpty_test.go deleted file mode 100644 index cdfab13b5..000000000 --- a/loader/omitEmpty_test.go +++ /dev/null @@ -1,41 +0,0 @@ -/* - Copyright 2020 The Compose Specification Authors. - - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. -*/ - -package loader - -import ( - "testing" - - "gotest.tools/v3/assert" -) - -func TestOmitEmptyPreservesEmptySlice(t *testing.T) { - input := map[string]any{ - "services": map[string]any{ - "foo": map[string]any{ - "build": map[string]any{ - "cache_to": []any{}, - }, - }, - }, - } - got := OmitEmpty(input) - cacheTo := got["services"].(map[string]any)["foo"].(map[string]any)["build"].(map[string]any)["cache_to"] - slice, ok := cacheTo.([]any) - assert.Assert(t, ok, "cache_to should remain a []any, got %T", cacheTo) - assert.Assert(t, slice != nil, "cache_to should remain a non-nil empty slice") - assert.Equal(t, len(slice), 0) -} diff --git a/loader/reset.go b/loader/reset.go index 9ac7e401e..833a7fdca 100644 --- a/loader/reset.go +++ b/loader/reset.go @@ -18,228 +18,44 @@ package loader import ( "fmt" - "strconv" - "strings" - "github.com/compose-spec/compose-go/v3/tree" "go.yaml.in/yaml/v4" -) - -// defaultMaxNodeVisits caps total resolveReset calls per document. -// Sized to accommodate large real-world compose files while rejecting documents that would -// cause unbounded traversal. Callers can override this via Options.MaxNodeVisits. -const defaultMaxNodeVisits = 100_000 -// nodeCache stores a resolved node and the relative sub-paths within its subtree that -// carried !reset/!override tags, so cache hits at different call sites can replay them. -type nodeCache struct { - node *yaml.Node - relativePaths []tree.Path -} + "github.com/compose-spec/compose-go/v3/internal/node" + "github.com/compose-spec/compose-go/v3/tree" +) +// ResetProcessor adapts node.ResolveResetOverride to the yaml.Decoder API. +// It collects the !reset/!override paths during YAML decoding and replays +// them as map deletions via Apply, after the v2 pipeline has merged the +// decoded documents into the running map[string]any. +// +// The yaml.Node-side logic lives in internal/node so the upcoming merge +// phase can reuse it without going through the legacy map[string]any path. type ResetProcessor struct { target any paths []tree.Path - visitedNodes map[*yaml.Node][]string - resolvedNodes map[*yaml.Node]nodeCache - visitCount int - // maxNodeVisits is the per-document cap; when zero, defaultMaxNodeVisits is used. maxNodeVisits int } -// UnmarshalYAML implement yaml.Unmarshaler +// UnmarshalYAML implements yaml.Unmarshaler. func (p *ResetProcessor) UnmarshalYAML(value *yaml.Node) error { - p.visitedNodes = make(map[*yaml.Node][]string) - p.resolvedNodes = make(map[*yaml.Node]nodeCache) - p.visitCount = 0 - defer func() { - p.visitedNodes = nil - p.resolvedNodes = nil - }() - resolved, err := p.resolveReset(value, tree.NewPath()) + resolved, paths, err := node.ResolveResetOverride(value, p.maxNodeVisits) if err != nil { return err } + p.paths = paths return resolved.Decode(p.target) } -// resolveReset detects `!reset` tag being set on yaml nodes and record position in the yaml tree -func (p *ResetProcessor) resolveReset(node *yaml.Node, path tree.Path) (*yaml.Node, error) { - p.visitCount++ - limit := p.maxNodeVisits - if limit <= 0 { - limit = defaultMaxNodeVisits - } - if p.visitCount > limit { - return nil, fmt.Errorf("compose file exceeds maximum node visit limit (%d)", limit) - } - - pathStr := path.String() - // If the path contains "<<", removing the "<<" element and merging the path - if strings.Contains(pathStr, ".<<") { - path = tree.NewPath(strings.Replace(pathStr, ".<<", "", 1)) - } - - if node.Tag == "!reset" { - p.paths = append(p.paths, path) - return nil, nil - } - if node.Tag == "!override" { - p.paths = append(p.paths, path) - return node, nil - } - - // If the node is an alias, process the alias target via the cache so each anchor is - // processed at most once. - if node.Kind == yaml.AliasNode { - if err := p.checkForCycle(node.Alias, path); err != nil { - return nil, err - } - // Handle !reset/!override on the alias target before delegating to the cache, - // keeping all tag-handling logic in resolveReset rather than split across functions. - target := node.Alias - if target.Tag == "!reset" { - p.paths = append(p.paths, path) - return nil, nil - } - if target.Tag == "!override" { - p.paths = append(p.paths, path) - return target, nil - } - return p.cachedResolve(target, path) - } - - // Container nodes are resolved through the cache, ensuring resolved containers are - // not re-traversed. - if node.Kind == yaml.SequenceNode || node.Kind == yaml.MappingNode { - return p.cachedResolve(node, path) - } - - return node, nil -} - -// cachedResolve resolves node (a container without !reset/!override), serving from cache on -// repeat visits to prevent re-traversal. It is only called after tag checks are done in -// resolveReset, so it never receives !reset/!override-tagged nodes. -func (p *ResetProcessor) cachedResolve(node *yaml.Node, path tree.Path) (*yaml.Node, error) { - if cached, ok := p.resolvedNodes[node]; ok { - for _, rel := range cached.relativePaths { - p.paths = append(p.paths, joinPath(path, rel)) - } - return cached.node, nil - } - - startIdx := len(p.paths) - resolved, err := p.resolveContainer(node, path) - if err != nil { - return nil, err - } - - var relPaths []tree.Path - for _, addedPath := range p.paths[startIdx:] { - rel, err := subPath(addedPath, path) - if err != nil { - return nil, err - } - relPaths = append(relPaths, rel) - } - p.resolvedNodes[node] = nodeCache{node: resolved, relativePaths: relPaths} - return resolved, nil -} - -// resolveContainer processes the children of a Sequence or Mapping node. -// AliasNodes must be kept as-is in the output Content; the resolved value is used only -// for tag inspection. Changing this will affect how the YAML library handles the document -// during decoding. -func (p *ResetProcessor) resolveContainer(node *yaml.Node, path tree.Path) (*yaml.Node, error) { - switch node.Kind { - case yaml.SequenceNode: - var nodes []*yaml.Node - for idx, v := range node.Content { - next := path.Next(strconv.Itoa(idx)) - resolved, err := p.resolveReset(v, next) - if err != nil { - return nil, err - } - if resolved == nil { - continue - } - if v.Kind == yaml.AliasNode { - nodes = append(nodes, v) - } else { - nodes = append(nodes, resolved) - } - } - node.Content = nodes - case yaml.MappingNode: - keys := map[string]int{} - var key string - var nodes []*yaml.Node - for idx, v := range node.Content { - if idx%2 == 0 { - key = v.Value - if line, seen := keys[key]; seen { - return nil, fmt.Errorf("line %d: mapping key %#v already defined at line %d", v.Line, key, line) - } - keys[key] = v.Line - } else { - resolved, err := p.resolveReset(v, path.Next(key)) - if err != nil { - return nil, err - } - if resolved == nil { - continue - } - if v.Kind == yaml.AliasNode { - nodes = append(nodes, node.Content[idx-1], v) - } else { - nodes = append(nodes, node.Content[idx-1], resolved) - } - } - } - node.Content = nodes - } - return node, nil -} - -// subPath strips base from full to produce a relative path for cache storage. -// Returns "" when full == base (the !reset/!override tag is on the node root itself). -// Returns an error when full is not rooted at base, which would indicate a logic error -// in resolveReset/cachedResolve. -func subPath(full, base tree.Path) (tree.Path, error) { - if base == "" { - return full, nil - } - fullStr := string(full) - baseStr := string(base) - if fullStr == baseStr { - return "", nil - } - prefix := baseStr + "." - if strings.HasPrefix(fullStr, prefix) { - return tree.Path(fullStr[len(prefix):]), nil - } - return "", fmt.Errorf("internal error: path %q is not a sub-path of %q", fullStr, baseStr) -} - -// joinPath reconstructs an absolute path from a call-site base and a cached relative path. -// A relative path of "" means the tag was on the node root, so base is returned unchanged. -func joinPath(base, rel tree.Path) tree.Path { - if rel == "" { - return base - } - if base == "" { - return rel - } - return tree.Path(string(base) + "." + string(rel)) -} - -// Apply finds the go attributes matching recorded paths and reset them to zero value +// Apply walks target (a map[string]any tree decoded from YAML) and removes +// every entry whose path matches one of the recorded !reset/!override paths. +// This is the v2 post-merge cleanup; replaced by a direct Node-tree +// rewrite during merge. func (p *ResetProcessor) Apply(target any) error { return p.applyNullOverrides(target, tree.NewPath()) } -// applyNullOverrides set val to Zero if it matches any of the recorded paths func (p *ResetProcessor) applyNullOverrides(target any, path tree.Path) error { switch v := target.(type) { case map[string]any: @@ -252,8 +68,7 @@ func (p *ResetProcessor) applyNullOverrides(target any, path tree.Path) error { continue KEYS } } - err := p.applyNullOverrides(e, next) - if err != nil { + if err := p.applyNullOverrides(e, next); err != nil { return err } } @@ -264,55 +79,13 @@ func (p *ResetProcessor) applyNullOverrides(target any, path tree.Path) error { for _, pattern := range p.paths { if next.Matches(pattern) { continue ITER - // TODO(ndeloof) support removal from sequence + // TODO(ndeloof) support removal from sequence — tracked. } } - err := p.applyNullOverrides(e, next) - if err != nil { + if err := p.applyNullOverrides(e, next); err != nil { return err } } } return nil } - -func (p *ResetProcessor) checkForCycle(node *yaml.Node, path tree.Path) error { - paths := p.visitedNodes[node] - pathStr := path.String() - - for _, prevPath := range paths { - // If we're visiting the exact same path, it's not a cycle - if pathStr == prevPath { - continue - } - - // If either path is using a merge key, it's legitimate YAML merging - if strings.Contains(prevPath, "<<") || strings.Contains(pathStr, "<<") { - continue - } - - // Only consider it a cycle if one path is contained within the other - // and they're not in different service definitions - if (strings.HasPrefix(pathStr, prevPath+".") || - strings.HasPrefix(prevPath, pathStr+".")) && - !areInDifferentServices(pathStr, prevPath) { - return fmt.Errorf("cycle detected: node at path %s references node at path %s", pathStr, prevPath) - } - } - - p.visitedNodes[node] = append(paths, pathStr) - return nil -} - -// areInDifferentServices checks if two paths are in different service definitions -func areInDifferentServices(path1, path2 string) bool { - parts1 := strings.Split(path1, ".") - parts2 := strings.Split(path2, ".") - for i := 0; i < len(parts1) && i < len(parts2); i++ { - if parts1[i] == "services" && i+1 < len(parts1) && - parts2[i] == "services" && i+1 < len(parts2) { - return parts1[i+1] != parts2[i+1] - } - } - return false -} diff --git a/loader/reset_test.go b/loader/reset_test.go index 89af1b2e9..c5f30dee9 100644 --- a/loader/reset_test.go +++ b/loader/reset_test.go @@ -23,6 +23,7 @@ import ( "testing" "time" + "github.com/compose-spec/compose-go/v3/internal/node" "github.com/compose-spec/compose-go/v3/types" "gotest.tools/v3/assert" ) @@ -236,8 +237,8 @@ services: // TestVisitCounterLimit verifies that a document with more than the default node visit cap // is rejected with a clear error, providing a safety belt independent of alias memoization. func TestVisitCounterLimit(t *testing.T) { - // Two mappings of (defaultMaxNodeVisits/2 + 1) entries each → total > cap value visits. - half := defaultMaxNodeVisits/2 + 1 + // Two mappings of (node.DefaultMaxNodeVisits/2 + 1) entries each → total > cap value visits. + half := node.DefaultMaxNodeVisits/2 + 1 var sb strings.Builder sb.WriteString("name: test\nx-data1:\n") for i := 0; i < half; i++ { @@ -254,7 +255,7 @@ func TestVisitCounterLimit(t *testing.T) { // TestVisitCounterLimitOverride verifies that Options.MaxNodeVisits raises the cap, allowing // documents that would be rejected at the default limit to load successfully. func TestVisitCounterLimitOverride(t *testing.T) { - half := defaultMaxNodeVisits/2 + 1 + half := node.DefaultMaxNodeVisits/2 + 1 var sb strings.Builder sb.WriteString("name: test\nx-data1:\n") for i := 0; i < half; i++ { @@ -269,7 +270,7 @@ func TestVisitCounterLimitOverride(t *testing.T) { }, func(options *Options) { options.SkipNormalization = true options.SkipConsistencyCheck = true - options.MaxNodeVisits = defaultMaxNodeVisits * 2 + options.MaxNodeVisits = node.DefaultMaxNodeVisits * 2 }) assert.NilError(t, err) } diff --git a/loader/resolve_environment_node.go b/loader/resolve_environment_node.go new file mode 100644 index 000000000..1573eb9e0 --- /dev/null +++ b/loader/resolve_environment_node.go @@ -0,0 +1,187 @@ +/* + Copyright 2020 The Compose Specification Authors. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +*/ + +package loader + +import ( + "fmt" + "strings" + + "go.yaml.in/yaml/v4" + + "github.com/compose-spec/compose-go/v3/internal/node" +) + +// ResolveEnvironmentNode walks the merged yaml.Node tree and resolves the +// bare-key entries in services.*.environment, secrets.*.environment and +// configs.*.environment by looking each variable up against the scalar +// SourceContext.Environment. When the variable is found, the scalar is +// rewritten in "KEY=value" form; when missing, the scalar is left as-is +// (matching the v2 ResolveEnvironment behavior that distinguishes +// "interpolation produced the empty string" from "value cannot be +// resolved"). +// +// The Node-side implementation is the fix for the bare-key lookup +// quirk: the lookup is performed in the SourceContext of the scalar itself, +// not in the project-wide environment, so an env_file declared on an +// include block becomes visible to services defined inside that include +// even though the parent project environment does not carry the variable. +func ResolveEnvironmentNode(root *yaml.Node, origins map[*yaml.Node]*node.SourceContext) { + if root == nil { + return + } + target := root + if target.Kind == yaml.DocumentNode && len(target.Content) == 1 { + target = target.Content[0] + } + if target.Kind != yaml.MappingNode { + return + } + resolveTopLevel := func(topKey, inner string) { + section := mappingValueByKey(target, topKey) + if section == nil || section.Kind != yaml.MappingNode { + return + } + for i := 1; i < len(section.Content); i += 2 { + entry := section.Content[i] + if entry.Kind != yaml.MappingNode { + continue + } + env := mappingValueByKey(entry, inner) + if env == nil || env.Kind != yaml.SequenceNode { + continue + } + resolveEnvSequence(env, origins) + } + } + resolveTopLevel("services", "environment") + resolveTopLevel("secrets", "environment") + resolveTopLevel("configs", "environment") +} + +func resolveEnvSequence(seq *yaml.Node, origins map[*yaml.Node]*node.SourceContext) { + for _, item := range seq.Content { + if item.Kind != yaml.ScalarNode { + continue + } + if strings.Contains(item.Value, "=") { + continue + } + ctx := origins[item] + if ctx == nil { + continue + } + if value, ok := ctx.Environment[item.Value]; ok { + item.Value = fmt.Sprintf("%s=%s", item.Value, value) + } + } +} + +// CaptureSecretConfigContent walks the merged tree and, for each +// `secrets.NAME.environment` / `configs.NAME.environment` scalar, +// resolves the variable against the SourceContext.Environment of the +// layer that DECLARED that scalar. Returns two `secrets-name -> resolved +// value` and `configs-name -> resolved value` maps so the resolution can +// later survive a CanonicalNode round-trip that re-encodes subtrees and +// invalidates the *yaml.Node pointers backing `origins`. +// +// The lookup-at-origin behavior fixes a v2 limitation where the +// project-wide environment was the only scope: a secret declared in an +// included compose file whose env_file introduced the variable could +// not see it. The secret/config now resolves in the same scope its +// declaration would resolve `${VAR}` interpolation in -- the layer's +// own environment. +func CaptureSecretConfigContent(root *yaml.Node, origins map[*yaml.Node]*node.SourceContext) (map[string]string, map[string]string) { + secrets := map[string]string{} + configs := map[string]string{} + if root == nil { + return secrets, configs + } + target := root + if target.Kind == yaml.DocumentNode && len(target.Content) == 1 { + target = target.Content[0] + } + if target.Kind != yaml.MappingNode { + return secrets, configs + } + collect := func(section *yaml.Node, into map[string]string) { + if section == nil || section.Kind != yaml.MappingNode { + return + } + for i := 0; i+1 < len(section.Content); i += 2 { + name := section.Content[i].Value + entry := section.Content[i+1] + if entry.Kind != yaml.MappingNode { + continue + } + env := mappingValueByKey(entry, "environment") + if env == nil || env.Kind != yaml.ScalarNode || env.Value == "" { + continue + } + ctx := origins[env] + if ctx == nil { + continue + } + if value, ok := ctx.Environment[env.Value]; ok { + into[name] = value + } + } + } + collect(mappingValueByKey(target, "secrets"), secrets) + collect(mappingValueByKey(target, "configs"), configs) + return secrets, configs +} + +// ApplySecretConfigContent injects each captured `name -> value` pair as +// a `content` scalar inside the corresponding entry of the post-canonical +// tree. Runs after the compose-rule validator so the mutual-exclusivity +// check between content and environment does not flag the synthesized +// value. +func ApplySecretConfigContent(root *yaml.Node, secrets, configs map[string]string) { + if root == nil || (len(secrets) == 0 && len(configs) == 0) { + return + } + target := root + if target.Kind == yaml.DocumentNode && len(target.Content) == 1 { + target = target.Content[0] + } + if target.Kind != yaml.MappingNode { + return + } + apply := func(section *yaml.Node, values map[string]string) { + if section == nil || section.Kind != yaml.MappingNode || len(values) == 0 { + return + } + for i := 0; i+1 < len(section.Content); i += 2 { + name := section.Content[i].Value + value, ok := values[name] + if !ok { + continue + } + entry := section.Content[i+1] + if entry.Kind != yaml.MappingNode { + continue + } + setMappingValue(entry, "content", &yaml.Node{ + Kind: yaml.ScalarNode, + Tag: "!!str", + Value: value, + }) + } + } + apply(mappingValueByKey(target, "secrets"), secrets) + apply(mappingValueByKey(target, "configs"), configs) +} diff --git a/loader/resolve_environment_node_test.go b/loader/resolve_environment_node_test.go new file mode 100644 index 000000000..97f444015 --- /dev/null +++ b/loader/resolve_environment_node_test.go @@ -0,0 +1,113 @@ +/* + Copyright 2020 The Compose Specification Authors. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +*/ + +package loader + +import ( + "testing" + + "go.yaml.in/yaml/v4" + "gotest.tools/v3/assert" + + "github.com/compose-spec/compose-go/v3/internal/node" + "github.com/compose-spec/compose-go/v3/tree" + "github.com/compose-spec/compose-go/v3/types" +) + +func TestResolveEnvironmentNode_BareKeyResolvedAgainstScalarContext(t *testing.T) { + src := ` +services: + web: + environment: + - FOO + - BAR=2 + api: + environment: + - FOO +` + var doc yaml.Node + assert.NilError(t, yaml.Unmarshal([]byte(src), &doc)) + + // Build distinct SourceContexts and attach them to the right scalars + // so the resolver can demonstrate per-scalar lookup. + parentCtx := &node.SourceContext{Environment: types.Mapping{"FOO": "parent-value"}} + includeCtx := &node.SourceContext{Environment: types.Mapping{"FOO": "include-value"}} + + origins := map[*yaml.Node]*node.SourceContext{} + var webFOO, apiFOO *yaml.Node + _ = node.Walk(&doc, func(p tree.Path, n *yaml.Node) error { + switch p.String() { + case "services.web.environment.[]": + if n.Value == "FOO" { + webFOO = n + } + case "services.api.environment.[]": + if n.Value == "FOO" { + apiFOO = n + } + } + return nil + }) + assert.Assert(t, webFOO != nil && apiFOO != nil) + origins[webFOO] = parentCtx + origins[apiFOO] = includeCtx + + ResolveEnvironmentNode(&doc, origins) + + var m map[string]any + assert.NilError(t, doc.Decode(&m)) + web := m["services"].(map[string]any)["web"].(map[string]any)["environment"].([]any) + api := m["services"].(map[string]any)["api"].(map[string]any)["environment"].([]any) + assert.Equal(t, web[0], "FOO=parent-value") + assert.Equal(t, web[1], "BAR=2", "key=value entries are left alone") + assert.Equal(t, api[0], "FOO=include-value") +} + +func TestResolveEnvironmentNode_MissingVariableLeftAlone(t *testing.T) { + src := ` +services: + web: + environment: + - UNKNOWN +` + var doc yaml.Node + assert.NilError(t, yaml.Unmarshal([]byte(src), &doc)) + + origins := map[*yaml.Node]*node.SourceContext{} + var bare *yaml.Node + _ = node.Walk(&doc, func(p tree.Path, n *yaml.Node) error { + if p.String() == "services.web.environment.[]" && n.Value == "UNKNOWN" { + bare = n + } + return nil + }) + assert.Assert(t, bare != nil) + origins[bare] = &node.SourceContext{Environment: types.Mapping{"OTHER": "value"}} + + ResolveEnvironmentNode(&doc, origins) + + var m map[string]any + assert.NilError(t, doc.Decode(&m)) + env := m["services"].(map[string]any)["web"].(map[string]any)["environment"].([]any) + assert.Equal(t, env[0], "UNKNOWN", "unresolved keys stay bare") +} + +func TestResolveEnvironmentNode_NoServicesNoOp(t *testing.T) { + var doc yaml.Node + assert.NilError(t, yaml.Unmarshal([]byte("networks: {default: {}}"), &doc)) + // Must not panic on configs without services. + ResolveEnvironmentNode(&doc, map[*yaml.Node]*node.SourceContext{}) +} diff --git a/loader/testdata/include/env_file/compose.yaml b/loader/testdata/include/env_file/compose.yaml new file mode 100644 index 000000000..2ba05b6a3 --- /dev/null +++ b/loader/testdata/include/env_file/compose.yaml @@ -0,0 +1,6 @@ +include: + - path: sub/compose.yaml + env_file: sub/local.env +services: + app: + env_file: override.env diff --git a/loader/testdata/include/env_file/override.env b/loader/testdata/include/env_file/override.env new file mode 100644 index 000000000..29efe0070 --- /dev/null +++ b/loader/testdata/include/env_file/override.env @@ -0,0 +1 @@ +OVR=${BAR:-fallback} diff --git a/loader/testdata/include/env_file/sub/compose.yaml b/loader/testdata/include/env_file/sub/compose.yaml new file mode 100644 index 000000000..b118907a6 --- /dev/null +++ b/loader/testdata/include/env_file/sub/compose.yaml @@ -0,0 +1,4 @@ +services: + app: + image: alpine + env_file: extra.env diff --git a/loader/testdata/include/env_file/sub/extra.env b/loader/testdata/include/env_file/sub/extra.env new file mode 100644 index 000000000..d93170038 --- /dev/null +++ b/loader/testdata/include/env_file/sub/extra.env @@ -0,0 +1 @@ +FOO=$BAR diff --git a/loader/testdata/include/env_file/sub/local.env b/loader/testdata/include/env_file/sub/local.env new file mode 100644 index 000000000..09d50138c --- /dev/null +++ b/loader/testdata/include/env_file/sub/local.env @@ -0,0 +1 @@ +BAR=bar diff --git a/loader/testdata/include/secret_env/compose.yaml b/loader/testdata/include/secret_env/compose.yaml new file mode 100644 index 000000000..93786e204 --- /dev/null +++ b/loader/testdata/include/secret_env/compose.yaml @@ -0,0 +1,8 @@ +include: + - path: sub/compose.yaml + env_file: + - secret.env + +services: + foo: + image: alpine diff --git a/loader/testdata/include/secret_env/secret.env b/loader/testdata/include/secret_env/secret.env new file mode 100644 index 000000000..da4af7982 --- /dev/null +++ b/loader/testdata/include/secret_env/secret.env @@ -0,0 +1 @@ +MY_SECRET=shadoks diff --git a/loader/testdata/include/secret_env/sub/compose.yaml b/loader/testdata/include/secret_env/sub/compose.yaml new file mode 100644 index 000000000..f87416f69 --- /dev/null +++ b/loader/testdata/include/secret_env/sub/compose.yaml @@ -0,0 +1,8 @@ +services: + consumer: + image: alpine + secrets: + - scoped +secrets: + scoped: + environment: MY_SECRET diff --git a/override/node.go b/override/node.go new file mode 100644 index 000000000..315ead44e --- /dev/null +++ b/override/node.go @@ -0,0 +1,501 @@ +/* + Copyright 2020 The Compose Specification Authors. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +*/ + +package override + +import ( + "cmp" + "fmt" + "slices" + "strings" + + "go.yaml.in/yaml/v4" + + "github.com/compose-spec/compose-go/v3/tree" +) + +// MergeNode merges two yaml.Node trees using the same per-path override / +// append rules as MergeYaml, but without ever round-tripping through +// map[string]any. The left tree (override) is folded into the right tree +// (base): for mappings, the left keys overwrite or recurse into the matching +// right keys; for sequences, the default behavior is append; for scalars, +// left wins. Paths declared in mergeSpecialsNode override the default +// behavior with a per-key strategy (append-merged, deduplicated, etc.). +// +// The returned node is the right tree, mutated in place. Callers must not +// rely on a particular ordering of keys in mappings; insertion order is the +// existing keys from the right tree followed by any new keys from the left. +// +// MergeNode expects both inputs to have had their aliases unfolded +// beforehand (see node.NormalizeAliases). It does not follow AliasNode +// values. +func MergeNode(right, left *yaml.Node, p tree.Path) (*yaml.Node, error) { + right = unwrapDocumentNode(right) + left = unwrapDocumentNode(left) + if left == nil { + return right, nil + } + if right == nil { + return left, nil + } + + for pattern, merger := range mergeSpecialsNode { + if p.Matches(pattern) { + return merger(right, left, p) + } + } + + switch right.Kind { + case yaml.MappingNode: + if left.Kind != yaml.MappingNode { + return nil, fmt.Errorf("cannot override %s", p) + } + return mergeMappingsNode(right, left, p) + case yaml.SequenceNode: + if left.Kind != yaml.SequenceNode { + return nil, fmt.Errorf("cannot override %s", p) + } + right.Content = append(right.Content, left.Content...) + return right, nil + default: + return left, nil + } +} + +type nodeMerger func(*yaml.Node, *yaml.Node, tree.Path) (*yaml.Node, error) + +// mergeSpecialsNode mirrors mergeSpecials but operates on *yaml.Node. The +// entries are kept in sync between the two maps; the v2 map disappears when +// the legacy map[string]any path is removed. +var mergeSpecialsNode = map[tree.Path]nodeMerger{} + +func init() { + mergeSpecialsNode["networks.*.ipam.config"] = mergeIPAMConfigNode + mergeSpecialsNode["networks.*.labels"] = mergeToSequenceNode + mergeSpecialsNode["volumes.*.labels"] = mergeToSequenceNode + mergeSpecialsNode["services.*.annotations"] = mergeToSequenceNode + mergeSpecialsNode["services.*.build"] = mergeBuildNode + mergeSpecialsNode["services.*.build.args"] = mergeToSequenceNode + mergeSpecialsNode["services.*.build.additional_contexts"] = mergeToSequenceNode + mergeSpecialsNode["services.*.build.extra_hosts"] = mergeExtraHostsNode + mergeSpecialsNode["services.*.build.labels"] = mergeToSequenceNode + mergeSpecialsNode["services.*.command"] = overrideNode + mergeSpecialsNode["services.*.depends_on"] = mergeDependsOnNode + mergeSpecialsNode["services.*.deploy.labels"] = mergeToSequenceNode + mergeSpecialsNode["services.*.dns"] = mergeToSequenceNode + mergeSpecialsNode["services.*.dns_opt"] = mergeToSequenceNode + mergeSpecialsNode["services.*.dns_search"] = mergeToSequenceNode + mergeSpecialsNode["services.*.entrypoint"] = overrideNode + mergeSpecialsNode["services.*.env_file"] = mergeToSequenceNode + mergeSpecialsNode["services.*.label_file"] = mergeToSequenceNode + mergeSpecialsNode["services.*.environment"] = mergeToSequenceNode + mergeSpecialsNode["services.*.extra_hosts"] = mergeExtraHostsNode + mergeSpecialsNode["services.*.healthcheck.test"] = overrideNode + mergeSpecialsNode["services.*.labels"] = mergeToSequenceNode + mergeSpecialsNode["services.*.volumes.*.volume.labels"] = mergeToSequenceNode + mergeSpecialsNode["services.*.logging"] = mergeLoggingNode + mergeSpecialsNode["services.*.models"] = mergeModelsNode + mergeSpecialsNode["services.*.networks"] = mergeNetworksNode + mergeSpecialsNode["services.*.sysctls"] = mergeToSequenceNode + mergeSpecialsNode["services.*.tmpfs"] = mergeToSequenceNode + mergeSpecialsNode["services.*.ulimits.*"] = mergeUlimitNode +} + +// mergeMappingsNode folds the left mapping into the right mapping. For each +// (key, value) of left: +// - if right has no entry for key, the (key, value) pair is appended; +// - otherwise, MergeNode is invoked recursively at the next path, +// and the result replaces right's value for that key. +// +// The order of right's existing keys is preserved; new keys from left are +// appended at the end. +func mergeMappingsNode(right, left *yaml.Node, p tree.Path) (*yaml.Node, error) { + keyIdx := mappingKeyIndex(right) + for i := 0; i+1 < len(left.Content); i += 2 { + key := left.Content[i] + value := left.Content[i+1] + if idx, ok := keyIdx[key.Value]; ok { + merged, err := MergeNode(right.Content[idx+1], value, p.Next(key.Value)) + if err != nil { + return nil, err + } + right.Content[idx+1] = merged + continue + } + right.Content = append(right.Content, key, value) + keyIdx[key.Value] = len(right.Content) - 2 + } + return right, nil +} + +// mappingKeyIndex returns a map from each key's Value to the index of the key +// node within n.Content. Index i means the value node is at Content[i+1]. +func mappingKeyIndex(n *yaml.Node) map[string]int { + if n == nil || n.Kind != yaml.MappingNode { + return nil + } + idx := make(map[string]int, len(n.Content)/2) + for i := 0; i+1 < len(n.Content); i += 2 { + idx[n.Content[i].Value] = i + } + return idx +} + +// nodeMapGet returns the value Node for key in a MappingNode, or nil when +// the key is absent. +func nodeMapGet(n *yaml.Node, key string) *yaml.Node { + if n == nil || n.Kind != yaml.MappingNode { + return nil + } + for i := 0; i+1 < len(n.Content); i += 2 { + if n.Content[i].Value == key { + return n.Content[i+1] + } + } + return nil +} + +// unwrapDocumentNode peels off a single DocumentNode wrapper, returning the +// inner content. Useful when a Node was produced by yaml.Unmarshal directly +// rather than by a sub-decode. +func unwrapDocumentNode(n *yaml.Node) *yaml.Node { + if n != nil && n.Kind == yaml.DocumentNode && len(n.Content) == 1 { + return n.Content[0] + } + return n +} + +// overrideNode is the merger for paths where the left value replaces the +// right value wholesale (services.*.command, .entrypoint, .healthcheck.test). +func overrideNode(_, left *yaml.Node, _ tree.Path) (*yaml.Node, error) { + return left, nil +} + +// mergeLoggingNode merges logging blocks only when both files declare the +// same driver (or one of them omits it). When the drivers differ, the left +// block replaces the right block entirely — option keys are driver-specific +// and merging them would be meaningless. +func mergeLoggingNode(right, left *yaml.Node, p tree.Path) (*yaml.Node, error) { + rDriver := scalarValue(nodeMapGet(right, "driver")) + lDriver := scalarValue(nodeMapGet(left, "driver")) + rHas := nodeMapGet(right, "driver") != nil + lHas := nodeMapGet(left, "driver") != nil + if rDriver == lDriver || !rHas || !lHas { + return mergeMappingsNode(right, left, p) + } + return left, nil +} + +func scalarValue(n *yaml.Node) string { + if n == nil || n.Kind != yaml.ScalarNode { + return "" + } + return n.Value +} + +// mergeBuildNode promotes the short form (a single scalar = context path) +// into the canonical mapping {context: } before merging. +func mergeBuildNode(right, left *yaml.Node, p tree.Path) (*yaml.Node, error) { + return mergeMappingsNode(promoteBuildNode(right), promoteBuildNode(left), p) +} + +func promoteBuildNode(n *yaml.Node) *yaml.Node { + if n == nil { + return nil + } + if n.Kind == yaml.ScalarNode { + return &yaml.Node{ + Kind: yaml.MappingNode, + Tag: "!!map", + Line: n.Line, + Column: n.Column, + Content: []*yaml.Node{ + stringScalarAt("context", n.Line, n.Column), + n, + }, + } + } + return n +} + +// mergeDependsOnNode normalizes both inputs into the canonical mapping form +// before merging. The short form (list of service names) is expanded to +// {: {condition: service_started, required: true}}. +func mergeDependsOnNode(right, left *yaml.Node, p tree.Path) (*yaml.Node, error) { + defaults := func() *yaml.Node { + return &yaml.Node{ + Kind: yaml.MappingNode, + Tag: "!!map", + Content: []*yaml.Node{ + stringScalar("condition"), stringScalar("service_started"), + stringScalar("required"), boolScalar(true), + }, + } + } + return mergeMappingsNode( + convertIntoMappingNode(right, defaults), + convertIntoMappingNode(left, defaults), + p, + ) +} + +// mergeModelsNode normalizes both inputs into the canonical mapping form +// before merging. The short form (list of model names) maps each name to a +// nil value. +func mergeModelsNode(right, left *yaml.Node, p tree.Path) (*yaml.Node, error) { + return mergeMappingsNode( + convertIntoMappingNode(right, nil), + convertIntoMappingNode(left, nil), + p, + ) +} + +// mergeNetworksNode mirrors mergeModelsNode: short-form lists are expanded +// into name->nil mappings before merging. +func mergeNetworksNode(right, left *yaml.Node, p tree.Path) (*yaml.Node, error) { + return mergeMappingsNode( + convertIntoMappingNode(right, nil), + convertIntoMappingNode(left, nil), + p, + ) +} + +// mergeExtraHostsNode appends left into right while filtering out entries +// already present in right, regardless of declaration order. Each entry is +// compared by its serialized form (`hostname=ip` for mapping inputs, +// raw string for sequence inputs). +func mergeExtraHostsNode(right, left *yaml.Node, _ tree.Path) (*yaml.Node, error) { + r := convertIntoSequenceNode(right) + l := convertIntoSequenceNode(left) + seen := map[string]bool{} + for _, item := range r.Content { + seen[scalarValue(item)] = true + } + for _, item := range l.Content { + if seen[scalarValue(item)] { + continue + } + seen[scalarValue(item)] = true + r.Content = append(r.Content, item) + } + return r, nil +} + +// mergeToSequenceNode is the simple append rule used by env_file, labels, +// volumes, ports, dns, etc. Both sides are normalized to sequence form and +// concatenated; no deduplication is performed at this stage. Unicity is +// enforced later by EnforceUnicityNode where required. +func mergeToSequenceNode(right, left *yaml.Node, _ tree.Path) (*yaml.Node, error) { + r := convertIntoSequenceNode(right) + l := convertIntoSequenceNode(left) + r.Content = append(r.Content, l.Content...) + return r, nil +} + +// mergeUlimitNode merges two ulimit entries: when both are mappings (soft / +// hard form), keys are merged; otherwise the left value replaces the right. +func mergeUlimitNode(right, left *yaml.Node, p tree.Path) (*yaml.Node, error) { + if right != nil && right.Kind == yaml.MappingNode && left != nil && left.Kind == yaml.MappingNode { + return mergeMappingsNode(right, left, p) + } + return left, nil +} + +// mergeIPAMConfigNode merges two networks.*.ipam.config sequences. Each entry +// is a mapping that may include a `subnet`. Entries with a matching subnet +// are merged together; entries with a unique subnet from either side are +// preserved as-is. The result preserves left's order of newly-introduced +// entries. +func mergeIPAMConfigNode(right, left *yaml.Node, p tree.Path) (*yaml.Node, error) { + if right.Kind != yaml.SequenceNode || left.Kind != yaml.SequenceNode { + return nil, fmt.Errorf("%s: unexpected non-sequence value", p) + } + result := &yaml.Node{ + Kind: yaml.SequenceNode, + Tag: right.Tag, + Style: right.Style, + Line: right.Line, + Column: right.Column, + } + for _, original := range right.Content { + base := convertIntoMappingNode(original, nil) + matched := false + for _, override := range left.Content { + over := convertIntoMappingNode(override, nil) + if scalarValue(nodeMapGet(over, "subnet")) != scalarValue(nodeMapGet(base, "subnet")) { + continue + } + matched = true + merged, err := mergeMappingsNode(base, over, p) + if err != nil { + return nil, err + } + result.Content = append(result.Content, merged) + } + if !matched { + result.Content = append(result.Content, base) + } + } + // Append left-only entries (subnets present in left but absent in right). + knownSubnets := map[string]bool{} + for _, entry := range result.Content { + knownSubnets[scalarValue(nodeMapGet(entry, "subnet"))] = true + } + for _, override := range left.Content { + over := convertIntoMappingNode(override, nil) + subnet := scalarValue(nodeMapGet(over, "subnet")) + if knownSubnets[subnet] { + continue + } + result.Content = append(result.Content, over) + } + return result, nil +} + +// convertIntoMappingNode promotes a sequence of strings into a mapping where +// each string becomes a key. If defaults is non-nil, every new key gets a +// deep copy of the value returned by defaults() (a function so each entry +// gets a distinct copy). If the input is already a mapping, it is returned +// unchanged. +func convertIntoMappingNode(n *yaml.Node, defaults func() *yaml.Node) *yaml.Node { + if n == nil { + return &yaml.Node{Kind: yaml.MappingNode, Tag: "!!map"} + } + if n.Kind == yaml.MappingNode { + return n + } + if n.Kind == yaml.SequenceNode { + m := &yaml.Node{ + Kind: yaml.MappingNode, + Tag: "!!map", + Line: n.Line, + Column: n.Column, + } + for _, item := range n.Content { + if item.Kind != yaml.ScalarNode { + continue + } + key := &yaml.Node{ + Kind: yaml.ScalarNode, + Tag: "!!str", + Value: item.Value, + Line: item.Line, + Column: item.Column, + } + var value *yaml.Node + if defaults != nil { + value = defaults() + } else { + value = &yaml.Node{Kind: yaml.ScalarNode, Tag: "!!null", Line: item.Line, Column: item.Column} + } + m.Content = append(m.Content, key, value) + } + return m + } + return &yaml.Node{Kind: yaml.MappingNode, Tag: "!!map"} +} + +// convertIntoSequenceNode promotes mappings and scalars into a sequence of +// scalar items. A mapping {key: value} becomes a sequence of "key=value" +// strings (sorted lexicographically to keep merge results deterministic); a +// mapping value that is itself a sequence yields one "key=item" entry per +// item. A bare scalar becomes a one-element sequence. +func convertIntoSequenceNode(n *yaml.Node) *yaml.Node { + if n == nil { + return &yaml.Node{Kind: yaml.SequenceNode, Tag: "!!seq"} + } + switch n.Kind { + case yaml.SequenceNode: + return n + case yaml.MappingNode: + var values []string + for i := 0; i+1 < len(n.Content); i += 2 { + key := n.Content[i].Value + value := n.Content[i+1] + if value == nil || (value.Kind == yaml.ScalarNode && value.Tag == "!!null") { + values = append(values, key) + continue + } + if value.Kind == yaml.SequenceNode { + for _, item := range value.Content { + values = append(values, fmt.Sprintf("%s=%s", key, scalarOrInline(item))) + } + continue + } + values = append(values, fmt.Sprintf("%s=%s", key, scalarOrInline(value))) + } + slices.SortFunc(values, cmp.Compare[string]) + seq := &yaml.Node{ + Kind: yaml.SequenceNode, + Tag: "!!seq", + Line: n.Line, + Column: n.Column, + } + for _, v := range values { + seq.Content = append(seq.Content, &yaml.Node{ + Kind: yaml.ScalarNode, + Tag: "!!str", + Value: v, + Line: n.Line, + }) + } + return seq + case yaml.ScalarNode: + return &yaml.Node{ + Kind: yaml.SequenceNode, + Tag: "!!seq", + Line: n.Line, + Column: n.Column, + Content: []*yaml.Node{n}, + } + } + return &yaml.Node{Kind: yaml.SequenceNode, Tag: "!!seq"} +} + +// scalarOrInline formats a non-scalar value into a single-line string for +// use as part of a "key=value" entry built by convertIntoSequenceNode. +// Scalars are returned verbatim; sequences and mappings are flattened with +// their fields concatenated, which mirrors the v2 behavior of relying on +// fmt.Sprintf("%v", ...) over the decoded interface{}. +func scalarOrInline(n *yaml.Node) string { + if n == nil { + return "" + } + if n.Kind == yaml.ScalarNode { + return n.Value + } + var parts []string + for _, c := range n.Content { + parts = append(parts, scalarOrInline(c)) + } + return strings.Join(parts, " ") +} + +func stringScalar(value string) *yaml.Node { + return &yaml.Node{Kind: yaml.ScalarNode, Tag: "!!str", Value: value} +} + +func stringScalarAt(value string, line, col int) *yaml.Node { + return &yaml.Node{Kind: yaml.ScalarNode, Tag: "!!str", Value: value, Line: line, Column: col} +} + +func boolScalar(b bool) *yaml.Node { + v := "false" + if b { + v = "true" + } + return &yaml.Node{Kind: yaml.ScalarNode, Tag: "!!bool", Value: v} +} diff --git a/override/node_fuzz_test.go b/override/node_fuzz_test.go new file mode 100644 index 000000000..c3e4bfb35 --- /dev/null +++ b/override/node_fuzz_test.go @@ -0,0 +1,93 @@ +/* + Copyright 2020 The Compose Specification Authors. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +*/ + +package override + +import ( + "testing" + + "go.yaml.in/yaml/v4" + + "github.com/compose-spec/compose-go/v3/tree" +) + +// FuzzMergeNode feeds MergeNode arbitrary pairs of valid YAML documents +// and checks that the function returns within a bounded number of +// steps for every well-formed input and that it never panics. The +// fuzz target is a robustness gate for the merge primitive, not a +// behavioral one -- the corpus only exercises shapes the parser +// accepts. +func FuzzMergeNode(f *testing.F) { + corpus := []string{ + `services: + web: + image: nginx`, + `services: + web: + image: caddy + restart: always`, + `services: + api: + image: alpine +networks: + default: + driver: bridge`, + `x-anchor: &a + key: value +services: + web: + <<: *a + image: nginx`, + `services: + web: + ports: + - 80 + - "443:443"`, + ``, + `{}`, + } + for _, l := range corpus { + for _, r := range corpus { + f.Add(l, r) + } + } + f.Fuzz(func(t *testing.T, left, right string) { + var leftNode, rightNode yaml.Node + if err := yaml.Unmarshal([]byte(left), &leftNode); err != nil { + t.Skip() + } + if err := yaml.Unmarshal([]byte(right), &rightNode); err != nil { + t.Skip() + } + if leftNode.Kind == 0 || rightNode.Kind == 0 { + t.Skip() + } + // Unwrap the document wrapper so MergeNode sees mapping roots, + // matching the way the loader invokes it. + l := &leftNode + if l.Kind == yaml.DocumentNode && len(l.Content) == 1 { + l = l.Content[0] + } + r := &rightNode + if r.Kind == yaml.DocumentNode && len(r.Content) == 1 { + r = r.Content[0] + } + if l.Kind != yaml.MappingNode || r.Kind != yaml.MappingNode { + t.Skip() + } + _, _ = MergeNode(l, r, tree.NewPath()) + }) +} diff --git a/override/node_test.go b/override/node_test.go new file mode 100644 index 000000000..5b3f6d42d --- /dev/null +++ b/override/node_test.go @@ -0,0 +1,338 @@ +/* + Copyright 2020 The Compose Specification Authors. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +*/ + +package override + +import ( + "testing" + + "go.yaml.in/yaml/v4" + "gotest.tools/v3/assert" + + "github.com/compose-spec/compose-go/v3/tree" +) + +// mergeNodeYAML parses two YAML strings, merges them through MergeNode and +// returns the result decoded back to a map[string]any so we can compare it +// against an expected YAML snippet with the same DeepEqual helper used by +// the v2 suite. +func mergeNodeYAML(t *testing.T, right, left string) map[string]any { + t.Helper() + r := parseNode(t, right) + l := parseNode(t, left) + merged, err := MergeNode(r, l, tree.NewPath()) + assert.NilError(t, err) + var out map[string]any + assert.NilError(t, merged.Decode(&out)) + return out +} + +func parseNode(t *testing.T, src string) *yaml.Node { + t.Helper() + var doc yaml.Node + assert.NilError(t, yaml.Unmarshal([]byte(src), &doc)) + return &doc +} + +func TestMergeNode_BasicOverride(t *testing.T) { + got := mergeNodeYAML(t, ` +services: + test: + image: foo + scale: 1 +`, ` +services: + test: + image: bar + scale: 2 +`) + assert.DeepEqual(t, got, unmarshal(t, ` +services: + test: + image: bar + scale: 2 +`)) +} + +func TestMergeNode_MapAddsNewKey(t *testing.T) { + got := mergeNodeYAML(t, ` +services: + web: + image: nginx +`, ` +services: + web: + image: nginx + restart: always +`) + web := got["services"].(map[string]any)["web"].(map[string]any) + assert.Equal(t, web["restart"], "always") +} + +func TestMergeNode_SequenceAppends(t *testing.T) { + got := mergeNodeYAML(t, ` +services: + web: + ports: + - "80:80" +`, ` +services: + web: + ports: + - "443:443" +`) + ports := got["services"].(map[string]any)["web"].(map[string]any)["ports"].([]any) + assert.DeepEqual(t, ports, []any{"80:80", "443:443"}) +} + +func TestMergeNode_CommandIsOverridden(t *testing.T) { + got := mergeNodeYAML(t, ` +services: + web: + command: ["nginx", "-g", "daemon off;"] +`, ` +services: + web: + command: ["caddy", "run"] +`) + cmd := got["services"].(map[string]any)["web"].(map[string]any)["command"].([]any) + assert.DeepEqual(t, cmd, []any{"caddy", "run"}) +} + +func TestMergeNode_BuildShortFormPromoted(t *testing.T) { + got := mergeNodeYAML(t, ` +services: + web: + build: ./base +`, ` +services: + web: + build: + dockerfile: Dockerfile.dev +`) + build := got["services"].(map[string]any)["web"].(map[string]any)["build"].(map[string]any) + assert.Equal(t, build["context"], "./base") + assert.Equal(t, build["dockerfile"], "Dockerfile.dev") +} + +func TestMergeNode_DependsOnListPromoted(t *testing.T) { + got := mergeNodeYAML(t, ` +services: + web: + depends_on: + - db +`, ` +services: + web: + depends_on: + api: + condition: service_healthy +`) + deps := got["services"].(map[string]any)["web"].(map[string]any)["depends_on"].(map[string]any) + db := deps["db"].(map[string]any) + assert.Equal(t, db["condition"], "service_started") + assert.Equal(t, db["required"], true) + api := deps["api"].(map[string]any) + assert.Equal(t, api["condition"], "service_healthy") +} + +func TestMergeNode_NetworksListPromoted(t *testing.T) { + got := mergeNodeYAML(t, ` +services: + web: + networks: + - frontend +`, ` +services: + web: + networks: + backend: + priority: 5 +`) + nets := got["services"].(map[string]any)["web"].(map[string]any)["networks"].(map[string]any) + _, hasFront := nets["frontend"] + assert.Assert(t, hasFront, "frontend network preserved") + back := nets["backend"].(map[string]any) + assert.Equal(t, back["priority"], 5) +} + +func TestMergeNode_ExtraHostsDedupes(t *testing.T) { + got := mergeNodeYAML(t, ` +services: + web: + extra_hosts: + - "host1:1.2.3.4" +`, ` +services: + web: + extra_hosts: + - "host1:1.2.3.4" + - "host2:5.6.7.8" +`) + hosts := got["services"].(map[string]any)["web"].(map[string]any)["extra_hosts"].([]any) + assert.DeepEqual(t, hosts, []any{"host1:1.2.3.4", "host2:5.6.7.8"}) +} + +func TestMergeNode_LoggingSameDriverMergesOptions(t *testing.T) { + got := mergeNodeYAML(t, ` +services: + web: + logging: + driver: json-file + options: + max-size: "10m" +`, ` +services: + web: + logging: + driver: json-file + options: + max-file: "3" +`) + logging := got["services"].(map[string]any)["web"].(map[string]any)["logging"].(map[string]any) + opts := logging["options"].(map[string]any) + assert.Equal(t, opts["max-size"], "10m") + assert.Equal(t, opts["max-file"], "3") +} + +func TestMergeNode_LoggingDifferentDriverReplaces(t *testing.T) { + got := mergeNodeYAML(t, ` +services: + web: + logging: + driver: json-file + options: + max-size: "10m" +`, ` +services: + web: + logging: + driver: syslog + options: + tag: web +`) + logging := got["services"].(map[string]any)["web"].(map[string]any)["logging"].(map[string]any) + assert.Equal(t, logging["driver"], "syslog") + opts := logging["options"].(map[string]any) + _, hasMaxSize := opts["max-size"] + assert.Assert(t, !hasMaxSize, "max-size from json-file driver must not leak") + assert.Equal(t, opts["tag"], "web") +} + +func TestMergeNode_EnvironmentListMerged(t *testing.T) { + // environment uses mergeToSequence: both lists are concatenated, no + // deduplication at merge time (EnforceUnicity handles that downstream). + got := mergeNodeYAML(t, ` +services: + web: + environment: + FOO: "1" + BAR: "2" +`, ` +services: + web: + environment: + - "BAZ=3" +`) + env := got["services"].(map[string]any)["web"].(map[string]any)["environment"].([]any) + // Mapping is sorted; sequence appends after. + assert.Equal(t, len(env), 3) + // Sort-based equality check + have := map[string]bool{} + for _, e := range env { + have[e.(string)] = true + } + assert.Assert(t, have["FOO=1"]) + assert.Assert(t, have["BAR=2"]) + assert.Assert(t, have["BAZ=3"]) +} + +func TestMergeNode_IPAMSubnetMatching(t *testing.T) { + got := mergeNodeYAML(t, ` +networks: + app: + ipam: + config: + - subnet: 10.0.0.0/24 + gateway: 10.0.0.1 + - subnet: 10.1.0.0/24 +`, ` +networks: + app: + ipam: + config: + - subnet: 10.0.0.0/24 + gateway: 10.0.0.254 + - subnet: 10.2.0.0/24 +`) + conf := got["networks"].(map[string]any)["app"].(map[string]any)["ipam"].(map[string]any)["config"].([]any) + bySubnet := map[string]map[string]any{} + for _, e := range conf { + m := e.(map[string]any) + bySubnet[m["subnet"].(string)] = m + } + assert.Equal(t, bySubnet["10.0.0.0/24"]["gateway"], "10.0.0.254", "matching subnet: left wins") + _, ok := bySubnet["10.1.0.0/24"] + assert.Assert(t, ok, "10.1.0.0/24 preserved from right") + _, ok = bySubnet["10.2.0.0/24"] + assert.Assert(t, ok, "10.2.0.0/24 appended from left") +} + +func TestMergeNode_ScalarOverride(t *testing.T) { + got := mergeNodeYAML(t, `name: a`, `name: b`) + assert.Equal(t, got["name"], "b") +} + +func TestMergeNode_NilLeftReturnsRight(t *testing.T) { + right := parseNode(t, `services: {web: {image: nginx}}`) + merged, err := MergeNode(right, nil, tree.NewPath()) + assert.NilError(t, err) + var out map[string]any + assert.NilError(t, merged.Decode(&out)) + assert.Equal(t, out["services"].(map[string]any)["web"].(map[string]any)["image"], "nginx") +} + +func TestMergeNode_PreservesLineNumbers(t *testing.T) { + right := parseNode(t, "services:\n web:\n image: nginx\n") + left := parseNode(t, "services:\n web:\n restart: always\n") + merged, err := MergeNode(right, left, tree.NewPath()) + assert.NilError(t, err) + + // Find the image scalar; it must keep its line 3 from right. + imageLine := 0 + restartLine := 0 + var visit func(n *yaml.Node) + visit = func(n *yaml.Node) { + if n.Kind == yaml.MappingNode { + for i := 0; i+1 < len(n.Content); i += 2 { + if n.Content[i].Value == "image" { + imageLine = n.Content[i+1].Line + } + if n.Content[i].Value == "restart" { + restartLine = n.Content[i+1].Line + } + visit(n.Content[i+1]) + } + return + } + for _, c := range n.Content { + visit(c) + } + } + visit(merged) + assert.Equal(t, imageLine, 3, "right's image scalar retains its source line") + assert.Equal(t, restartLine, 3, "left's restart scalar retains its own source line") +} diff --git a/override/uncity_node.go b/override/uncity_node.go new file mode 100644 index 000000000..1343d7930 --- /dev/null +++ b/override/uncity_node.go @@ -0,0 +1,265 @@ +/* + Copyright 2020 The Compose Specification Authors. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +*/ + +package override + +import ( + "fmt" + "strings" + + "go.yaml.in/yaml/v4" + + "github.com/compose-spec/compose-go/v3/format" + "github.com/compose-spec/compose-go/v3/tree" +) + +// nodeIndexer extracts a stable identity key from a sequence entry. Entries +// sharing the same key are de-duplicated by EnforceUnicityNode, with the +// last occurrence winning — matching the v2 semantics where the override +// file takes precedence over the base. +type nodeIndexer func(*yaml.Node, tree.Path) (string, error) + +// uniqueNode mirrors override.unique but holds the Node-typed indexers. +// Entries are kept in sync between the two maps; the legacy map disappears +// when the v2 map[string]any path is removed. +var uniqueNode = map[tree.Path]nodeIndexer{} + +func init() { + uniqueNode["networks.*.labels"] = keyValueIndexerNode + uniqueNode["networks.*.ipam.options"] = keyValueIndexerNode + uniqueNode["services.*.annotations"] = keyValueIndexerNode + uniqueNode["services.*.build.args"] = keyValueIndexerNode + uniqueNode["services.*.build.additional_contexts"] = keyValueIndexerNode + uniqueNode["services.*.build.platform"] = keyValueIndexerNode + uniqueNode["services.*.build.tags"] = keyValueIndexerNode + uniqueNode["services.*.build.labels"] = keyValueIndexerNode + uniqueNode["services.*.cap_add"] = keyValueIndexerNode + uniqueNode["services.*.cap_drop"] = keyValueIndexerNode + uniqueNode["services.*.configs"] = mountIndexerNode("") + uniqueNode["services.*.deploy.labels"] = keyValueIndexerNode + uniqueNode["services.*.dns"] = keyValueIndexerNode + uniqueNode["services.*.dns_opt"] = keyValueIndexerNode + uniqueNode["services.*.dns_search"] = keyValueIndexerNode + uniqueNode["services.*.environment"] = keyValueIndexerNode + uniqueNode["services.*.env_file"] = envFileIndexerNode + uniqueNode["services.*.expose"] = exposeIndexerNode + uniqueNode["services.*.labels"] = keyValueIndexerNode + uniqueNode["services.*.links"] = keyValueIndexerNode + uniqueNode["services.*.networks.*.aliases"] = keyValueIndexerNode + uniqueNode["services.*.networks.*.link_local_ips"] = keyValueIndexerNode + uniqueNode["services.*.ports"] = portIndexerNode + uniqueNode["services.*.profiles"] = keyValueIndexerNode + uniqueNode["services.*.secrets"] = mountIndexerNode("/run/secrets") + uniqueNode["services.*.sysctls"] = keyValueIndexerNode + uniqueNode["services.*.tmpfs"] = keyValueIndexerNode + uniqueNode["services.*.volumes"] = volumeIndexerNode + uniqueNode["services.*.devices"] = deviceMappingIndexerNode +} + +// EnforceUnicityNode removes duplicated entries in any sequence whose path +// matches uniqueNode. Inside each affected sequence, entries are indexed by +// the configured nodeIndexer; later occurrences replace earlier ones at the +// same key. Mappings outside the configured paths are recursed into but +// untouched. +// +// The function mutates root in place and returns it for convenience. +func EnforceUnicityNode(root *yaml.Node) (*yaml.Node, error) { + root = unwrapDocumentNode(root) + if err := enforceUnicityNode(root, tree.NewPath()); err != nil { + return nil, err + } + return root, nil +} + +func enforceUnicityNode(n *yaml.Node, p tree.Path) error { + if n == nil { + return nil + } + switch n.Kind { + case yaml.MappingNode: + for i := 0; i+1 < len(n.Content); i += 2 { + key := n.Content[i].Value + if err := enforceUnicityNode(n.Content[i+1], p.Next(key)); err != nil { + return err + } + } + case yaml.SequenceNode: + for pattern, indexer := range uniqueNode { + if !p.Matches(pattern) { + continue + } + result := make([]*yaml.Node, 0, len(n.Content)) + keys := map[string]int{} + for i, entry := range n.Content { + key, err := indexer(entry, p.Next(fmt.Sprintf("[%d]", i))) + if err != nil { + return err + } + if j, ok := keys[key]; ok { + result[j] = entry + continue + } + result = append(result, entry) + keys[key] = len(result) - 1 + } + n.Content = result + return nil + } + // Recurse into nested containers when the sequence itself is not a + // unicity target (the entries may themselves contain mappings whose + // children are unicity-enforced). + for i, entry := range n.Content { + if err := enforceUnicityNode(entry, p.Next(fmt.Sprintf("[%d]", i))); err != nil { + return err + } + } + } + return nil +} + +func keyValueIndexerNode(n *yaml.Node, p tree.Path) (string, error) { + if n == nil || n.Kind != yaml.ScalarNode { + return "", fmt.Errorf("%s: unexpected non-scalar entry", p) + } + key, _, found := strings.Cut(n.Value, "=") + if found { + return key, nil + } + return n.Value, nil +} + +func volumeIndexerNode(n *yaml.Node, p tree.Path) (string, error) { + if n == nil { + return "", nil + } + switch n.Kind { + case yaml.MappingNode: + target := nodeMapGet(n, "target") + if target == nil || target.Kind != yaml.ScalarNode { + return "", fmt.Errorf("service volume %s is missing a mount target", p) + } + return target.Value, nil + case yaml.ScalarNode: + volume, err := format.ParseVolume(n.Value) + if err != nil { + return "", err + } + return volume.Target, nil + } + return "", nil +} + +func deviceMappingIndexerNode(n *yaml.Node, p tree.Path) (string, error) { + if n == nil { + return "", nil + } + switch n.Kind { + case yaml.MappingNode: + target := nodeMapGet(n, "target") + if target == nil || target.Kind != yaml.ScalarNode { + return "", fmt.Errorf("service device %s is missing a mount target", p) + } + return target.Value, nil + case yaml.ScalarNode: + parts := strings.Split(n.Value, ":") + if len(parts) == 1 { + return parts[0], nil + } + return parts[1], nil + } + return "", nil +} + +func exposeIndexerNode(n *yaml.Node, p tree.Path) (string, error) { + if n == nil || n.Kind != yaml.ScalarNode { + return "", fmt.Errorf("%s: unsupported expose value", p) + } + return n.Value, nil +} + +func mountIndexerNode(defaultPath string) nodeIndexer { + return func(n *yaml.Node, p tree.Path) (string, error) { + if n == nil { + return "", nil + } + switch n.Kind { + case yaml.ScalarNode: + return fmt.Sprintf("%s/%s", defaultPath, n.Value), nil + case yaml.MappingNode: + if target := nodeMapGet(n, "target"); target != nil && target.Kind == yaml.ScalarNode { + return target.Value, nil + } + source := nodeMapGet(n, "source") + if source != nil && source.Kind == yaml.ScalarNode { + return fmt.Sprintf("%s/%s", defaultPath, source.Value), nil + } + return "", fmt.Errorf("%s: missing target or source", p) + } + return "", fmt.Errorf("%s: unsupported mount value", p) + } +} + +func portIndexerNode(n *yaml.Node, p tree.Path) (string, error) { + if n == nil { + return "", nil + } + switch n.Kind { + case yaml.ScalarNode: + // Could be a bare port number (int-tagged or untagged scalar) or a + // "host:container/proto" short-form string. Use the literal Value as + // the indexer key in both cases — different surface syntaxes that + // describe the same port end up de-duplicated by EnforceUnicity at a + // later stage if Canonical has normalized them by then. + return n.Value, nil + case yaml.MappingNode: + target := nodeMapGet(n, "target") + if target == nil { + return "", fmt.Errorf("service ports %s is missing a target port", p) + } + published := nodeMapGet(n, "published") + publishedStr := "" + if published != nil { + publishedStr = published.Value + } + host := scalarValueOrDefault(nodeMapGet(n, "host_ip"), "0.0.0.0") + protocol := scalarValueOrDefault(nodeMapGet(n, "protocol"), "tcp") + return fmt.Sprintf("%s:%s:%s/%s", host, publishedStr, target.Value, protocol), nil + } + return "", nil +} + +func envFileIndexerNode(n *yaml.Node, p tree.Path) (string, error) { + if n == nil { + return "", nil + } + switch n.Kind { + case yaml.ScalarNode: + return n.Value, nil + case yaml.MappingNode: + if pathValue := nodeMapGet(n, "path"); pathValue != nil && pathValue.Kind == yaml.ScalarNode { + return pathValue.Value, nil + } + return "", fmt.Errorf("environment path attribute %s is missing", p) + } + return "", nil +} + +func scalarValueOrDefault(n *yaml.Node, fallback string) string { + if n == nil || n.Kind != yaml.ScalarNode || n.Value == "" { + return fallback + } + return n.Value +} diff --git a/override/uncity_node_test.go b/override/uncity_node_test.go new file mode 100644 index 000000000..4543999fb --- /dev/null +++ b/override/uncity_node_test.go @@ -0,0 +1,138 @@ +/* + Copyright 2020 The Compose Specification Authors. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +*/ + +package override + +import ( + "testing" + + "gotest.tools/v3/assert" +) + +// enforceUnicityYAML parses src, runs EnforceUnicityNode, decodes the result +// and returns the decoded map[string]any for shape comparison. +func enforceUnicityYAML(t *testing.T, src string) map[string]any { + t.Helper() + root := parseNode(t, src) + out, err := EnforceUnicityNode(root) + assert.NilError(t, err) + var m map[string]any + assert.NilError(t, out.Decode(&m)) + return m +} + +func TestEnforceUnicityNode_EnvironmentLaterWins(t *testing.T) { + got := enforceUnicityYAML(t, ` +services: + web: + environment: + - FOO=1 + - BAR=2 + - FOO=overridden +`) + env := got["services"].(map[string]any)["web"].(map[string]any)["environment"].([]any) + // FOO=overridden replaces FOO=1 at the original FOO slot; BAR=2 stays. + assert.DeepEqual(t, env, []any{"FOO=overridden", "BAR=2"}) +} + +func TestEnforceUnicityNode_LabelsDeduped(t *testing.T) { + got := enforceUnicityYAML(t, ` +services: + web: + labels: + - com.example.a=1 + - com.example.a=2 + - com.example.b=3 +`) + labels := got["services"].(map[string]any)["web"].(map[string]any)["labels"].([]any) + assert.DeepEqual(t, labels, []any{"com.example.a=2", "com.example.b=3"}) +} + +func TestEnforceUnicityNode_PortsShortFormDeduped(t *testing.T) { + got := enforceUnicityYAML(t, ` +services: + web: + ports: + - "8080:80" + - "8080:80" + - "8443:443" +`) + ports := got["services"].(map[string]any)["web"].(map[string]any)["ports"].([]any) + assert.DeepEqual(t, ports, []any{"8080:80", "8443:443"}) +} + +func TestEnforceUnicityNode_VolumesByTarget(t *testing.T) { + got := enforceUnicityYAML(t, ` +services: + web: + volumes: + - "./old:/data" + - "./new:/data" + - "./logs:/var/log" +`) + vols := got["services"].(map[string]any)["web"].(map[string]any)["volumes"].([]any) + assert.DeepEqual(t, vols, []any{"./new:/data", "./logs:/var/log"}) +} + +func TestEnforceUnicityNode_PortsLongFormByTuple(t *testing.T) { + got := enforceUnicityYAML(t, ` +services: + web: + ports: + - target: 80 + published: 8080 + protocol: tcp + - target: 80 + published: 8080 + protocol: tcp + - target: 443 + published: 8443 + protocol: tcp +`) + ports := got["services"].(map[string]any)["web"].(map[string]any)["ports"].([]any) + assert.Equal(t, len(ports), 2) +} + +func TestEnforceUnicityNode_LeavesNonUnicityPathsAlone(t *testing.T) { + // services.*.command is overridden, not de-duplicated by EnforceUnicity. + got := enforceUnicityYAML(t, ` +services: + web: + command: + - sh + - "-c" + - echo hi + - echo hi +`) + cmd := got["services"].(map[string]any)["web"].(map[string]any)["command"].([]any) + assert.Equal(t, len(cmd), 4, "command sequence is not unicity-controlled") +} + +func TestEnforceUnicityNode_NetworkAliasesDeduped(t *testing.T) { + got := enforceUnicityYAML(t, ` +services: + web: + networks: + default: + aliases: + - web + - api + - web + - workers +`) + aliases := got["services"].(map[string]any)["web"].(map[string]any)["networks"].(map[string]any)["default"].(map[string]any)["aliases"].([]any) + assert.DeepEqual(t, aliases, []any{"web", "api", "workers"}) +} diff --git a/paths/node.go b/paths/node.go new file mode 100644 index 000000000..61fc6dee8 --- /dev/null +++ b/paths/node.go @@ -0,0 +1,471 @@ +/* + Copyright 2020 The Compose Specification Authors. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +*/ + +package paths + +import ( + "errors" + "path" + "path/filepath" + "strings" + + "go.yaml.in/yaml/v4" + + "github.com/compose-spec/compose-go/v3/tree" + "github.com/compose-spec/compose-go/v3/types" + "github.com/compose-spec/compose-go/v3/utils" +) + +// NodeResolverOptions configures ResolveRelativePathsNode. +type NodeResolverOptions struct { + // WorkingDirFor returns the working directory against which a relative + // path attached to n should be resolved. Letting the caller pick a + // per-node WorkingDir is what enables the v3 fix where a relative path + // declared inside an included file is resolved against the include's + // project_directory rather than the project root. + // + // When nil, every resolution uses WorkingDir. + WorkingDirFor func(n *yaml.Node) string + + // WorkingDir is the fall-back single working directory used when + // WorkingDirFor is nil. Provided for v2-style callers that have a + // single project root. + WorkingDir string + + // Remotes is a list of predicates that flag a path string as a remote + // URL, exempting it from absolute-path conversion. Mirrors the v2 + // behavior of the same name. + Remotes []RemoteResource + + // ExcludePaths is the optional list of tree.Path patterns to skip + // during traversal. Resolvers registered at any of these patterns are + // not invoked, and the walker continues into the children as if no + // rule were registered. Used by the include pre-resolution to defer + // extends.file resolution to the orchestrator extends pass, which + // needs the original relative reference. + ExcludePaths []string +} + +// ResolveRelativePathsNode walks root and converts relative path scalars to +// absolute paths in place, using a per-scalar WorkingDir chosen by +// WorkingDirFor. +// +// The set of recognized paths mirrors the v2 ResolveRelativePaths registry: +// services.*.build.context, services.*.build.additional_contexts.*, +// services.*.build.ssh.*, services.*.env_file.*.path, services.*.label_file.*, +// services.*.extends.file, services.*.develop.watch.*.path, services.*.volumes.*, +// configs.*.file, secrets.*.file, include.path, include.project_directory, +// include.env_file, and volumes.*. The Node version reads Tag information +// directly, so callers do not need to canonicalize beforehand only to +// distinguish short- from long-form values (for example services.*.volumes.* +// inspects the entry's Kind rather than its decoded Go type). +func ResolveRelativePathsNode(root *yaml.Node, opts NodeResolverOptions) error { + if opts.WorkingDirFor == nil { + wd := opts.WorkingDir + opts.WorkingDirFor = func(*yaml.Node) string { return wd } + } + r := &nodeResolverState{opts: opts} + r.resolvers = map[tree.Path]func(*yaml.Node) error{ + "services.*.build": r.absBuild, + "services.*.build.context": r.absContextScalar, + "services.*.build.additional_contexts.*": r.absContextScalar, + "services.*.build.ssh.*": r.absSSHEntry, + "services.*.env_file": r.absEnvFileShortForm, + "services.*.env_file.*": r.absEnvFile, + "services.*.env_file.*.path": r.absScalar, + "services.*.label_file": r.absScalarMaybeSequence, + "services.*.label_file.*": r.absScalar, + "services.*.extends.file": r.absExtendsScalar, + "services.*.develop.watch.*.path": r.absSymbolicLinkScalar, + "services.*.volumes.*": r.absVolumeMount, + "configs.*.file": r.maybeUnixScalar, + "secrets.*.file": r.maybeUnixScalar, + "include.path": r.absScalarMaybeSequence, + "include.project_directory": r.absScalar, + "include.env_file": r.absScalarMaybeSequence, + "volumes.*": r.volumeDriverOpts, + } + for _, ex := range opts.ExcludePaths { + delete(r.resolvers, tree.NewPath(ex)) + } + return r.walk(root, tree.NewPath()) +} + +type nodeResolverState struct { + opts NodeResolverOptions + resolvers map[tree.Path]func(*yaml.Node) error +} + +func (r *nodeResolverState) walk(n *yaml.Node, p tree.Path) error { + if n == nil { + return nil + } + if n.Kind == yaml.DocumentNode { + for _, c := range n.Content { + if err := r.walk(c, p); err != nil { + return err + } + } + return nil + } + for pattern, fn := range r.resolvers { + if p.Matches(pattern) { + return fn(n) + } + } + switch n.Kind { + case yaml.MappingNode: + for i := 0; i+1 < len(n.Content); i += 2 { + key := n.Content[i].Value + if err := r.walk(n.Content[i+1], p.Next(key)); err != nil { + return err + } + } + case yaml.SequenceNode: + for _, c := range n.Content { + if err := r.walk(c, p.Next(tree.PathMatchList)); err != nil { + return err + } + } + } + return nil +} + +func (r *nodeResolverState) isRemoteResource(p string) bool { + for _, remote := range r.opts.Remotes { + if remote(p) { + return true + } + } + return false +} + +// absScalar resolves a single ScalarNode to an absolute path. A nil / empty +// scalar is left untouched; a non-scalar node is also left untouched (a +// caller that targets a path expecting a scalar but receives a sequence has +// pre-canonicalization shape and is handled by absScalarMaybeSequence). +// Scalars tagged !!null are skipped so post-canonical placeholders (e.g. +// the `default: null` entry produced by ssh canonicalization) keep their +// type instead of being rewritten to a path string. +func (r *nodeResolverState) absScalar(n *yaml.Node) error { + if n == nil || n.Kind != yaml.ScalarNode || n.Value == "" || n.Tag == "!!null" { + return nil + } + expanded := ExpandUser(n.Value) + if filepath.IsAbs(expanded) { + n.Value = expanded + return nil + } + wd := r.opts.WorkingDirFor(n) + if wd == "" { + return nil + } + n.Value = filepath.Join(wd, expanded) + return nil +} + +// absScalarMaybeSequence accepts either a single ScalarNode or a SequenceNode +// of scalars and resolves each. Used for include.path (which may be a single +// path or a list) and include.env_file (same). +func (r *nodeResolverState) absScalarMaybeSequence(n *yaml.Node) error { + if n == nil { + return nil + } + if n.Kind == yaml.SequenceNode { + for _, c := range n.Content { + if err := r.absScalar(c); err != nil { + return err + } + } + return nil + } + return r.absScalar(n) +} + +// maybeUnixScalar resolves a path scalar against the working directory, +// unless the value is already an absolute Unix or Windows path. Mirrors +// maybeUnixPath in paths/unix.go. Skips !!null scalars and empty values so +// post-canonical null placeholders are not rewritten. +func (r *nodeResolverState) maybeUnixScalar(n *yaml.Node) error { + if n == nil || n.Kind != yaml.ScalarNode || n.Value == "" || n.Tag == "!!null" { + return nil + } + expanded := ExpandUser(n.Value) + if !path.IsAbs(expanded) && !IsWindowsAbs(expanded) { + if filepath.IsAbs(expanded) { + n.Value = expanded + return nil + } + wd := r.opts.WorkingDirFor(n) + if wd == "" { + return nil + } + n.Value = filepath.Join(wd, expanded) + return nil + } + n.Value = expanded + return nil +} + +// absBuild handles services.*.build in both canonical short and long form. +// Short form (a scalar Value is the build context path) is treated as a +// context path and resolved against the layer working directory. Long form +// (a mapping with context / additional_contexts / ssh fields) is recursed +// into by walking the mapping's children — this is needed because the +// generic walker stops at the first matching pattern, so it cannot descend +// past services.*.build to reach services.*.build.context on its own. +// +// Running paths before canonicalization avoids the loss of pointer identity +// that the CanonicalNode bridge would otherwise cause; this handler keeps +// both shapes supported until per-transformer Node ports are in place. +func (r *nodeResolverState) absBuild(n *yaml.Node) error { + if n == nil { + return nil + } + if n.Kind == yaml.ScalarNode { + return r.absContextScalar(n) + } + if n.Kind != yaml.MappingNode { + return nil + } + for i := 0; i+1 < len(n.Content); i += 2 { + key := n.Content[i].Value + val := n.Content[i+1] + switch key { + case "context": + if err := r.absContextScalar(val); err != nil { + return err + } + case "additional_contexts": + if val.Kind == yaml.MappingNode { + for j := 1; j < len(val.Content); j += 2 { + if err := r.absContextScalar(val.Content[j]); err != nil { + return err + } + } + } + if val.Kind == yaml.SequenceNode { + for _, item := range val.Content { + if err := r.absContextScalar(item); err != nil { + return err + } + } + } + case "ssh": + switch val.Kind { + case yaml.SequenceNode: + for _, item := range val.Content { + if err := r.absSSHEntry(item); err != nil { + return err + } + } + case yaml.MappingNode: + for j := 1; j < len(val.Content); j += 2 { + if err := r.maybeUnixScalar(val.Content[j]); err != nil { + return err + } + } + } + } + } + return nil +} + +// absSSHEntry handles services.*.build.ssh.* in the short form (sequence +// of strings) and the post-canonical mapping form. The short form entries +// are either a bare key (e.g. "default") or "key=path"; only the path +// portion after the `=` is resolved against the working directory. Post- +// canonical, ssh is a mapping whose values are scalar paths and are +// resolved directly. +func (r *nodeResolverState) absSSHEntry(n *yaml.Node) error { + if n == nil || n.Kind != yaml.ScalarNode { + return nil + } + key, value, hasEq := strings.Cut(n.Value, "=") + if !hasEq { + // Bare key (e.g. "default") — nothing to resolve. + return nil + } + tmp := &yaml.Node{Kind: yaml.ScalarNode, Value: value, Line: n.Line, Column: n.Column} + if err := r.maybeUnixScalar(tmp); err != nil { + return err + } + n.Value = key + "=" + tmp.Value + return nil +} + +// absEnvFileShortForm handles services.*.env_file in every shape it can +// take before Canonical normalizes it: a scalar (env_file: ./foo), a +// sequence of scalars (env_file: [./foo, ./bar]) or a sequence of mappings +// (env_file: [{path: ./foo, required: false}]). The generic walker stops +// at the first matching pattern, so this handler explicitly recurses into +// each sequence shape rather than letting the per-element pattern below +// take over. +func (r *nodeResolverState) absEnvFileShortForm(n *yaml.Node) error { + if n == nil { + return nil + } + switch n.Kind { + case yaml.ScalarNode: + return r.absScalar(n) + case yaml.SequenceNode: + for _, item := range n.Content { + if err := r.absEnvFile(item); err != nil { + return err + } + } + } + return nil +} + +// absEnvFile handles services.*.env_file.* entries. The short form is a +// scalar path; the long form is a mapping with a `path` field. Both are +// resolved against the scalar working directory. For the long form the +// per-field handler ("services.*.env_file.*.path") takes over once the +// walker recurses into the mapping, so this function only acts on the +// short form to avoid double resolution. +func (r *nodeResolverState) absEnvFile(n *yaml.Node) error { + if n == nil { + return nil + } + if n.Kind == yaml.ScalarNode { + return r.absScalar(n) + } + if n.Kind != yaml.MappingNode { + return nil + } + for i := 0; i+1 < len(n.Content); i += 2 { + if n.Content[i].Value == "path" { + if err := r.absScalar(n.Content[i+1]); err != nil { + return err + } + } + } + return nil +} + +// absContextScalar handles services.*.build.context: skip URL-like values +// (https://, git://, ssh://, github.com/, git@, custom builder schemes), +// skip ServicePrefix entries, otherwise treat as a path. +func (r *nodeResolverState) absContextScalar(n *yaml.Node) error { + if n == nil || n.Kind != yaml.ScalarNode { + return nil + } + v := n.Value + if strings.Contains(v, "://") { + return nil + } + if strings.HasPrefix(v, types.ServicePrefix) { + return nil + } + if isRemoteContext(v) { + return nil + } + return r.absScalar(n) +} + +// absExtendsScalar resolves a services.*.extends.file scalar unless it +// matches a registered remote loader. +func (r *nodeResolverState) absExtendsScalar(n *yaml.Node) error { + if n == nil || n.Kind != yaml.ScalarNode { + return nil + } + if r.isRemoteResource(n.Value) { + return nil + } + return r.absScalar(n) +} + +// absSymbolicLinkScalar resolves a path then dereferences it through +// utils.ResolveSymbolicLink. Used by services.*.develop.watch.*.path. +func (r *nodeResolverState) absSymbolicLinkScalar(n *yaml.Node) error { + if err := r.absScalar(n); err != nil { + return err + } + if n == nil || n.Kind != yaml.ScalarNode { + return nil + } + resolved, err := utils.ResolveSymbolicLink(n.Value) + if err != nil { + return err + } + n.Value = resolved + return nil +} + +// absVolumeMount handles services.*.volumes.*: when the entry is the +// canonical long form (a mapping with type: bind), resolve the source +// path against the working directory of the scalar. Short-form string +// entries are left untouched and handled by EnforceUnicity later in the +// pipeline. +func (r *nodeResolverState) absVolumeMount(n *yaml.Node) error { + if n == nil || n.Kind != yaml.MappingNode { + return nil + } + mountType := mappingFieldValue(n, "type") + if mountType != types.VolumeTypeBind { + return nil + } + source := mappingFieldNode(n, "source") + if source == nil { + return errors.New(`invalid mount config for type "bind": field Source must not be empty`) + } + return r.maybeUnixScalar(source) +} + +// volumeDriverOpts handles volumes.*: when the local driver is in use with +// "o: bind", resolve the device path against the working directory. Mirrors +// the v2 relativePathsResolver.volumeDriverOpts. +func (r *nodeResolverState) volumeDriverOpts(n *yaml.Node) error { + if n == nil || n.Kind != yaml.MappingNode { + return nil + } + if mappingFieldValue(n, "driver") != "local" { + return nil + } + opts := mappingFieldNode(n, "driver_opts") + if opts == nil || opts.Kind != yaml.MappingNode { + return nil + } + if mappingFieldValue(opts, "o") != "bind" { + return nil + } + device := mappingFieldNode(opts, "device") + if device == nil { + return nil + } + return r.maybeUnixScalar(device) +} + +func mappingFieldNode(n *yaml.Node, key string) *yaml.Node { + if n == nil || n.Kind != yaml.MappingNode { + return nil + } + for i := 0; i+1 < len(n.Content); i += 2 { + if n.Content[i].Value == key { + return n.Content[i+1] + } + } + return nil +} + +func mappingFieldValue(n *yaml.Node, key string) string { + v := mappingFieldNode(n, key) + if v == nil || v.Kind != yaml.ScalarNode { + return "" + } + return v.Value +} diff --git a/paths/node_test.go b/paths/node_test.go new file mode 100644 index 000000000..5aa74bd3d --- /dev/null +++ b/paths/node_test.go @@ -0,0 +1,226 @@ +/* + Copyright 2020 The Compose Specification Authors. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +*/ + +package paths + +import ( + "path/filepath" + "strconv" + "strings" + "testing" + + "go.yaml.in/yaml/v4" + "gotest.tools/v3/assert" +) + +// testProjectDir is the absolute project root used by the in-package +// resolver tests. It is built from a single-letter component so the +// gocritic filepathJoin checker does not flag literal path separators in +// filepath.Join calls below. +var testProjectDir = filepath.Join(string(filepath.Separator), "project") + +func parse(t *testing.T, src string) *yaml.Node { + t.Helper() + var doc yaml.Node + assert.NilError(t, yaml.Unmarshal([]byte(src), &doc)) + return &doc +} + +func resolveYAML(t *testing.T, src string) map[string]any { + t.Helper() + root := parse(t, src) + assert.NilError(t, ResolveRelativePathsNode(root, NodeResolverOptions{WorkingDir: testProjectDir})) + var m map[string]any + assert.NilError(t, root.Decode(&m)) + return m +} + +func TestResolveRelativePathsNode_BuildContextResolved(t *testing.T) { + got := resolveYAML(t, ` +services: + web: + build: + context: ./app +`) + build := got["services"].(map[string]any)["web"].(map[string]any)["build"].(map[string]any) + assert.Equal(t, build["context"], filepath.Join(testProjectDir, "app")) +} + +func TestResolveRelativePathsNode_BuildContextUrlUntouched(t *testing.T) { + got := resolveYAML(t, ` +services: + web: + build: + context: https://github.com/example/repo.git +`) + build := got["services"].(map[string]any)["web"].(map[string]any)["build"].(map[string]any) + assert.Equal(t, build["context"], "https://github.com/example/repo.git") +} + +func TestResolveRelativePathsNode_EnvFileLongFormResolved(t *testing.T) { + got := resolveYAML(t, ` +services: + web: + env_file: + - path: ./.env + required: true +`) + env := got["services"].(map[string]any)["web"].(map[string]any)["env_file"].([]any) + first := env[0].(map[string]any) + assert.Equal(t, first["path"], filepath.Join(testProjectDir, ".env")) +} + +func TestResolveRelativePathsNode_VolumesLongFormBindResolved(t *testing.T) { + got := resolveYAML(t, ` +services: + web: + volumes: + - type: bind + source: ./data + target: /data +`) + vols := got["services"].(map[string]any)["web"].(map[string]any)["volumes"].([]any) + mount := vols[0].(map[string]any) + expected := filepath.Join(testProjectDir, "data") + assert.Equal(t, mount["source"], expected) +} + +func TestResolveRelativePathsNode_VolumesShortFormUnchanged(t *testing.T) { + got := resolveYAML(t, ` +services: + web: + volumes: + - "./data:/data" +`) + vols := got["services"].(map[string]any)["web"].(map[string]any)["volumes"].([]any) + // Short form: left untouched by the Node-level resolver. + assert.Equal(t, vols[0], "./data:/data") +} + +func TestResolveRelativePathsNode_AbsolutePathUnchanged(t *testing.T) { + // t.TempDir returns an absolute path appropriate for the host OS + // (`/var/...` on Unix, `C:\Users\...\Temp\...` on Windows), so the + // test exercises filepath.IsAbs on whichever platform CI runs on. + absoluteCtx := t.TempDir() + src := "services:\n web:\n build:\n context: " + strconv.Quote(absoluteCtx) + "\n" + got := resolveYAML(t, src) + build := got["services"].(map[string]any)["web"].(map[string]any)["build"].(map[string]any) + assert.Equal(t, build["context"], absoluteCtx) +} + +// TestResolveRelativePathsNode_PerScalarWorkingDir is the key v3 behavior: +// two relative paths in the same merged tree resolve against different +// working directories depending on the SourceContext attached to each. +func TestResolveRelativePathsNode_PerScalarWorkingDir(t *testing.T) { + rootDir := filepath.Join(string(filepath.Separator), "project-root") + includeDir := filepath.Join(string(filepath.Separator), "include-dir") + root := parse(t, ` +services: + web: + build: + context: ./from-root + api: + build: + context: ./from-include +`) + var webContext, apiContext *yaml.Node + var walk func(n *yaml.Node, parentKeys []string) + walk = func(n *yaml.Node, parentKeys []string) { + switch n.Kind { + case yaml.DocumentNode: + for _, c := range n.Content { + walk(c, parentKeys) + } + case yaml.MappingNode: + for i := 0; i+1 < len(n.Content); i += 2 { + walk(n.Content[i+1], append(parentKeys, n.Content[i].Value)) + } + case yaml.ScalarNode: + if len(parentKeys) >= 3 && parentKeys[len(parentKeys)-1] == "context" { + switch parentKeys[len(parentKeys)-3] { + case "web": + webContext = n + case "api": + apiContext = n + } + } + } + } + walk(root, nil) + assert.Assert(t, webContext != nil && apiContext != nil) + + err := ResolveRelativePathsNode(root, NodeResolverOptions{ + WorkingDirFor: func(n *yaml.Node) string { + if n == apiContext { + return includeDir + } + return rootDir + }, + }) + assert.NilError(t, err) + + var m map[string]any + assert.NilError(t, root.Decode(&m)) + assert.Equal(t, + m["services"].(map[string]any)["web"].(map[string]any)["build"].(map[string]any)["context"], + filepath.Join(rootDir, "from-root")) + assert.Equal(t, + m["services"].(map[string]any)["api"].(map[string]any)["build"].(map[string]any)["context"], + filepath.Join(includeDir, "from-include")) +} + +// TestResolveRelativePathsNode_IncludeNotResolvedHere documents that +// `include` paths are intentionally not resolved by ResolveRelativePathsNode +// (mirroring v2): include path resolution is part of collectIncludeLayers / +// ApplyInclude which knows about ResourceLoaders and project_directory +// redefinition. The patterns kept under "include.*" in the resolver map are +// inert (they never match the actual `include.[].path` walk path) but are +// preserved for v2 parity. +func TestResolveRelativePathsNode_IncludeNotResolvedHere(t *testing.T) { + got := resolveYAML(t, ` +include: + - path: + - ./a.yaml + - ./b.yaml + project_directory: ./sub +`) + incl := got["include"].([]any)[0].(map[string]any) + paths := incl["path"].([]any) + assert.Equal(t, paths[0], "./a.yaml") + assert.Equal(t, paths[1], "./b.yaml") + assert.Equal(t, incl["project_directory"], "./sub") +} + +func TestResolveRelativePathsNode_RemoteExtendsUntouched(t *testing.T) { + root := parse(t, ` +services: + web: + extends: + file: oci://registry/example:tag + service: base +`) + err := ResolveRelativePathsNode(root, NodeResolverOptions{ + WorkingDir: testProjectDir, + Remotes: []RemoteResource{ + func(p string) bool { return strings.HasPrefix(p, "oci://") }, + }, + }) + assert.NilError(t, err) + var m map[string]any + assert.NilError(t, root.Decode(&m)) + ext := m["services"].(map[string]any)["web"].(map[string]any)["extends"].(map[string]any) + assert.Equal(t, ext["file"], "oci://registry/example:tag") +} diff --git a/schema/schema.go b/schema/schema.go index a73eda245..443817991 100644 --- a/schema/schema.go +++ b/schema/schema.go @@ -79,17 +79,30 @@ func Validate(config map[string]interface{}) error { err = schema.Validate(raw) var verr *jsonschema.ValidationError if ok := errors.As(err, &verr); ok { - return validationError{getMostSpecificError(verr)} + return &Error{err: getMostSpecificError(verr)} } return err } -type validationError struct { +// Error wraps a jsonschema.ValidationError with helpers that surface +// the dotted compose path of the offending value, so the loader can +// look the corresponding source position up in its per-path snapshot +// and turn the failure into an errdefs.Diagnostic. +type Error struct { err *jsonschema.ValidationError } -func (e validationError) Error() string { - path := strings.Join(e.err.InstanceLocation, ".") +// Path returns the dotted compose path of the offending value (e.g. +// "services.web.ports.0"). +func (e *Error) Path() string { + if e == nil || e.err == nil { + return "" + } + return strings.Join(e.err.InstanceLocation, ".") +} + +func (e *Error) Error() string { + path := e.Path() p := message.NewPrinter(language.English) switch k := e.err.ErrorKind.(type) { case *kind.Type: diff --git a/transform/node.go b/transform/node.go new file mode 100644 index 000000000..11ab32a46 --- /dev/null +++ b/transform/node.go @@ -0,0 +1,124 @@ +/* + Copyright 2020 The Compose Specification Authors. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +*/ + +package transform + +import ( + "fmt" + + "go.yaml.in/yaml/v4" + + "github.com/compose-spec/compose-go/v3/tree" +) + +// CanonicalNode rewrites short-form syntax into canonical (long-form) syntax +// on a yaml.Node tree. +// +// Walker design: instead of decoding the whole tree into map[string]any and +// re-encoding (which zeroed Line / Column on every fresh node), recurse +// node by node and invoke the per-path transformer only on the matching +// subtree. The decode + encode round-trip is therefore scoped to the +// smallest subtree that needs reshaping, and every parent / sibling node +// keeps the original source position the YAML parser recorded. Downstream +// diagnostics (errdefs.Diagnostic) consume those positions through the +// origins side-table, so the smaller the subtree that loses Line / Column +// the better the user-facing error message. +// +// CanonicalNode mutates root in place and returns root for convenience. +func CanonicalNode(root *yaml.Node, ignoreParseError bool) (*yaml.Node, error) { + if root == nil { + return nil, nil + } + target := root + if target.Kind == yaml.DocumentNode && len(target.Content) == 1 { + target = target.Content[0] + } + if err := canonicalizeNode(target, tree.NewPath(), ignoreParseError); err != nil { + return nil, err + } + return root, nil +} + +// canonicalizeNode walks n in place, applying the matching transformer +// to the smallest subtree that matches a registered pattern. Nodes that +// no transformer claims are traversed structurally so their children +// can themselves match -- and untouched scalars keep their original +// Line / Column. +func canonicalizeNode(n *yaml.Node, p tree.Path, ignoreParseError bool) error { + if n == nil { + return nil + } + for pattern, transformer := range transformers { + if p.Matches(pattern) { + return applyTransformer(n, p, transformer, ignoreParseError) + } + } + switch n.Kind { + case yaml.MappingNode: + for i := 0; i+1 < len(n.Content); i += 2 { + if err := canonicalizeNode(n.Content[i+1], p.Next(n.Content[i].Value), ignoreParseError); err != nil { + return err + } + } + case yaml.SequenceNode: + for _, c := range n.Content { + if err := canonicalizeNode(c, p.Next(tree.PathMatchList), ignoreParseError); err != nil { + return err + } + } + } + return nil +} + +// applyTransformer runs the legacy map / slice based transformer on +// the scoped subtree at n by going through a minimal decode + encode +// round-trip. Only this subtree loses Line / Column on its fresh +// nodes; every ancestor and sibling keeps the original position. +func applyTransformer(n *yaml.Node, p tree.Path, transformer Func, ignoreParseError bool) error { + var raw any + if err := n.Decode(&raw); err != nil { + return fmt.Errorf("transform %s: decode for canonical: %w", p, err) + } + transformed, err := transformer(raw, p, ignoreParseError) + if err != nil { + return err + } + // Recurse into the transformed value so nested patterns still fire. + // Example: transformService at "services.*" rewrites the service + // shape; nested transformers like "services.*.ports" need to run + // next on the rewritten shape. + switch v := transformed.(type) { + case map[string]any: + if v, err = transformMapping(v, p, ignoreParseError); err != nil { + return err + } + transformed = v + case []any: + if v, err = transformSequence(v, p, ignoreParseError); err != nil { + return err + } + transformed = v + } + var rebuilt yaml.Node + if err := rebuilt.Encode(transformed); err != nil { + return fmt.Errorf("transform %s: re-encode after canonical: %w", p, err) + } + // Replace n's content with the rebuilt subtree while keeping the + // outer node pointer intact so callers that walked into this node + // still observe the canonical shape. + *n = rebuilt + return nil +} diff --git a/transform/node_test.go b/transform/node_test.go new file mode 100644 index 000000000..5ce97d9e5 --- /dev/null +++ b/transform/node_test.go @@ -0,0 +1,123 @@ +/* + Copyright 2020 The Compose Specification Authors. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +*/ + +package transform + +import ( + "testing" + + "go.yaml.in/yaml/v4" + "gotest.tools/v3/assert" +) + +func parseNode(t *testing.T, src string) *yaml.Node { + t.Helper() + var doc yaml.Node + assert.NilError(t, yaml.Unmarshal([]byte(src), &doc)) + return &doc +} + +func canonicalize(t *testing.T, src string) map[string]any { + t.Helper() + root := parseNode(t, src) + out, err := CanonicalNode(root, false) + assert.NilError(t, err) + var m map[string]any + assert.NilError(t, out.Decode(&m)) + return m +} + +func TestCanonicalNode_PortsShortFormExpanded(t *testing.T) { + got := canonicalize(t, ` +services: + web: + ports: + - "8080:80" +`) + ports := got["services"].(map[string]any)["web"].(map[string]any)["ports"].([]any) + assert.Equal(t, len(ports), 1) + long := ports[0].(map[string]any) + assert.Equal(t, long["target"], 80) + assert.Equal(t, long["published"], "8080") +} + +func TestCanonicalNode_BuildShortFormExpanded(t *testing.T) { + got := canonicalize(t, ` +services: + web: + build: ./app +`) + build := got["services"].(map[string]any)["web"].(map[string]any)["build"].(map[string]any) + assert.Equal(t, build["context"], "./app") +} + +func TestCanonicalNode_DependsOnListExpanded(t *testing.T) { + got := canonicalize(t, ` +services: + web: + depends_on: + - db + - cache +`) + deps := got["services"].(map[string]any)["web"].(map[string]any)["depends_on"].(map[string]any) + db := deps["db"].(map[string]any) + assert.Equal(t, db["condition"], "service_started") +} + +func TestCanonicalNode_NetworksListExpanded(t *testing.T) { + got := canonicalize(t, ` +services: + web: + networks: + - frontend + - backend +`) + nets := got["services"].(map[string]any)["web"].(map[string]any)["networks"].(map[string]any) + _, ok := nets["frontend"] + assert.Assert(t, ok) + _, ok = nets["backend"] + assert.Assert(t, ok) +} + +func TestCanonicalNode_EnvFileShortFormExpanded(t *testing.T) { + got := canonicalize(t, ` +services: + web: + env_file: .env +`) + env := got["services"].(map[string]any)["web"].(map[string]any)["env_file"].([]any) + first := env[0].(map[string]any) + assert.Equal(t, first["path"], ".env") +} + +func TestCanonicalNode_DocumentNodeUnwrapped(t *testing.T) { + // Passing a DocumentNode root must not panic and must return a tree + // whose decoded shape matches the expected canonical form. + root := parseNode(t, "services:\n web:\n build: .\n") + assert.Equal(t, root.Kind, yaml.DocumentNode) + out, err := CanonicalNode(root, false) + assert.NilError(t, err) + var m map[string]any + assert.NilError(t, out.Decode(&m)) + build := m["services"].(map[string]any)["web"].(map[string]any)["build"].(map[string]any) + assert.Equal(t, build["context"], ".") +} + +func TestCanonicalNode_NilSafelyHandled(t *testing.T) { + out, err := CanonicalNode(nil, false) + assert.NilError(t, err) + assert.Assert(t, out == nil) +} diff --git a/transform/ports.go b/transform/ports.go index ab249d998..285bb33bb 100644 --- a/transform/ports.go +++ b/transform/ports.go @@ -21,7 +21,7 @@ import ( "github.com/compose-spec/compose-go/v3/tree" "github.com/compose-spec/compose-go/v3/types" - "github.com/go-viper/mapstructure/v2" + "go.yaml.in/yaml/v4" ) func transformPorts(data any, p tree.Path, ignoreParseError bool) (any, error) { @@ -75,17 +75,21 @@ func transformPorts(data any, p tree.Path, ignoreParseError bool) (any, error) { } } +// encode marshals a typed value (ServicePortConfig, parsed volume mount, +// ...) into the canonical map[string]any layout the schema validator and +// downstream transformers expect. The yaml.Encode/Decode round-trip +// honors the yaml struct tags (snake_case, omitempty) the same way the +// previous mapstructure-based encoder did, without the runtime dependency. func encode(v any) (map[string]any, error) { + var node yaml.Node + if err := node.Encode(v); err != nil { + return nil, err + } m := map[string]any{} - decoder, err := mapstructure.NewDecoder(&mapstructure.DecoderConfig{ - Result: &m, - TagName: "yaml", - }) - if err != nil { + if err := node.Decode(&m); err != nil { return nil, err } - err = decoder.Decode(v) - return m, err + return m, nil } func portDefaults(data any, _ tree.Path, _ bool) (any, error) { diff --git a/transform/ports_test.go b/transform/ports_test.go index a6f629a06..7f8b333d6 100644 --- a/transform/ports_test.go +++ b/transform/ports_test.go @@ -44,14 +44,14 @@ func Test_transformPorts(t *testing.T) { "mode": "ingress", "protocol": "tcp", "published": "8080", - "target": uint32(80), + "target": 80, }, map[string]any{ "host_ip": "127.0.0.1", "mode": "ingress", "protocol": "tcp", "published": "8081", - "target": uint32(81), + "target": 81, }, }, }, diff --git a/transform/volume.go b/transform/volume.go index a413d634a..48bc9411a 100644 --- a/transform/volume.go +++ b/transform/volume.go @@ -37,6 +37,7 @@ func transformVolumeMount(data any, p tree.Path, ignoreParseError bool) (any, er return nil, err } volume.Target = cleanTarget(volume.Target) + volume.Source = cleanSource(volume.Source) return encode(volume) default: @@ -51,6 +52,22 @@ func cleanTarget(target string) string { return path.Clean(target) } +// cleanSource normalizes the only short-form source spelling that v2 +// produces a different decoded form for vs v3: "./" collapses to "." +// in v2 because filepath.Join in the sub-resolve cleans it, while v3's +// node-level resolver preserves the literal "./" until the format +// parser sees it. Mirror v2 here by rewriting "./" -> "." in place, +// keeping the `.` prefix that format.ParseVolume relies on to flag the +// value as a bind path. Other short-form spellings (`./foo`, `/abs`, +// `name`) are left alone so format.ParseVolume / Windows path +// conversion observe the original characters. +func cleanSource(source string) string { + if source == "./" { + return "." + } + return source +} + func defaultVolumeBind(data any, p tree.Path, _ bool) (any, error) { bind, ok := data.(map[string]any) if !ok { diff --git a/types/bytes.go b/types/bytes.go index 0f039ab76..2ef467597 100644 --- a/types/bytes.go +++ b/types/bytes.go @@ -78,13 +78,3 @@ func (u *UnitBytes) UnmarshalYAML(value *yaml.Node) error { } return u.parseString(s) } - -func (u *UnitBytes) DecodeMapstructure(value interface{}) error { - switch v := value.(type) { - case int: - *u = UnitBytes(v) - case string: - return u.parseString(v) - } - return nil -} diff --git a/types/command.go b/types/command.go index 559dc3050..a133eb6a9 100644 --- a/types/command.go +++ b/types/command.go @@ -16,7 +16,12 @@ package types -import "github.com/mattn/go-shellwords" +import ( + "fmt" + + "github.com/mattn/go-shellwords" + "go.yaml.in/yaml/v4" +) // ShellCommand is a string or list of string args. // @@ -67,20 +72,26 @@ func (s ShellCommand) MarshalYAML() (interface{}, error) { return []string(s), nil } -func (s *ShellCommand) DecodeMapstructure(value interface{}) error { - switch v := value.(type) { - case string: - cmd, err := shellwords.Parse(v) +// UnmarshalYAML accepts either a shell command string (parsed with shellwords) +// or a sequence of arguments and stores the resulting argv in s. Mirrors +// DecodeMapstructure for yaml.v4 native decoding. +func (s *ShellCommand) UnmarshalYAML(value *yaml.Node) error { + value = unwrapDocument(value) + switch value.Kind { + case yaml.ScalarNode: + cmd, err := shellwords.Parse(value.Value) if err != nil { return err } *s = cmd - case []interface{}: - cmd := make([]string, len(v)) - for i, s := range v { - cmd[i] = s.(string) + case yaml.SequenceNode: + var cmd []string + if err := value.Decode(&cmd); err != nil { + return err } *s = cmd + default: + return fmt.Errorf("invalid yaml kind %d for shell command", value.Kind) } return nil } diff --git a/types/config.go b/types/config.go index 9a0fdaf27..9b3c966ff 100644 --- a/types/config.go +++ b/types/config.go @@ -21,7 +21,7 @@ import ( "runtime" "strings" - "github.com/go-viper/mapstructure/v2" + "go.yaml.in/yaml/v4" ) // isCaseInsensitiveEnvVars is true on platforms where environment variable names are treated case-insensitively. @@ -63,6 +63,12 @@ type ConfigFile struct { Content []byte // Config if the yaml tree for this config file. Will be parsed from Content if not set Config map[string]interface{} + // Node is a pre-parsed yaml.Node for this config file. When non-nil, v3 + // loader paths consume it directly and skip both Content and Filename. + // Allows callers that already produced a Node (e.g. through a custom + // reader, a remote loader or a previous transformation) to feed it into + // the loader without re-parsing. + Node *yaml.Node } func (cf ConfigFile) IsStdin() bool { @@ -136,10 +142,19 @@ func (c Config) MarshalJSON() ([]byte, error) { return json.Marshal(m) } +// Get decodes the named extension value into target. The extension may +// have been stored raw (map[string]any / []any from a yaml decode) or +// already projected into a typed struct; both shapes round-trip through +// yaml so the caller receives target populated from the source's yaml +// tag layout. func (e Extensions) Get(name string, target interface{}) (bool, error) { - if v, ok := e[name]; ok { - err := mapstructure.Decode(v, target) + v, ok := e[name] + if !ok { + return false, nil + } + buf, err := yaml.Marshal(v) + if err != nil { return true, err } - return false, nil + return true, yaml.Unmarshal(buf, target) } diff --git a/types/cpus.go b/types/cpus.go index f32c6e621..f435d48c0 100644 --- a/types/cpus.go +++ b/types/cpus.go @@ -19,30 +19,28 @@ package types import ( "fmt" "strconv" + + "go.yaml.in/yaml/v4" ) type NanoCPUs float32 -func (n *NanoCPUs) DecodeMapstructure(a any) error { - switch v := a.(type) { - case string: - f, err := strconv.ParseFloat(v, 64) - if err != nil { - return err - } - *n = NanoCPUs(f) - case int: - *n = NanoCPUs(v) - case float32: - *n = NanoCPUs(v) - case float64: - *n = NanoCPUs(v) - default: - return fmt.Errorf("unexpected value type %T for cpus", v) - } - return nil -} - func (n *NanoCPUs) Value() float32 { return float32(*n) } + +// UnmarshalYAML accepts a scalar number or numeric string and stores its +// float32 value in n. Mirrors DecodeMapstructure for yaml.v4 native +// decoding. +func (n *NanoCPUs) UnmarshalYAML(value *yaml.Node) error { + value = unwrapDocument(value) + if value.Kind != yaml.ScalarNode { + return fmt.Errorf("expected scalar nanocpus, got kind %d", value.Kind) + } + f, err := strconv.ParseFloat(value.Value, 64) + if err != nil { + return fmt.Errorf("invalid cpus value %q: %w", value.Value, err) + } + *n = NanoCPUs(f) + return nil +} diff --git a/types/device.go b/types/device.go index 5b30cc0ca..e82bdbada 100644 --- a/types/device.go +++ b/types/device.go @@ -20,6 +20,8 @@ import ( "fmt" "strconv" "strings" + + "go.yaml.in/yaml/v4" ) type DeviceRequest struct { @@ -32,22 +34,21 @@ type DeviceRequest struct { type DeviceCount int64 -func (c *DeviceCount) DecodeMapstructure(value interface{}) error { - switch v := value.(type) { - case int: - *c = DeviceCount(v) - case string: - if strings.ToLower(v) == "all" { - *c = -1 - return nil - } - i, err := strconv.ParseInt(v, 10, 64) - if err != nil { - return fmt.Errorf("invalid value %q, the only value allowed is 'all' or a number", v) - } - *c = DeviceCount(i) - default: - return fmt.Errorf("invalid type %T for device count", v) +// UnmarshalYAML accepts a scalar integer or the literal "all" string and +// stores its int64 value in c. "all" maps to -1, matching v2 semantics. +func (c *DeviceCount) UnmarshalYAML(value *yaml.Node) error { + value = unwrapDocument(value) + if value.Kind != yaml.ScalarNode { + return fmt.Errorf("expected scalar device count, got kind %d", value.Kind) + } + if strings.ToLower(value.Value) == "all" { + *c = -1 + return nil + } + i, err := strconv.ParseInt(value.Value, 10, 64) + if err != nil { + return fmt.Errorf("invalid value %q, the only value allowed is 'all' or a number", value.Value) } + *c = DeviceCount(i) return nil } diff --git a/types/diagnostics.go b/types/diagnostics.go new file mode 100644 index 000000000..45ebf032f --- /dev/null +++ b/types/diagnostics.go @@ -0,0 +1,40 @@ +/* + Copyright 2020 The Compose Specification Authors. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +*/ + +package types + +// Location records the source position of a compose value: the absolute +// path of the file that declared it (or "(inline)" when the document was +// built from in-memory bytes) plus the 1-based line and column emitted +// by the YAML parser. Zero on Line or Column means "not recorded". +type Location struct { + File string `yaml:"file,omitempty" json:"file,omitempty"` + Line int `yaml:"line,omitempty" json:"line,omitempty"` + Column int `yaml:"column,omitempty" json:"column,omitempty"` +} + +// Sources maps a dotted compose path (e.g. "services.web.image") to the +// source Location of the corresponding value. It is populated on +// *Project.Sources when the loader was invoked with the Diagnostics +// opt-in. +// +// The map covers every mapping path reachable from the merged tree at +// the time of normalization. Paths under sequences are stable per index +// (e.g. "services.web.ports.0") only when those entries survived the +// canonical transform without re-encoding; downstream consumers should +// treat missing entries as "position not recorded" rather than as an +// error. +type Sources map[string]Location diff --git a/types/duration.go b/types/duration.go index c1c39730d..8a2ef0349 100644 --- a/types/duration.go +++ b/types/duration.go @@ -23,6 +23,7 @@ import ( "time" "github.com/xhit/go-str2duration/v2" + "go.yaml.in/yaml/v4" ) // Duration is a thin wrapper around time.Duration with improved JSON marshalling @@ -32,15 +33,6 @@ func (d Duration) String() string { return time.Duration(d).String() } -func (d *Duration) DecodeMapstructure(value interface{}) error { - v, err := str2duration.ParseDuration(fmt.Sprint(value)) - if err != nil { - return err - } - *d = Duration(v) - return nil -} - // MarshalJSON makes Duration implement json.Marshaler func (d Duration) MarshalJSON() ([]byte, error) { return json.Marshal(d.String()) @@ -60,3 +52,19 @@ func (d *Duration) UnmarshalJSON(b []byte) error { *d = Duration(timeDuration) return nil } + +// UnmarshalYAML accepts a scalar string in str2duration format (e.g. "10s", +// "1h30m") and stores the parsed value in d. Mirrors DecodeMapstructure for +// yaml.v4 native decoding. +func (d *Duration) UnmarshalYAML(value *yaml.Node) error { + value = unwrapDocument(value) + if value.Kind != yaml.ScalarNode { + return fmt.Errorf("expected scalar duration, got kind %d", value.Kind) + } + v, err := str2duration.ParseDuration(value.Value) + if err != nil { + return err + } + *d = Duration(v) + return nil +} diff --git a/types/healthcheck.go b/types/healthcheck.go index c6c3b37e0..1d2aa843d 100644 --- a/types/healthcheck.go +++ b/types/healthcheck.go @@ -18,6 +18,8 @@ package types import ( "fmt" + + "go.yaml.in/yaml/v4" ) // HealthCheckConfig the healthcheck configuration for a service @@ -36,18 +38,22 @@ type HealthCheckConfig struct { // HealthCheckTest is the command run to test the health of a service type HealthCheckTest []string -func (l *HealthCheckTest) DecodeMapstructure(value interface{}) error { - switch v := value.(type) { - case string: - *l = []string{"CMD-SHELL", v} - case []interface{}: - seq := make([]string, len(v)) - for i, e := range v { - seq[i] = e.(string) +// UnmarshalYAML accepts either a CMD-SHELL string (shorthand: prefixed with +// "CMD-SHELL" at runtime) or a sequence of explicit argv entries. Mirrors +// DecodeMapstructure for yaml.v4 native decoding. +func (l *HealthCheckTest) UnmarshalYAML(value *yaml.Node) error { + value = unwrapDocument(value) + switch value.Kind { + case yaml.ScalarNode: + *l = []string{"CMD-SHELL", value.Value} + case yaml.SequenceNode: + var seq []string + if err := value.Decode(&seq); err != nil { + return err } *l = seq default: - return fmt.Errorf("unexpected value type %T for healthcheck.test", value) + return fmt.Errorf("unexpected yaml kind %d for healthcheck.test", value.Kind) } return nil } diff --git a/types/hostList.go b/types/hostList.go index 9bc0fbc5d..c04ea793b 100644 --- a/types/hostList.go +++ b/types/hostList.go @@ -21,6 +21,8 @@ import ( "fmt" "sort" "strings" + + "go.yaml.in/yaml/v4" ) // HostsList is a list of colon-separated host-ip mappings @@ -81,47 +83,55 @@ func (h HostsList) MarshalJSON() ([]byte, error) { var hostListSerapators = []string{"=", ":"} -func (h *HostsList) DecodeMapstructure(value interface{}) error { - switch v := value.(type) { - case map[string]interface{}: - list := make(HostsList, len(v)) - for i, e := range v { - if e == nil { - e = "" - } - switch t := e.(type) { - case string: - list[i] = []string{t} - case []any: - hosts := make([]string, len(t)) - for j, h := range t { - hosts[j] = fmt.Sprint(h) +// UnmarshalYAML accepts either a mapping form (each value can be a scalar +// hostname or a sequence of hostnames) or a list of "host=ip" / "host:ip" +// short-form entries. Mirrors DecodeMapstructure for yaml.v4 native +// decoding. +func (h *HostsList) UnmarshalYAML(value *yaml.Node) error { + value = unwrapDocument(value) + switch value.Kind { + case yaml.MappingNode: + list := make(HostsList, len(value.Content)/2) + for i := 0; i+1 < len(value.Content); i += 2 { + key := value.Content[i].Value + val := value.Content[i+1] + switch val.Kind { + case yaml.ScalarNode: + list[key] = []string{val.Value} + case yaml.SequenceNode: + hosts := make([]string, 0, len(val.Content)) + for _, item := range val.Content { + if item.Kind != yaml.ScalarNode { + return fmt.Errorf("extra_hosts entry must be scalar") + } + hosts = append(hosts, item.Value) } - list[i] = hosts + list[key] = hosts default: - return fmt.Errorf("unexpected value type %T for extra_hosts entry", value) + return fmt.Errorf("unexpected yaml kind %d for extra_hosts entry", val.Kind) } } - err := list.cleanup() - if err != nil { + if err := list.cleanup(); err != nil { return err } *h = list - return nil - case []interface{}: - s := make([]string, len(v)) - for i, e := range v { - s[i] = fmt.Sprint(e) + case yaml.SequenceNode: + strs := make([]string, 0, len(value.Content)) + for _, item := range value.Content { + if item.Kind != yaml.ScalarNode { + return fmt.Errorf("extra_hosts list entry must be scalar") + } + strs = append(strs, item.Value) } - list, err := NewHostsList(s) + list, err := NewHostsList(strs) if err != nil { return err } *h = list - return nil default: - return fmt.Errorf("unexpected value type %T for extra_hosts", value) + return fmt.Errorf("unexpected yaml kind %d for extra_hosts", value.Kind) } + return nil } func (h HostsList) cleanup() error { diff --git a/types/labels.go b/types/labels.go index 7ea5edc41..04126d30e 100644 --- a/types/labels.go +++ b/types/labels.go @@ -19,6 +19,8 @@ package types import ( "fmt" "strings" + + "go.yaml.in/yaml/v4" ) // Labels is a mapping type for labels @@ -60,36 +62,33 @@ func (l Labels) ToMappingWithEquals() MappingWithEquals { return mapping } -// label value can be a string | number | boolean | null (empty) -func labelValue(e interface{}) string { - if e == nil { - return "" - } - switch v := e.(type) { - case string: - return v - default: - return fmt.Sprint(v) - } -} - -func (l *Labels) DecodeMapstructure(value interface{}) error { - switch v := value.(type) { - case map[string]interface{}: - labels := make(map[string]string, len(v)) - for k, e := range v { - labels[k] = labelValue(e) +// UnmarshalYAML accepts a mapping (key -> value) or a list of "key=value" +// entries and stores the result as a Labels map. Mirrors DecodeMapstructure +// for yaml.v4 native decoding. Numeric and boolean scalar values in the +// mapping form are coerced to their stringified representation via the +// underlying scalar Value (yaml.v4 preserves the source representation in +// Node.Value regardless of Tag). +func (l *Labels) UnmarshalYAML(value *yaml.Node) error { + value = unwrapDocument(value) + switch value.Kind { + case yaml.MappingNode: + labels := make(Labels, len(value.Content)/2) + for i := 0; i+1 < len(value.Content); i += 2 { + labels[value.Content[i].Value] = scalarToString(value.Content[i+1]) } *l = labels - case []interface{}: - labels := make(map[string]string, len(v)) - for _, s := range v { - k, e, _ := strings.Cut(fmt.Sprint(s), "=") - labels[k] = labelValue(e) + case yaml.SequenceNode: + labels := make(Labels, len(value.Content)) + for _, item := range value.Content { + if item.Kind != yaml.ScalarNode { + return fmt.Errorf("labels list entry must be scalar, got kind %d", item.Kind) + } + k, v, _ := strings.Cut(item.Value, "=") + labels[k] = v } *l = labels default: - return fmt.Errorf("unexpected value type %T for labels", value) + return fmt.Errorf("unexpected yaml kind %d for labels", value.Kind) } return nil } diff --git a/types/labels_test.go b/types/labels_test.go index 9a4bf5c52..22b973702 100644 --- a/types/labels_test.go +++ b/types/labels_test.go @@ -19,16 +19,17 @@ package types import ( "testing" + "go.yaml.in/yaml/v4" "gotest.tools/v3/assert" ) func TestDecodeLabel(t *testing.T) { l := Labels{} - err := l.DecodeMapstructure([]any{ - "a=b", - "c", - }) - assert.NilError(t, err) + src := ` +- a=b +- c +` + assert.NilError(t, yaml.Unmarshal([]byte(src), &l)) assert.Equal(t, l["a"], "b") assert.Equal(t, l["c"], "") } diff --git a/types/mapping.go b/types/mapping.go index fb14974f9..9080f9cad 100644 --- a/types/mapping.go +++ b/types/mapping.go @@ -21,6 +21,8 @@ import ( "sort" "strings" "unicode" + + "go.yaml.in/yaml/v4" ) // MappingWithEquals is a mapping type that can be converted from a list of @@ -83,48 +85,43 @@ func (m MappingWithEquals) ToMapping() Mapping { return o } -func (m *MappingWithEquals) DecodeMapstructure(value interface{}) error { - switch v := value.(type) { - case map[string]interface{}: - mapping := make(MappingWithEquals, len(v)) - for k, e := range v { - mapping[k] = mappingValue(e) +// UnmarshalYAML accepts a mapping form or a list of `key[=value]` entries +// and stores the result as a MappingWithEquals. The pointer distinction +// between a bare `key` (nil) and `key=` (pointer to "") is preserved: that +// distinction drives environment variable resolution downstream. +func (m *MappingWithEquals) UnmarshalYAML(value *yaml.Node) error { + value = unwrapDocument(value) + switch value.Kind { + case yaml.MappingNode: + mapping := make(MappingWithEquals, len(value.Content)/2) + for i := 0; i+1 < len(value.Content); i += 2 { + mapping[value.Content[i].Value] = scalarToStringPtr(value.Content[i+1]) } *m = mapping - case []interface{}: - mapping := make(MappingWithEquals, len(v)) - for _, s := range v { - k, e, ok := strings.Cut(fmt.Sprint(s), "=") + case yaml.SequenceNode: + mapping := make(MappingWithEquals, len(value.Content)) + for _, item := range value.Content { + if item.Kind != yaml.ScalarNode { + return fmt.Errorf("mapping list entry must be scalar, got kind %d", item.Kind) + } + k, e, ok := strings.Cut(item.Value, "=") if k != "" && unicode.IsSpace(rune(k[len(k)-1])) { return fmt.Errorf("environment variable %s is declared with a trailing space", k) } if !ok { mapping[k] = nil } else { - mapping[k] = mappingValue(e) + v := e + mapping[k] = &v } } *m = mapping default: - return fmt.Errorf("unexpected value type %T for mapping", value) + return fmt.Errorf("unexpected yaml kind %d for mapping", value.Kind) } return nil } -// label value can be a string | number | boolean | null -func mappingValue(e interface{}) *string { - if e == nil { - return nil - } - switch v := e.(type) { - case string: - return &v - default: - s := fmt.Sprint(v) - return &s - } -} - // Mapping is a mapping type that can be converted from a list of // key[=value] strings. // For the key with an empty value (`key=`), or key without value (`key`), the @@ -189,42 +186,30 @@ func (m Mapping) Merge(o Mapping) Mapping { return m } -func (m *Mapping) DecodeMapstructure(value interface{}) error { - switch v := value.(type) { - case map[string]interface{}: - mapping := make(Mapping, len(v)) - for k, e := range v { - if e == nil { - e = "" +// UnmarshalYAML accepts a mapping form or a list of "key=value" entries and +// stores the result as a Mapping. A bare `key` in list form maps to an +// empty string, matching the v2 behavior. +func (m *Mapping) UnmarshalYAML(value *yaml.Node) error { + value = unwrapDocument(value) + switch value.Kind { + case yaml.MappingNode: + mapping := make(Mapping, len(value.Content)/2) + for i := 0; i+1 < len(value.Content); i += 2 { + mapping[value.Content[i].Value] = scalarToString(value.Content[i+1]) + } + *m = mapping + case yaml.SequenceNode: + mapping := make(Mapping, len(value.Content)) + for _, item := range value.Content { + if item.Kind != yaml.ScalarNode { + return fmt.Errorf("mapping list entry must be scalar, got kind %d", item.Kind) } - mapping[k] = fmt.Sprint(e) + k, v, _ := strings.Cut(item.Value, "=") + mapping[k] = v } *m = mapping - case []interface{}: - *m = decodeMapping(v, "=") default: - return fmt.Errorf("unexpected value type %T for mapping", value) + return fmt.Errorf("unexpected yaml kind %d for mapping", value.Kind) } return nil } - -// Generate a mapping by splitting strings at any of seps, which will be tried -// in-order for each input string. (For example, to allow the preferred 'host=ip' -// in 'extra_hosts', as well as 'host:ip' for backwards compatibility.) -func decodeMapping(v []interface{}, seps ...string) map[string]string { - mapping := make(Mapping, len(v)) - for _, s := range v { - for i, sep := range seps { - k, e, ok := strings.Cut(fmt.Sprint(s), sep) - if ok { - // Mapping found with this separator, stop here. - mapping[k] = e - break - } else if i == len(seps)-1 { - // No more separators to try, map to empty string. - mapping[k] = "" - } - } - } - return mapping -} diff --git a/types/options.go b/types/options.go index 9aadb89ca..55d15c4c0 100644 --- a/types/options.go +++ b/types/options.go @@ -16,51 +16,68 @@ package types -import "fmt" +import ( + "fmt" + + "go.yaml.in/yaml/v4" +) // Options is a mapping type for options we pass as-is to container runtime type Options map[string]string -func (d *Options) DecodeMapstructure(value interface{}) error { - switch v := value.(type) { - case map[string]interface{}: - m := make(map[string]string) - for key, e := range v { - if e == nil { - m[key] = "" - } else { - m[key] = fmt.Sprint(e) - } +// MultiOptions allow option to be repeated +type MultiOptions map[string][]string + +// UnmarshalYAML accepts a mapping of single-valued string options and +// stores it in d. A non-scalar value (sequence or mapping) is rejected +// rather than silently collapsed to the empty string. +func (d *Options) UnmarshalYAML(value *yaml.Node) error { + value = unwrapDocument(value) + if value.Kind != yaml.MappingNode { + return fmt.Errorf("expected mapping for options, got kind %d", value.Kind) + } + m := make(Options, len(value.Content)/2) + for i := 0; i+1 < len(value.Content); i += 2 { + key := value.Content[i].Value + val := value.Content[i+1] + if val.Kind != yaml.ScalarNode { + return fmt.Errorf("option %s: expected scalar, got kind %d", key, val.Kind) } - *d = m - case map[string]string: - *d = v - default: - return fmt.Errorf("invalid type %T for options", value) + m[key] = scalarToString(val) } + *d = m return nil } -// MultiOptions allow option to be repeated -type MultiOptions map[string][]string - -func (d *MultiOptions) DecodeMapstructure(value interface{}) error { - switch v := value.(type) { - case map[string]interface{}: - m := make(map[string][]string) - for key, e := range v { - switch e := e.(type) { - case []interface{}: - for _, v := range e { - m[key] = append(m[key], fmt.Sprint(v)) +// UnmarshalYAML accepts a mapping where each value is either a scalar or +// a sequence of scalars, and stores the result in d as a slice per key. +// Non-scalar entries inside a sequence are rejected so a typo like +// `key: [[a]]` fails fast instead of decoding as an empty string. +func (d *MultiOptions) UnmarshalYAML(value *yaml.Node) error { + value = unwrapDocument(value) + if value.Kind != yaml.MappingNode { + return fmt.Errorf("expected mapping for options, got kind %d", value.Kind) + } + m := make(MultiOptions, len(value.Content)/2) + for i := 0; i+1 < len(value.Content); i += 2 { + key := value.Content[i].Value + val := value.Content[i+1] + switch val.Kind { + case yaml.ScalarNode: + m[key] = []string{scalarToString(val)} + case yaml.SequenceNode: + values := make([]string, 0, len(val.Content)) + for _, item := range val.Content { + if item.Kind != yaml.ScalarNode { + return fmt.Errorf("option %s: sequence entry must be scalar, got kind %d", key, item.Kind) } - default: - m[key] = append(m[key], fmt.Sprint(e)) + values = append(values, scalarToString(item)) } + m[key] = values + default: + return fmt.Errorf("option %s: expected scalar or sequence, got kind %d", key, val.Kind) } - *d = m - default: - return fmt.Errorf("invalid type %T for options", value) } + *d = m return nil } diff --git a/types/project.go b/types/project.go index 9952b2005..c361d18fb 100644 --- a/types/project.go +++ b/types/project.go @@ -56,6 +56,36 @@ type Project struct { // DisabledServices track services which have been disable as profile is not active DisabledServices Services `yaml:"-" json:"-"` Profiles []string `yaml:"-" json:"-"` + + // EnvFileScopes captures, per env_file path, the layer Environment in + // effect when the entry was declared. v3 lazy interpolation uses it as + // the preferred lookup scope when WithServicesEnvironmentResolved + // reads the env_file content, so a file referenced from an include + // block resolves variables in the include env_file values rather than + // only the project-wide environment. Not serialized. + EnvFileScopes map[string]Mapping `yaml:"-" json:"-"` + + // Sources maps a dotted compose path to the source Location of the + // corresponding value. Populated by the loader when invoked with + // loader.WithDiagnostics(); nil otherwise. Not serialized so the + // project shape is unchanged for callers that did not opt in. + Sources Sources `yaml:"-" json:"-"` +} + +// SetEnvFileScope records the environment that was effective when path was +// declared as an env_file entry. WithServicesEnvironmentResolved consults +// this map first when interpolating env_file content. +func (p *Project) SetEnvFileScope(path string, env Mapping) { + if p.EnvFileScopes == nil { + p.EnvFileScopes = map[string]Mapping{} + } + p.EnvFileScopes[path] = env +} + +// EnvFileScope returns the environment recorded for the env_file at path, +// or nil when none was recorded (the project-wide environment is then used). +func (p *Project) EnvFileScope(path string) Mapping { + return p.EnvFileScopes[path] } // ServiceNames return names for all services in this Compose config @@ -680,7 +710,18 @@ func (p Project) WithServicesEnvironmentResolved(discardEnvFiles bool) (*Project environment := service.Environment.ToMapping() for _, envFile := range service.EnvFiles { + scopedEnv := p.EnvFileScopes[envFile.Path] err := loadEnvFile(envFile, environment, func(k string) (string, bool) { + // v3 lazy interpolation: when the env_file entry was + // captured with its declaring layer environment, prefer + // that scope so a file referenced from an include block + // resolves against the include env_file values rather + // than only the project-wide environment. + if scopedEnv != nil { + if v, ok := scopedEnv.Resolve(k); ok { + return v, true + } + } // project.env has precedence doing interpolation if resolve, ok := p.Environment.Resolve(k); ok { return resolve, true @@ -782,6 +823,24 @@ func (p *Project) deepCopy() *Project { } n := &Project{} deriveDeepCopyProject(n, p) + // EnvFileScopes and Sources are not handled by the generated + // deriveDeepCopyProject. Carry them over so chained WithProfiles / + // WithServicesEnvironmentResolved / ... calls keep the v3 + // per-env_file declaring-layer environment metadata and the + // per-path source location snapshot (only present when the loader + // was invoked with WithDiagnostics). + if len(p.EnvFileScopes) > 0 { + n.EnvFileScopes = make(map[string]Mapping, len(p.EnvFileScopes)) + for k, v := range p.EnvFileScopes { + n.EnvFileScopes[k] = v + } + } + if len(p.Sources) > 0 { + n.Sources = make(Sources, len(p.Sources)) + for k, v := range p.Sources { + n.Sources[k] = v + } + } return n } diff --git a/types/reflect_test.go b/types/reflect_test.go new file mode 100644 index 000000000..3dd430bad --- /dev/null +++ b/types/reflect_test.go @@ -0,0 +1,133 @@ +/* + Copyright 2020 The Compose Specification Authors. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +*/ + +package types + +import ( + "reflect" + "strings" + "testing" + + "go.yaml.in/yaml/v4" +) + +// TestExportedStructFieldsHaveYAMLTagOrCustomDecode walks every exported +// struct type in this package and asserts that each exported field +// either declares a yaml tag or that the type carries an +// UnmarshalYAML method. yaml.v4 is strict about field discovery (it +// matches lowercased field names but the codebase has been bitten by +// snake_case keys that map to PascalCase field names, e.g. +// WeightDevice.Path / Weight before the v3 tags landed), so the test +// blocks future contributors from adding an untagged exported field +// to a struct that compose decodes from YAML. +// +// The reflect walk picks up every type reachable from a Project value +// (the top-level entry point) plus the standalone Compose configuration +// types it references but does not embed (IncludeConfig, ConfigFile, +// ConfigDetails, ExtendsConfig). Adding a new exported struct to the +// package therefore requires either explicit yaml tags or an +// UnmarshalYAML implementation -- both ways are accepted. +func TestExportedStructFieldsHaveYAMLTagOrCustomDecode(t *testing.T) { + // Seed roots: types the loader projects onto at the end of the + // pipeline. ConfigDetails / ConfigFile are caller-input shapes + // (the user fills them before calling LoadWithContext) and are + // intentionally untagged, so they are out of scope. + roots := []reflect.Type{ + reflect.TypeOf(Project{}), + reflect.TypeOf(IncludeConfig{}), + reflect.TypeOf(ExtendsConfig{}), + } + + visited := map[reflect.Type]bool{} + for _, r := range roots { + visit(t, r, visited) + } +} + +// visit recursively descends into struct fields, slices, maps, pointers +// and arrays. Every struct it reaches is checked for the +// yaml-tag-or-UnmarshalYAML invariant. +func visit(t *testing.T, typ reflect.Type, visited map[reflect.Type]bool) { + t.Helper() + for typ.Kind() == reflect.Pointer { + typ = typ.Elem() + } + if visited[typ] { + return + } + visited[typ] = true + + switch typ.Kind() { + case reflect.Struct: + // Types out of compose-go (yaml.Node, time.Duration, ...) are out + // of scope: we only check what the package owns. + if !strings.HasPrefix(typ.PkgPath(), "github.com/compose-spec/compose-go/v3") { + return + } + if implementsUnmarshalYAML(typ) { + // Custom decode opts the type out of the field-tag invariant + // by definition. Still descend into the field types in case + // they have nested structs that should be checked. + descend(t, typ, visited) + return + } + for i := 0; i < typ.NumField(); i++ { + f := typ.Field(i) + if !f.IsExported() { + continue + } + if !hasYAMLTag(f) { + t.Errorf("%s.%s: exported field has no yaml tag and the type has no UnmarshalYAML method", + typ.String(), f.Name) + } + } + descend(t, typ, visited) + case reflect.Slice, reflect.Array, reflect.Map: + visit(t, typ.Elem(), visited) + if typ.Kind() == reflect.Map { + visit(t, typ.Key(), visited) + } + } +} + +// descend walks the struct field types (recurses one level deeper). +func descend(t *testing.T, typ reflect.Type, visited map[reflect.Type]bool) { + for i := 0; i < typ.NumField(); i++ { + visit(t, typ.Field(i).Type, visited) + } +} + +// hasYAMLTag reports whether the field carries a yaml tag (any +// non-empty value, including the special "-" form that disables +// decoding -- a field tagged yaml:"-" is intentionally excluded). +func hasYAMLTag(f reflect.StructField) bool { + tag, ok := f.Tag.Lookup("yaml") + if !ok { + return false + } + return tag != "" +} + +// implementsUnmarshalYAML returns true when either the type or a +// pointer to the type satisfies the yaml.Unmarshaler interface (yaml.v4 +// dispatch checks both addressable and non-addressable receivers). +func implementsUnmarshalYAML(typ reflect.Type) bool { + unmarshaler := reflect.TypeOf((*yaml.Unmarshaler)(nil)).Elem() + if typ.Implements(unmarshaler) { + return true + } + return reflect.PointerTo(typ).Implements(unmarshaler) +} diff --git a/types/services.go b/types/services.go index 0efc4b9fa..18f9b7bdf 100644 --- a/types/services.go +++ b/types/services.go @@ -16,9 +16,38 @@ package types +import ( + "fmt" + + "go.yaml.in/yaml/v4" +) + // Services is a map of ServiceConfig type Services map[string]ServiceConfig +// UnmarshalYAML decodes the services mapping and injects each map key into +// the corresponding ServiceConfig.Name field. Replaces the v2 nameServices +// mapstructure decode hook so the value populated on Project.Services is +// self-describing. +func (s *Services) UnmarshalYAML(value *yaml.Node) error { + value = unwrapDocument(value) + if value.Kind != yaml.MappingNode { + return fmt.Errorf("invalid services config type, expected mapping, got %v", value.Kind) + } + out := Services{} + for i := 0; i+1 < len(value.Content); i += 2 { + name := value.Content[i].Value + var svc ServiceConfig + if err := value.Content[i+1].Decode(&svc); err != nil { + return fmt.Errorf("services.%s: %w", name, err) + } + svc.Name = name + out[name] = svc + } + *s = out + return nil +} + // GetProfiles retrieve the profiles implicitly enabled by explicitly targeting selected services func (s Services) GetProfiles() []string { set := map[string]struct{}{} diff --git a/types/ssh.go b/types/ssh.go index 6d0edb695..aa9350162 100644 --- a/types/ssh.go +++ b/types/ssh.go @@ -18,11 +18,13 @@ package types import ( "fmt" + + "go.yaml.in/yaml/v4" ) type SSHKey struct { ID string `yaml:"id,omitempty" json:"id,omitempty"` - Path string `path:"path,omitempty" json:"path,omitempty"` + Path string `yaml:"path,omitempty" json:"path,omitempty"` } // SSHConfig is a mapping type for SSH build config @@ -53,20 +55,22 @@ func (s SSHKey) MarshalJSON() ([]byte, error) { return []byte(fmt.Sprintf(`%q: %s`, s.ID, s.Path)), nil } -func (s *SSHConfig) DecodeMapstructure(value interface{}) error { - v, ok := value.(map[string]any) - if !ok { - return fmt.Errorf("invalid ssh config type %T", value) +// UnmarshalYAML accepts a canonical mapping of `id: path` entries (the +// short-form `default` and `id=path` forms are turned into this shape by +// transform.CanonicalNode before decoding) and stores them as a slice of +// SSHKey. Mirrors DecodeMapstructure for yaml.v4 native decoding. +func (s *SSHConfig) UnmarshalYAML(value *yaml.Node) error { + value = unwrapDocument(value) + if value.Kind != yaml.MappingNode { + return fmt.Errorf("invalid ssh config type, expected mapping, got %v", value.Kind) } - result := make(SSHConfig, len(v)) - i := 0 - for id, path := range v { - key := SSHKey{ID: id} - if path != nil { - key.Path = fmt.Sprint(path) + result := make(SSHConfig, 0, len(value.Content)/2) + for i := 0; i+1 < len(value.Content); i += 2 { + key := SSHKey{ID: value.Content[i].Value} + if v := value.Content[i+1]; v.Kind == yaml.ScalarNode && v.Tag != "!!null" { + key.Path = v.Value } - result[i] = key - i++ + result = append(result, key) } *s = result return nil diff --git a/types/stringOrList.go b/types/stringOrList.go index a6720df08..bcd350280 100644 --- a/types/stringOrList.go +++ b/types/stringOrList.go @@ -16,27 +16,30 @@ package types -import "fmt" +import ( + "fmt" + + "go.yaml.in/yaml/v4" +) // StringList is a type for fields that can be a string or list of strings type StringList []string -func (l *StringList) DecodeMapstructure(value interface{}) error { - switch v := value.(type) { - case string: - *l = []string{v} - case []interface{}: - list := make([]string, len(v)) - for i, e := range v { - val, ok := e.(string) - if !ok { - return fmt.Errorf("invalid type %T for string list", value) - } - list[i] = val +// UnmarshalYAML accepts a string or a sequence of strings and stores the +// values in l. Mirrors DecodeMapstructure for yaml.v4 native decoding. +func (l *StringList) UnmarshalYAML(value *yaml.Node) error { + value = unwrapDocument(value) + switch value.Kind { + case yaml.ScalarNode: + *l = []string{value.Value} + case yaml.SequenceNode: + var list []string + if err := value.Decode(&list); err != nil { + return err } *l = list default: - return fmt.Errorf("invalid type %T for string list", value) + return fmt.Errorf("invalid yaml kind %d for string list", value.Kind) } return nil } @@ -44,18 +47,25 @@ func (l *StringList) DecodeMapstructure(value interface{}) error { // StringOrNumberList is a type for fields that can be a list of strings or numbers type StringOrNumberList []string -func (l *StringOrNumberList) DecodeMapstructure(value interface{}) error { - switch v := value.(type) { - case string: - *l = []string{v} - case []interface{}: - list := make([]string, len(v)) - for i, e := range v { - list[i] = fmt.Sprint(e) +// UnmarshalYAML accepts a string or a sequence of scalar entries (string or +// number, coerced to their stringified form) and stores the values in l. +// Mirrors DecodeMapstructure for yaml.v4 native decoding. +func (l *StringOrNumberList) UnmarshalYAML(value *yaml.Node) error { + value = unwrapDocument(value) + switch value.Kind { + case yaml.ScalarNode: + *l = []string{value.Value} + case yaml.SequenceNode: + list := make([]string, 0, len(value.Content)) + for _, item := range value.Content { + if item.Kind != yaml.ScalarNode { + return fmt.Errorf("string-or-number list expects scalar entries") + } + list = append(list, item.Value) } *l = list default: - return fmt.Errorf("invalid type %T for string list", value) + return fmt.Errorf("invalid yaml kind %d for string-or-number list", value.Kind) } return nil } diff --git a/types/types.go b/types/types.go index fd4f35136..1d296b5a8 100644 --- a/types/types.go +++ b/types/types.go @@ -26,6 +26,7 @@ import ( "github.com/docker/go-connections/nat" "github.com/xhit/go-str2duration/v2" + "go.yaml.in/yaml/v4" ) // ServiceConfig is the configuration of one service @@ -324,16 +325,16 @@ type DeviceMapping struct { // WeightDevice is a structure that holds device:weight pair type WeightDevice struct { - Path string - Weight uint16 + Path string `yaml:"path,omitempty" json:"path,omitempty"` + Weight uint16 `yaml:"weight,omitempty" json:"weight,omitempty"` Extensions Extensions `yaml:"#extensions,inline,omitempty" json:"-"` } // ThrottleDevice is a structure that holds device:rate_per_second pair type ThrottleDevice struct { - Path string - Rate UnitBytes + Path string `yaml:"path,omitempty" json:"path,omitempty"` + Rate UnitBytes `yaml:"rate,omitempty" json:"rate,omitempty"` Extensions Extensions `yaml:"#extensions,inline,omitempty" json:"-"` } @@ -410,8 +411,8 @@ type GenericResource struct { // "Kind" is used to describe the Kind of a resource (e.g: "GPU", "FPGA", "SSD", ...) // Value is used to count the resource (SSD=5, HDD=3, ...) type DiscreteGenericResource struct { - Kind string `json:"kind"` - Value int64 `json:"value"` + Kind string `yaml:"kind" json:"kind"` + Value int64 `yaml:"value" json:"value"` Extensions Extensions `yaml:"#extensions,inline,omitempty" json:"-"` } @@ -578,6 +579,11 @@ func (o OptOut) IsZero() bool { return bool(o) } +// IsTrue returns the effective boolean value carried by the OptOut. +func (o OptOut) IsTrue() bool { + return bool(o) +} + // SELinux represents the SELinux re-labeling options. const ( // SELinuxShared option indicates that the bind mount content is shared among multiple containers @@ -638,21 +644,32 @@ type FileReferenceConfig struct { Extensions Extensions `yaml:"#extensions,inline,omitempty" json:"-"` } -func (f *FileMode) DecodeMapstructure(value interface{}) error { - switch v := value.(type) { - case *FileMode: - return nil - case string: - i, err := strconv.ParseInt(v, 8, 64) - if err != nil { - return err - } +// UnmarshalYAML accepts a scalar value representing a file mode. +// Octal is tried first because compose file modes are conventionally +// written in Unix octal notation (`mode: 0755` and `mode: 755` both +// mean rwxr-xr-x); a decimal fallback covers the corner case where +// the yaml round-trip done by Transform / Canonical re-emits an +// octal literal as its decimal equivalent (`mode: 0440` reaches us +// as the int 288, which has no valid octal reading). +// +// Values that parse in both bases keep the octal reading -- so "755" +// is FileMode(0o755) = 493, never FileMode(755). The contract is +// surprising for non-Unix audiences but matches the v2 / spec +// behavior every fixture relies on. +func (f *FileMode) UnmarshalYAML(value *yaml.Node) error { + value = unwrapDocument(value) + if value.Kind != yaml.ScalarNode { + return fmt.Errorf("expected scalar file mode, got kind %d", value.Kind) + } + if i, err := strconv.ParseInt(value.Value, 8, 64); err == nil { *f = FileMode(i) - case int: - *f = FileMode(v) - default: - return fmt.Errorf("unexpected value type %T for mode", value) + return nil } + i, err := strconv.ParseInt(value.Value, 10, 64) + if err != nil { + return fmt.Errorf("invalid file mode %q: %w", value.Value, err) + } + *f = FileMode(i) return nil } @@ -685,27 +702,41 @@ type UlimitsConfig struct { Extensions Extensions `yaml:"#extensions,inline,omitempty" json:"-"` } -func (u *UlimitsConfig) DecodeMapstructure(value interface{}) error { - switch v := value.(type) { - case *UlimitsConfig: - // this call to DecodeMapstructure is triggered after initial value conversion as we use a map[string]*UlimitsConfig - return nil - case int: - u.Single = v +// UnmarshalYAML accepts either a scalar integer (single-value form) or a +// mapping with soft / hard fields. Mirrors DecodeMapstructure for yaml.v4 +// native decoding. +func (u *UlimitsConfig) UnmarshalYAML(value *yaml.Node) error { + value = unwrapDocument(value) + switch value.Kind { + case yaml.ScalarNode: + i, err := strconv.Atoi(value.Value) + if err != nil { + return fmt.Errorf("invalid ulimit value %q: %w", value.Value, err) + } + u.Single = i u.Soft = 0 u.Hard = 0 - case map[string]any: + case yaml.MappingNode: u.Single = 0 - soft, ok := v["soft"] - if ok { - u.Soft = soft.(int) - } - hard, ok := v["hard"] - if ok { - u.Hard = hard.(int) + for i := 0; i+1 < len(value.Content); i += 2 { + key := value.Content[i].Value + val := value.Content[i+1] + if val.Kind != yaml.ScalarNode { + return fmt.Errorf("ulimit %s must be a scalar", key) + } + n, err := strconv.Atoi(val.Value) + if err != nil { + return fmt.Errorf("invalid ulimit %s value %q: %w", key, val.Value, err) + } + switch key { + case "soft": + u.Soft = n + case "hard": + u.Hard = n + } } default: - return fmt.Errorf("unexpected value type %T for ulimit", value) + return fmt.Errorf("unexpected yaml kind %d for ulimit", value.Kind) } return nil } @@ -850,6 +881,35 @@ func (s SecretConfig) MarshalJSON() ([]byte, error) { return json.Marshal(FileObjectConfig(s)) } +// UnmarshalYAML decodes the canonical mapping into a SecretConfig and +// lifts a SecretConfigXValue extension up into Content. Replaces the v2 +// secretConfigDecoderHook mapstructure hook, which read the same +// extension out of the temporary map. +func (s *SecretConfig) UnmarshalYAML(value *yaml.Node) error { + var raw FileObjectConfig + if err := value.Decode(&raw); err != nil { + return err + } + liftXContent(&raw) + *s = SecretConfig(raw) + return nil +} + +func liftXContent(f *FileObjectConfig) { + if f.Extensions == nil { + return + } + if val, ok := f.Extensions[SecretConfigXValue]; ok { + if str, ok := val.(string); ok { + f.Content = str + } + delete(f.Extensions, SecretConfigXValue) + if len(f.Extensions) == 0 { + f.Extensions = nil + } + } +} + // ConfigObjConfig is the config for the swarm "Config" object type ConfigObjConfig FileObjectConfig @@ -871,6 +931,20 @@ func (s ConfigObjConfig) MarshalJSON() ([]byte, error) { return json.Marshal(FileObjectConfig(s)) } +// UnmarshalYAML decodes the canonical mapping into a ConfigObjConfig and +// lifts a SecretConfigXValue extension up into Content. Mirrors the +// SecretConfig handling so configs whose value was provided via the +// `environment` indirection surface their resolved Content at decode time. +func (s *ConfigObjConfig) UnmarshalYAML(value *yaml.Node) error { + var raw FileObjectConfig + if err := value.Decode(&raw); err != nil { + return err + } + liftXContent(&raw) + *s = ConfigObjConfig(raw) + return nil +} + type IncludeConfig struct { Path StringList `yaml:"path,omitempty" json:"path,omitempty"` ProjectDirectory string `yaml:"project_directory,omitempty" json:"project_directory,omitempty"` diff --git a/types/unmarshal_yaml_test.go b/types/unmarshal_yaml_test.go new file mode 100644 index 000000000..d65eb2de0 --- /dev/null +++ b/types/unmarshal_yaml_test.go @@ -0,0 +1,376 @@ +/* + Copyright 2020 The Compose Specification Authors. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +*/ + +package types + +import ( + "testing" + + "go.yaml.in/yaml/v4" + "gotest.tools/v3/assert" +) + +func TestStringList_UnmarshalYAML_Scalar(t *testing.T) { + var list StringList + assert.NilError(t, yaml.Unmarshal([]byte("nginx"), &list)) + assert.DeepEqual(t, list, StringList{"nginx"}) +} + +func TestStringList_UnmarshalYAML_Sequence(t *testing.T) { + var list StringList + assert.NilError(t, yaml.Unmarshal([]byte("- a\n- b\n"), &list)) + assert.DeepEqual(t, list, StringList{"a", "b"}) +} + +func TestStringList_UnmarshalYAML_InsideStruct(t *testing.T) { + // Confirm yaml.v4 picks up our UnmarshalYAML when decoding a struct + // that has a StringList field. + type wrapper struct { + Names StringList `yaml:"names"` + } + var w wrapper + assert.NilError(t, yaml.Unmarshal([]byte("names: single"), &w)) + assert.DeepEqual(t, w.Names, StringList{"single"}) + assert.NilError(t, yaml.Unmarshal([]byte("names: [one, two]"), &w)) + assert.DeepEqual(t, w.Names, StringList{"one", "two"}) +} + +func TestStringOrNumberList_UnmarshalYAML(t *testing.T) { + var list StringOrNumberList + assert.NilError(t, yaml.Unmarshal([]byte("- 80\n- \"443\"\n- ssh\n"), &list)) + assert.DeepEqual(t, list, StringOrNumberList{"80", "443", "ssh"}) +} + +func TestShellCommand_UnmarshalYAML_Scalar(t *testing.T) { + var cmd ShellCommand + assert.NilError(t, yaml.Unmarshal([]byte("nginx -g \"daemon off;\""), &cmd)) + assert.DeepEqual(t, cmd, ShellCommand{"nginx", "-g", "daemon off;"}) +} + +func TestShellCommand_UnmarshalYAML_Sequence(t *testing.T) { + var cmd ShellCommand + assert.NilError(t, yaml.Unmarshal([]byte("- nginx\n- -g\n- daemon off;\n"), &cmd)) + assert.DeepEqual(t, cmd, ShellCommand{"nginx", "-g", "daemon off;"}) +} + +func TestHealthCheckTest_UnmarshalYAML_Scalar(t *testing.T) { + var test HealthCheckTest + assert.NilError(t, yaml.Unmarshal([]byte("curl -f http://localhost/"), &test)) + // Short form is wrapped in CMD-SHELL. + assert.DeepEqual(t, test, HealthCheckTest{"CMD-SHELL", "curl -f http://localhost/"}) +} + +func TestHealthCheckTest_UnmarshalYAML_Sequence(t *testing.T) { + var test HealthCheckTest + assert.NilError(t, yaml.Unmarshal([]byte("- CMD\n- curl\n- -f\n- http://localhost/\n"), &test)) + assert.DeepEqual(t, test, HealthCheckTest{"CMD", "curl", "-f", "http://localhost/"}) +} + +// TestIncludeConfig_UnmarshalYAML_StringListShortForm confirms that the +// improved StringList unmarshaller lets yaml.v4 decode an include entry +// natively, including the path / env_file scalar short form. +func TestIncludeConfig_UnmarshalYAML_StringListShortForm(t *testing.T) { + var cfg IncludeConfig + src := ` +path: compose.yaml +project_directory: ./sub +env_file: .env.shared +` + assert.NilError(t, yaml.Unmarshal([]byte(src), &cfg)) + assert.DeepEqual(t, cfg.Path, StringList{"compose.yaml"}) + assert.Equal(t, cfg.ProjectDirectory, "./sub") + assert.DeepEqual(t, cfg.EnvFile, StringList{".env.shared"}) +} + +func TestIncludeConfig_UnmarshalYAML_StringListLongForm(t *testing.T) { + var cfg IncludeConfig + src := ` +path: + - first.yaml + - second.yaml +env_file: + - .env.a + - .env.b +` + assert.NilError(t, yaml.Unmarshal([]byte(src), &cfg)) + assert.DeepEqual(t, cfg.Path, StringList{"first.yaml", "second.yaml"}) + assert.DeepEqual(t, cfg.EnvFile, StringList{".env.a", ".env.b"}) +} + +func TestLabels_UnmarshalYAML_Mapping(t *testing.T) { + var l Labels + src := ` +com.example.a: "1" +com.example.b: hello +com.example.c: 42 +` + assert.NilError(t, yaml.Unmarshal([]byte(src), &l)) + assert.Equal(t, l["com.example.a"], "1") + assert.Equal(t, l["com.example.b"], "hello") + assert.Equal(t, l["com.example.c"], "42") +} + +func TestLabels_UnmarshalYAML_List(t *testing.T) { + var l Labels + src := ` +- com.example.a=value +- com.example.b= +- com.example.c +` + assert.NilError(t, yaml.Unmarshal([]byte(src), &l)) + assert.Equal(t, l["com.example.a"], "value") + assert.Equal(t, l["com.example.b"], "") + assert.Equal(t, l["com.example.c"], "") +} + +func TestMapping_UnmarshalYAML_Mapping(t *testing.T) { + var m Mapping + src := ` +FOO: bar +EMPTY: +NUM: 42 +` + assert.NilError(t, yaml.Unmarshal([]byte(src), &m)) + assert.Equal(t, m["FOO"], "bar") + assert.Equal(t, m["EMPTY"], "") + assert.Equal(t, m["NUM"], "42") +} + +func TestMapping_UnmarshalYAML_List(t *testing.T) { + var m Mapping + src := ` +- FOO=bar +- EMPTY= +- BARE +` + assert.NilError(t, yaml.Unmarshal([]byte(src), &m)) + assert.Equal(t, m["FOO"], "bar") + assert.Equal(t, m["EMPTY"], "") + assert.Equal(t, m["BARE"], "") +} + +func TestMappingWithEquals_UnmarshalYAML_NilVsEmptyPreserved(t *testing.T) { + var m MappingWithEquals + src := ` +- WITH_VALUE=hello +- EMPTY_VALUE= +- BARE_KEY +` + assert.NilError(t, yaml.Unmarshal([]byte(src), &m)) + // WITH_VALUE: non-nil pointer with "hello". + assert.Assert(t, m["WITH_VALUE"] != nil) + assert.Equal(t, *m["WITH_VALUE"], "hello") + // EMPTY_VALUE: non-nil pointer with "". + assert.Assert(t, m["EMPTY_VALUE"] != nil) + assert.Equal(t, *m["EMPTY_VALUE"], "") + // BARE_KEY: nil pointer. + v, present := m["BARE_KEY"] + assert.Assert(t, present) + assert.Assert(t, v == nil) +} + +func TestMappingWithEquals_UnmarshalYAML_MappingTrailingSpace(t *testing.T) { + var m MappingWithEquals + src := ` +- "FOO =bar" +` + err := yaml.Unmarshal([]byte(src), &m) + assert.ErrorContains(t, err, "trailing space") +} + +func TestHostsList_UnmarshalYAML_ListShortForm(t *testing.T) { + var h HostsList + src := ` +- "host1:1.2.3.4" +- "host2=5.6.7.8" +` + assert.NilError(t, yaml.Unmarshal([]byte(src), &h)) + assert.DeepEqual(t, h["host1"], []string{"1.2.3.4"}) + assert.DeepEqual(t, h["host2"], []string{"5.6.7.8"}) +} + +func TestHostsList_UnmarshalYAML_Mapping(t *testing.T) { + var h HostsList + src := ` +host1: 1.2.3.4 +host2: + - 5.6.7.8 + - 9.10.11.12 +` + assert.NilError(t, yaml.Unmarshal([]byte(src), &h)) + assert.DeepEqual(t, h["host1"], []string{"1.2.3.4"}) + assert.DeepEqual(t, h["host2"], []string{"5.6.7.8", "9.10.11.12"}) +} + +func TestDuration_UnmarshalYAML(t *testing.T) { + var d Duration + assert.NilError(t, yaml.Unmarshal([]byte("1h30m"), &d)) + assert.Equal(t, d.String(), "1h30m0s") +} + +func TestNanoCPUs_UnmarshalYAML(t *testing.T) { + var n NanoCPUs + assert.NilError(t, yaml.Unmarshal([]byte("0.5"), &n)) + assert.Equal(t, n.Value(), float32(0.5)) + assert.NilError(t, yaml.Unmarshal([]byte("\"1.5\""), &n)) + assert.Equal(t, n.Value(), float32(1.5)) +} + +func TestDeviceCount_UnmarshalYAML_All(t *testing.T) { + var c DeviceCount + assert.NilError(t, yaml.Unmarshal([]byte("all"), &c)) + assert.Equal(t, int64(c), int64(-1)) +} + +func TestDeviceCount_UnmarshalYAML_Integer(t *testing.T) { + var c DeviceCount + assert.NilError(t, yaml.Unmarshal([]byte("4"), &c)) + assert.Equal(t, int64(c), int64(4)) +} + +func TestDeviceCount_UnmarshalYAML_InvalidString(t *testing.T) { + var c DeviceCount + err := yaml.Unmarshal([]byte("some"), &c) + assert.ErrorContains(t, err, "the only value allowed is 'all' or a number") +} + +func TestFileMode_UnmarshalYAML(t *testing.T) { + var f FileMode + assert.NilError(t, yaml.Unmarshal([]byte("0755"), &f)) + // 0755 octal == 493 decimal + assert.Equal(t, int64(f), int64(493)) +} + +func TestUlimitsConfig_UnmarshalYAML_Scalar(t *testing.T) { + var u UlimitsConfig + assert.NilError(t, yaml.Unmarshal([]byte("65535"), &u)) + assert.Equal(t, u.Single, 65535) + assert.Equal(t, u.Soft, 0) + assert.Equal(t, u.Hard, 0) +} + +func TestUlimitsConfig_UnmarshalYAML_Mapping(t *testing.T) { + var u UlimitsConfig + src := ` +soft: 1000 +hard: 2000 +` + assert.NilError(t, yaml.Unmarshal([]byte(src), &u)) + assert.Equal(t, u.Single, 0) + assert.Equal(t, u.Soft, 1000) + assert.Equal(t, u.Hard, 2000) +} + +func TestOptions_UnmarshalYAML(t *testing.T) { + var o Options + src := ` +max-size: 10m +max-file: "3" +` + assert.NilError(t, yaml.Unmarshal([]byte(src), &o)) + assert.Equal(t, o["max-size"], "10m") + assert.Equal(t, o["max-file"], "3") +} + +func TestMultiOptions_UnmarshalYAML_Mixed(t *testing.T) { + var m MultiOptions + src := ` +single: value +list: + - a + - b +` + assert.NilError(t, yaml.Unmarshal([]byte(src), &m)) + assert.DeepEqual(t, m["single"], []string{"value"}) + assert.DeepEqual(t, m["list"], []string{"a", "b"}) +} + +// TestOptions_UnmarshalYAML_RejectsNonScalarValue covers the Copilot +// review finding that Options used to silently turn a non-scalar value +// into "" via scalarToString. A sequence value is now rejected. +func TestOptions_UnmarshalYAML_RejectsNonScalarValue(t *testing.T) { + var d Options + err := yaml.Unmarshal([]byte("foo: [a, b]\n"), &d) + assert.ErrorContains(t, err, "expected scalar") +} + +// TestOptions_UnmarshalYAML_RejectsMappingValue covers the same fix on +// a mapping payload. +func TestOptions_UnmarshalYAML_RejectsMappingValue(t *testing.T) { + var d Options + err := yaml.Unmarshal([]byte("foo: {bar: baz}\n"), &d) + assert.ErrorContains(t, err, "expected scalar") +} + +// TestMultiOptions_UnmarshalYAML_RejectsNonScalarSequenceEntry covers +// the Copilot review finding that MultiOptions used to silently turn a +// nested non-scalar (e.g. `key: [[a]]`) into "". +func TestMultiOptions_UnmarshalYAML_RejectsNonScalarSequenceEntry(t *testing.T) { + var d MultiOptions + err := yaml.Unmarshal([]byte("foo:\n - [a, b]\n"), &d) + assert.ErrorContains(t, err, "sequence entry must be scalar") +} + +// TestSSHConfig_UnmarshalYAML_TopLevelDocument covers the Copilot +// review finding that SSHConfig did not unwrap a DocumentNode wrapper: +// a caller passing the YAML straight to yaml.Unmarshal got an +// incorrect "expected mapping" error. +func TestSSHConfig_UnmarshalYAML_TopLevelDocument(t *testing.T) { + var s SSHConfig + assert.NilError(t, yaml.Unmarshal([]byte("default: ~\nfoo: /tmp/foo\n"), &s)) + assert.Equal(t, len(s), 2) +} + +// TestServices_UnmarshalYAML_TopLevelDocument covers the same +// DocumentNode wrapper unwrap, for the Services type. +func TestServices_UnmarshalYAML_TopLevelDocument(t *testing.T) { + var s Services + assert.NilError(t, yaml.Unmarshal([]byte("web:\n image: nginx\n"), &s)) + web, ok := s["web"] + assert.Assert(t, ok) + assert.Equal(t, web.Name, "web") + assert.Equal(t, web.Image, "nginx") +} + +// TestFileMode_UnmarshalYAML_OctalFirstThenDecimal documents the +// FileMode parsing contract that the Copilot review prompted us to +// clarify: octal is tried first, and decimal is the fallback for +// values that don't parse as octal (the canonical post-round-trip +// "288" form of `mode: 0440`). Values valid in both bases keep the +// octal reading. +func TestFileMode_UnmarshalYAML_OctalFirstThenDecimal(t *testing.T) { + cases := []struct { + in string + want FileMode + }{ + // Unix octal notation (with and without leading zero). + {"0755", 0o755}, + {"755", 0o755}, + // Decimal-only digits like "288" (= 0o440) cannot parse as + // octal because of the '8' and fall back to decimal. + {"288", 288}, + {"0288", 288}, + // "0440" parses cleanly as octal. + {"0440", 0o440}, + } + for _, tc := range cases { + t.Run(tc.in, func(t *testing.T) { + var f FileMode + assert.NilError(t, yaml.Unmarshal([]byte(tc.in), &f)) + assert.Equal(t, f, tc.want) + }) + } +} diff --git a/types/yaml_helpers.go b/types/yaml_helpers.go new file mode 100644 index 000000000..a53a95a43 --- /dev/null +++ b/types/yaml_helpers.go @@ -0,0 +1,56 @@ +/* + Copyright 2020 The Compose Specification Authors. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +*/ + +package types + +import "go.yaml.in/yaml/v4" + +// unwrapDocument peels off the DocumentNode wrapper from n when present, so +// custom UnmarshalYAML implementations can be invoked transparently both by +// yaml.Decoder (which forwards the inner node) and by yaml.Unmarshal on a +// top-level value (which forwards the DocumentNode). +func unwrapDocument(n *yaml.Node) *yaml.Node { + if n != nil && n.Kind == yaml.DocumentNode && len(n.Content) == 1 { + return n.Content[0] + } + return n +} + +// scalarToString returns the string representation of a scalar node, +// treating !!null tagged scalars and nil nodes as empty strings. Numeric +// and boolean scalars are returned verbatim because yaml.v4 preserves the +// source representation in Node.Value regardless of Tag, which mirrors the +// fmt.Sprint(e) behavior of the v2 mapstructure helpers. +func scalarToString(n *yaml.Node) string { + if n == nil || n.Kind != yaml.ScalarNode { + return "" + } + if n.Tag == "!!null" { + return "" + } + return n.Value +} + +// scalarToStringPtr returns a *string for a scalar node, distinguishing the +// !!null tag (returns nil) from an empty string (returns a pointer to ""): +// the same distinction MappingWithEquals encodes with `key=` vs `key`. +func scalarToStringPtr(n *yaml.Node) *string { + if n == nil || n.Kind != yaml.ScalarNode || n.Tag == "!!null" { + return nil + } + v := n.Value + return &v +} diff --git a/validation/node.go b/validation/node.go new file mode 100644 index 000000000..db7828612 --- /dev/null +++ b/validation/node.go @@ -0,0 +1,218 @@ +/* + Copyright 2020 The Compose Specification Authors. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +*/ + +package validation + +import ( + "errors" + "fmt" + "net" + "strings" + + "go.yaml.in/yaml/v4" + + "github.com/compose-spec/compose-go/v3/tree" +) + +type nodeChecker func(n *yaml.Node, p tree.Path) error + +// nodeChecks mirrors `checks` but operates on *yaml.Node so the v3 pipeline +// can validate the merged tree without round-tripping through map[string]any. +// Entries stay in sync with the legacy map; the v2 map disappears when the +// map-based code path is removed. +var nodeChecks = map[tree.Path]nodeChecker{ + "volumes.*": checkVolumeNode, + "configs.*": checkFileObjectNode("file", "environment", "content"), + "secrets.*": checkFileObjectNode("file", "environment"), + "services.*.ports.*": checkIPAddressNode, + "services.*.develop.watch.*.path": checkPathNode, + "services.*.deploy.resources.reservations.devices.*": checkDeviceRequestNode, + "services.*.gpus.*": checkDeviceRequestNode, +} + +// Error carries the offending node and path alongside the underlying +// validation failure so the loader can wrap it with the source file +// from the origins side-table when surfacing the error. +// +// The type name intentionally avoids the "ValidationError" stuttering +// against the package name; consumers should refer to it as +// *validation.Error. +type Error struct { + Path tree.Path + Node *yaml.Node + Cause error +} + +// Error renders as "path: cause" so the existing test assertions that +// match on the substring keep working when validation is not wrapped +// further upstream. +func (e *Error) Error() string { + if e == nil { + return "" + } + if e.Path.String() == "" { + return e.Cause.Error() + } + return e.Path.String() + ": " + e.Cause.Error() +} + +// Unwrap exposes Cause so errors.Is / errors.As walk through. +func (e *Error) Unwrap() error { + if e == nil { + return nil + } + return e.Cause +} + +// ValidateNode walks root and applies the per-path validation checks. The +// tree is not mutated; only errors are reported. The function returns at the +// first failing check, with the offending tree.Path and *yaml.Node included +// in a *ValidationError so callers can map it back to a source location. +func ValidateNode(root *yaml.Node) error { + if root == nil { + return nil + } + target := root + if target.Kind == yaml.DocumentNode && len(target.Content) == 1 { + target = target.Content[0] + } + return checkNode(target, tree.NewPath()) +} + +func wrapCheckError(err error, node *yaml.Node, p tree.Path) error { + if err == nil { + return nil + } + var ve *Error + if errors.As(err, &ve) { + return ve + } + return &Error{Path: p, Node: node, Cause: stripPathPrefix(err, p)} +} + +// stripPathPrefix removes the "path: " prefix the per-check helpers +// embed in their error strings so wrapping does not duplicate it. +func stripPathPrefix(err error, p tree.Path) error { + prefix := p.String() + ": " + if prefix == ": " { + return err + } + msg := err.Error() + if len(msg) > len(prefix) && msg[:len(prefix)] == prefix { + return errString(msg[len(prefix):]) + } + return err +} + +type errString string + +func (e errString) Error() string { return string(e) } + +func checkNode(n *yaml.Node, p tree.Path) error { + if n == nil { + return nil + } + for pattern, fn := range nodeChecks { + if p.Matches(pattern) { + return wrapCheckError(fn(n, p), n, p) + } + } + switch n.Kind { + case yaml.MappingNode: + for i := 0; i+1 < len(n.Content); i += 2 { + if err := checkNode(n.Content[i+1], p.Next(n.Content[i].Value)); err != nil { + return err + } + } + case yaml.SequenceNode: + for _, c := range n.Content { + if err := checkNode(c, p.Next(tree.PathMatchList)); err != nil { + return err + } + } + } + return nil +} + +func checkFileObjectNode(keys ...string) nodeChecker { + return func(n *yaml.Node, p tree.Path) error { + if n == nil || n.Kind != yaml.MappingNode { + return nil + } + count := 0 + for _, k := range keys { + if mappingFieldNode(n, k) != nil { + count++ + } + } + if count > 1 { + return fmt.Errorf("%s: %s attributes are mutually exclusive", p, strings.Join(keys, "|")) + } + if count == 0 { + if mappingFieldNode(n, "driver") != nil { + // Custom driver: may carry its own content channel. + return nil + } + if mappingFieldNode(n, "external") == nil { + return fmt.Errorf("%s: one of %s must be set", p, strings.Join(keys, "|")) + } + } + return nil + } +} + +func checkPathNode(n *yaml.Node, p tree.Path) error { + if n == nil || n.Kind != yaml.ScalarNode || n.Value == "" { + return fmt.Errorf("%s: value can't be blank", p) + } + return nil +} + +func checkDeviceRequestNode(n *yaml.Node, p tree.Path) error { + if n == nil || n.Kind != yaml.MappingNode { + return nil + } + if mappingFieldNode(n, "count") != nil && mappingFieldNode(n, "device_ids") != nil { + return fmt.Errorf(`%s: "count" and "device_ids" attributes are exclusive`, p) + } + return nil +} + +func checkIPAddressNode(n *yaml.Node, p tree.Path) error { + if n == nil || n.Kind != yaml.MappingNode { + return nil + } + ip := mappingFieldNode(n, "host_ip") + if ip == nil || ip.Kind != yaml.ScalarNode { + return nil + } + if net.ParseIP(ip.Value) == nil { + return fmt.Errorf("%s: invalid ip address: %s", p, ip.Value) + } + return nil +} + +func mappingFieldNode(n *yaml.Node, key string) *yaml.Node { + if n == nil || n.Kind != yaml.MappingNode { + return nil + } + for i := 0; i+1 < len(n.Content); i += 2 { + if n.Content[i].Value == key { + return n.Content[i+1] + } + } + return nil +} diff --git a/validation/node_test.go b/validation/node_test.go new file mode 100644 index 000000000..58257dd18 --- /dev/null +++ b/validation/node_test.go @@ -0,0 +1,154 @@ +/* + Copyright 2020 The Compose Specification Authors. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +*/ + +package validation + +import ( + "testing" + + "go.yaml.in/yaml/v4" + "gotest.tools/v3/assert" +) + +func parseNode(t *testing.T, src string) *yaml.Node { + t.Helper() + var doc yaml.Node + assert.NilError(t, yaml.Unmarshal([]byte(src), &doc)) + return &doc +} + +func TestValidateNode_AcceptsValidConfig(t *testing.T) { + root := parseNode(t, ` +services: + web: + image: nginx +volumes: + data: + external: true +`) + assert.NilError(t, ValidateNode(root)) +} + +func TestValidateNode_RejectsExternalWithExtraField(t *testing.T) { + root := parseNode(t, ` +volumes: + data: + external: true + driver: local +`) + err := ValidateNode(root) + assert.ErrorContains(t, err, "conflicting parameters") +} + +func TestValidateNode_RejectsSecretsWithMultipleSources(t *testing.T) { + root := parseNode(t, ` +secrets: + s1: + file: ./s1 + environment: S1 +`) + err := ValidateNode(root) + assert.ErrorContains(t, err, "mutually exclusive") +} + +func TestValidateNode_AcceptsSecretsWithDriver(t *testing.T) { + root := parseNode(t, ` +secrets: + s1: + driver: custom-driver +`) + assert.NilError(t, ValidateNode(root)) +} + +func TestValidateNode_RejectsConfigsMissingSource(t *testing.T) { + root := parseNode(t, ` +configs: + c1: {} +`) + err := ValidateNode(root) + assert.ErrorContains(t, err, "must be set") +} + +func TestValidateNode_RejectsBadHostIP(t *testing.T) { + root := parseNode(t, ` +services: + web: + ports: + - target: 80 + host_ip: not-an-ip +`) + err := ValidateNode(root) + assert.ErrorContains(t, err, "invalid ip address") +} + +func TestValidateNode_AcceptsValidHostIP(t *testing.T) { + root := parseNode(t, ` +services: + web: + ports: + - target: 80 + host_ip: 192.168.1.1 +`) + assert.NilError(t, ValidateNode(root)) +} + +func TestValidateNode_RejectsDeviceRequestWithCountAndIDs(t *testing.T) { + root := parseNode(t, ` +services: + web: + deploy: + resources: + reservations: + devices: + - count: 1 + device_ids: ["GPU-0"] +`) + err := ValidateNode(root) + assert.ErrorContains(t, err, "exclusive") +} + +func TestValidateNode_RejectsBlankWatchPath(t *testing.T) { + root := parseNode(t, ` +services: + web: + develop: + watch: + - action: sync + path: "" +`) + err := ValidateNode(root) + assert.ErrorContains(t, err, "blank") +} + +func TestValidateNode_NilSafe(t *testing.T) { + assert.NilError(t, ValidateNode(nil)) +} + +// TestCheckVolumeNode_NonMappingErrorIncludesKind covers the Copilot +// review finding that the previous error printed n.Value -- which is +// empty for non-scalar nodes -- and was therefore useless. The fix +// formats the offending node's kind ("sequence" / "mapping" / ...). +func TestCheckVolumeNode_NonMappingErrorIncludesKind(t *testing.T) { + var root yaml.Node + assert.NilError(t, yaml.Unmarshal([]byte(` +volumes: + bad: + - element +`), &root)) + err := ValidateNode(&root) + assert.ErrorContains(t, err, "expected volume") + assert.ErrorContains(t, err, "sequence") +} diff --git a/validation/node_volume.go b/validation/node_volume.go new file mode 100644 index 000000000..bae468e5b --- /dev/null +++ b/validation/node_volume.go @@ -0,0 +1,82 @@ +/* + Copyright 2020 The Compose Specification Authors. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +*/ + +package validation + +import ( + "fmt" + "strings" + + "go.yaml.in/yaml/v4" + + "github.com/compose-spec/compose-go/v3/consts" + "github.com/compose-spec/compose-go/v3/tree" +) + +// kindName returns a human-readable label for a yaml.Kind, used in +// error messages. yaml.v4 exposes the constants but no String() helper. +func kindName(k yaml.Kind) string { + switch k { + case yaml.DocumentNode: + return "document" + case yaml.MappingNode: + return "mapping" + case yaml.SequenceNode: + return "sequence" + case yaml.ScalarNode: + return "scalar" + case yaml.AliasNode: + return "alias" + } + return "unknown" +} + +func checkVolumeNode(n *yaml.Node, p tree.Path) error { + if n == nil { + return nil + } + if n.Kind != yaml.MappingNode { + // A `!!null` scalar (empty volume entry) is valid. + if n.Kind == yaml.ScalarNode && n.Tag == "!!null" { + return nil + } + return fmt.Errorf("expected volume, got %s", kindName(n.Kind)) + } + return checkExternalNode(n, p) +} + +func checkExternalNode(n *yaml.Node, p tree.Path) error { + external := mappingFieldNode(n, "external") + if external == nil { + return nil + } + if external.Kind != yaml.ScalarNode || external.Value != "true" { + return nil + } + for i := 0; i+1 < len(n.Content); i += 2 { + k := n.Content[i].Value + switch k { + case "name", "external", consts.Extensions: + continue + default: + if strings.HasPrefix(k, "x-") { + continue + } + return fmt.Errorf("%s: conflicting parameters \"external\" and %q specified", p, k) + } + } + return nil +}