[Library] Create build-catalog script and examples relocation

[semd]

Summary

[Library] Phase 1 — initial templates, categories vocab, build pipeline, and /examples relocation

First end-to-end pass at the Kibana Workflow Template Library source repo. Tech Preview target: Kibana 9.5+. Covers four merge-units worth of work that are easier to review as one body:

1. The first batch of installable templates.
2. The closed-vocabulary categories registry.
3. The repo-side build infrastructure (catalog generator + policy file + dependency manifest).
4. Reorganization of the legacy raw workflows out of the way so the repo's primary directory is the library.

The CDN endpoint, the publish-catalog.yml GitHub Action, and Marco's PR-time schema validator are explicitly out of scope and will follow.

---

Library content

Seven starter templates landed under library/workflows/<slug>/<slug>.yaml, each with a template-metadata block (slug, version, availability, name, description, optional solutions, categories, optional icon, optional install form). Picked for solution diversity and to exercise every codepath in the metadata schema:

• ip-reputation-check — AbuseIPDB + IP-API enrichment. Two install fields (connector, lookback days).
• hash-threat-check — VirusTotal lookup. Connector install field.
• mark-alert-as-acknowledged — security alert status flip. Minimal: no install form.
• semantic-knowledge-search — Elasticsearch semantic search with a data.parseJson-built filter clause. One install field (index name).
• web-search — Brave search via dedicated step type. Cross-solution (no solutions:).
• create-slack-channel — Slack v2 createConversation. Cross-solution.
• root-cause-analysis — alert-triggered, eight-step Agent Builder + Cases composition. One install field (agent id).

All seven use dedicated vendor step types where the schema offers them (abuseipdb.checkIp, virustotal.scanFileHash, slack2.createConversation, brave-search.webSearch, ai.agent, …). Generic http and kibana.request only appear as documented escape hatches when no dedicated step exists.

The closed categories vocabulary lives at library/categories.yaml (16 entries: enrichment, detection, response, hunting, threat-intel, notification, case-management, monitoring, root-cause-analysis, data-ingestion, data-transformation, reporting, search, ai-agent, integration, utility). New categories require a PR adding the entry in the same change.

---

Build infrastructure

scripts/build-catalog.mjs is the catalog generator. Inputs: kibana-versions.json (policy), library/categories.yaml (vocab), library/workflows/*/*.yaml (templates). Output: a complete dist/v1/ tree ready to upload to the CDN.

Notable design points:

• kibana-versions.json is a policy file, not an enumerated list. Three fields: latest (channel pointer; currently "main"), oldest (semver floor; currently "9.5.0"), cataloguePer (granularity; "minor"). The generator computes the live Kibana version set at run time, so we don't need a human PR every time Kibana cuts a new minor.
• main is resolved live from elastic/kibana@main's package.json .version, with any pre-release suffix stripped via semver.coerce. Falls back to KIBANA_MAIN_VERSION env override for local-dev and incident recovery.
• Named minors are discovered live from elastic/kibana's branch list via the GitHub API, filtered to branches matching ^X.Y$ and >= policy.oldest. Falls back to KIBANA_NAMED_MINORS env override (or "" to skip the API entirely) for local iteration without burning the 60/h unauth rate limit.
• Fail-closed validation. A malformed template, an unknown category id, a missing required field, a __install__.<name> reference with no matching install.form entry, an unreachable Kibana main, or an unauthenticated rate-limit exhaustion aborts the run with a non-zero exit. Never falls back to stale data — a failed publish is preferable to a wrong one.
• Actionable 403 error. When the GitHub branches API rate-limits, the script prints a multi-line message with the reset time, the rate-limit ceiling, and three concrete fixes (set GITHUB_TOKEN, use KIBANA_NAMED_MINORS="", or wait). Saves the next person from a confused stack trace.
• effectiveKibanaSemver on each per-version manifest.json so consumers and humans curling the CDN can see exactly what main (or any named minor) resolved to at publish time.

package.json pins js-yaml ^4.2.0 + semver ^7.8.4, sets engines.node >=20, and exposes npm run build:catalog. No glob dep — the script uses fs.readdir since the template directory layout is predictable.

---

solutions field: optional

template-metadata.solutions is optional. Three cases the build script handles:

• Field absent: the template is cross-solution and appears in every solution context (Kibana renders it alongside the solution-scoped ones).
• Field present as an empty array []: same as absent; the script normalizes this to absent in the catalog row, so consumers always see "no solutions key" for both cases.
• Field present as a non-empty array [security] / [security, observability] / [observability]: solution-scoped to those values.

Two of the seven templates (web-search, create-slack-channel) ship as cross-solution; the other five carry an explicit solutions:.

---

Repository reorganization

The historical /workflows/ directory has been moved wholesale to /examples/. ~140 files, all detected as renames by git.

Rationale: the repo's primary artifact going forward is the library under /library/workflows/. The legacy directory is reference material — useful for browsing the raw YAML grammar and as a sandbox for things that don't (yet) carry template-metadata. Renaming it to /examples/ makes that asymmetry honest without removing the content. External consumers that link at the old paths will need to update, but git's rename tracking keeps git log --follow and git blame continuity intact.

Build infrastructure stays at the repo root (package.json, scripts/, dist/, kibana-versions.json) — these are repo-level concerns, not library-internal artifacts.

---

Documentation

CONTRIBUTING.md was rewritten end-to-end as the library authoring guide. Sections: file layout and slug rules, the template-metadata block (required vs optional), categories vocabulary, install-form discipline including the kebab-case naming convention for __install__.<name> placeholders, the connector-type rule (.<vendor> equals the step-type prefix), style conventions (2-space indentation, no legacy banner headers, snake_case for workflow-body identifiers, prefer data.* steps over abusing console), local validation via npm run build:catalog including the env-var override table, semver policy, and PR flow.

README.md was slimmed from 608 lines to ~150 and refocused on the library. Kept the Elastic logo + header (as requested) and the badge strip. Covers: overview, repo structure, a minimal template format snippet, how Kibana consumes the catalog at install time, and pointers to the docs for deeper reading.

docs/ was largely left alone — concepts.md and schema.md are general workflow-engine references that apply to both library and example authors and didn't need changes. docs/importing.md got path updates (workflows/ → examples/) but kept its original content: it documents the raw-YAML import paths (Kibana UI / API / bulk) which remain useful for local dev. Library templates are NOT installed this way — they go through the catalog UI in Kibana 9.5+ (Phase 2) which renders the install form and substitutes __install__.* placeholders.

---

Validation

Every template parses with js-yaml, has the required template-metadata fields, has matching install.form ↔ __install__.<name> references with no orphans, and uses only category ids declared in library/categories.yaml. The build script enforces all of this.

End-to-end smoke test:

npm install
KIBANA_MAIN_VERSION=9.6.0 KIBANA_NAMED_MINORS="" npm run build:catalog

Produces dist/v1/{kibana-versions.json, main/catalogs/templates.json, main/manifest.json, templates/<slug>/<version>.yaml}. With KIBANA_NAMED_MINORS="9.5" you get the named-minor branch too. With no overrides (CI default), the script fetches the live values from elastic/kibana.

What the script does not yet enforce: the workflow body itself against the engine's step-type schema. That's Marco's PR-time validator, a separate piece coordinated for a sibling CI job. The publish workflow reserves a slot for it.

---

Decisions worth a second look

A handful of judgment calls during the template migrations are worth a reviewer's eye:

• hash-threat-check: categories are [enrichment, threat-intel]. The legacy directory placed it under security/detection/, but the operation is a vendor intel lookup that produces a verdict — same shape as ip-reputation-check. If you'd prefer [detection, threat-intel], the swap is one line.
• hash-threat-check: dropped the get_detailed_analysis step from the source. It hit /files/{hash}/behaviours via raw http and its output was never consumed by the report. No dedicated step type exists for that endpoint and a second .<vendor> connector would have been required. Easy to restore if there's a use case I missed.
• mark-alert-as-acknowledged: switched step type to security.setAlertStatus (the post-9.5 successor to the legacy kibana.SetAlertsStatus). Confirmed during migration.
• root-cause-analysis: the three kibana.request POST /api/agent_builder/converse calls collapsed to ai.agent. The conversation fetch (step 6) stays on kibana.request because no ai.* step exists for that. I assumed ai.agent's output exposes output.conversation_id and output.message — the JSON Schema describes inputs only, so the output shape is best-effort. If a real run shows different paths, the three references need adjusting in lockstep.
• root-cause-analysis: fixed a broken {{ consts.basePath }} reference in the source (the const was never declared). Replaced with the canonical {{ context.kibanaUrl }}. Tell me if context.kibanaUrl doesn't include the base path in your deployment model.
• ip-reputation-check: output paths for abuseipdb.checkIp are guessed as output.data.X (the source mixed output.data.data.X and output.data.X inconsistently — the latter was a bug). I normalized everything to output.data.X. If the dedicated step preserves the vendor envelope, the paths need output.data.data.X.

---

Out of scope (deliberate)

• The CDN endpoint, GCS bucket, and credentials — owned by DevEx/SRE; ticket open, ~1–3 week ETA.
• .github/workflows/publish-catalog.yml — the script is ready; the GH Action is the next PR.
• Marco's PR-time schema validator — coordinated; will slot into the publish workflow as a sibling step.
• The tarball release artifact for air-gap installs — known successor task.
• Composition primitives (__snippet__: / __role__:) — Kibana 9.6.

---

Acceptance checklist

• ≥ 6 templates in library/workflows/<slug>/<slug>.yaml with valid template-metadata.
• library/categories.yaml exists with the initial closed vocab.
• kibana-versions.json at the repo root (policy shape: latest, oldest, cataloguePer).
• scripts/build-catalog.mjs generates a complete dist/v1/ tree locally.
• CONTRIBUTING.md updated to describe the library authoring layout.
• DevEx/SRE: CDN + GCS bucket + credentials provisioned (external, in-flight).
• .github/workflows/publish-catalog.yml (next PR).
• CDN serves the expected URL shape (post-DevEx).