Skip to content

Add Keystone-on-kind integration test (forge)#294

Open
berendt wants to merge 8 commits into
mainfrom
implement/issue-293-keystone-integration-test
Open

Add Keystone-on-kind integration test (forge)#294
berendt wants to merge 8 commits into
mainfrom
implement/issue-293-keystone-integration-test

Conversation

@berendt

@berendt berendt commented Jun 30, 2026

Copy link
Copy Markdown
Member

Summary

Adds the first integration test for openstack-project-manager: it deploys a live, Keystone-only Keystone on a kind cluster via forge, then drives the real tox -e create path as the admin account to create one project and asserts the result through the OpenStack SDK. This is the first thing in the repo that exercises a real openstack.connect() / keystone path instead of the mocks used by test/unit/.

Closes #293

Change set (in commit order)

  1. 68776e6 Add forge Keystone deploy script and ControlPlane CR

    • test/integration/controlplane.yaml: a Keystone-only forge ControlPlane CR (release 2025.2, single replica, exposed at keystone.127-0-0-1.nip.io:8443/v3 via the openstack-gw gateway).
    • test/integration/deploy_keystone.sh: clones forge pinned to a known-good commit (FORGE_REF), uses forge's own install-test-deps to provide the pinned kind/kubectl toolchain, stands up the kind cluster + ControlPlane, then mints an ephemeral clouds.yaml from the operator-projected controlplane-keystone-admin-credentials Secret. It refuses to clobber an OS_CLIENT_CONFIG_FILE that points outside the dedicated in-tree path, serializes clouds.yaml with a YAML library (not a heredoc) so an arbitrary admin password can't corrupt the file, and pre-creates the member and load-balancer_member roles. The latter is needed because create.py indexes its role cache directly and a bare Keystone ships member but not load-balancer_member (Octavia normally provisions it) — create.py itself is left untouched.
  2. b7bd7ec Add integration run.sh orchestration and SDK verifier

    • test/integration/run.sh: orchestrates deploy → tox -e create (as admin, with --nodomain-name-prefix so the project is named exactly opm-integration-test) → verify.py. An EXIT trap always tears down the kind cluster, but only when this run created it (a pre-existing debug cluster is reused and left in place); on failure it dumps diagnostics first. Best-effort project deletion keeps reruns idempotent unless KEEP_PROJECT=1.
    • test/integration/verify.py: a standalone openstacksdk script (deliberately not named test_*.py, so tox -e test never collects it) that asserts the default domain resolves, the opm-integration-test project exists in it, and the matching per-project group exists; it exits non-zero on any failed assertion.
    • .gitignore: ignores the generated test/integration/clouds.yaml.
    • tox.ini: passes OS_CLIENT_CONFIG_FILE through to the create env.
  3. 1be0a3c Add Makefile integration targets

    • make integration (full provision/create/verify/teardown cycle), make integration-up (deploy and leave the cluster running for local debugging), make integration-down (delete the kind cluster).
  4. b470a8e Add Ansible pre/test playbooks for integration job

    • playbooks/pre-integration.yml: prepares an IPv6-only Zuul node — ensure-pip/ensure-pipenv/ensure-docker roles, the libldap2-dev/libsasl2-dev build deps the create env needs, and an accept_ra=2 sysctl fix so the node keeps honouring router advertisements once kind enables IPv6 forwarding.
    • playbooks/test-integration.yml: installs tox + openstacksdk into a throwaway venv and runs make integration from the checkout.
  5. bb61192 Add periodic-daily Zuul integration job

    • .zuul.yaml: defines openstack-project-manager-integration (ubuntu-noble, 2400s timeout) and adds it to periodic-daily only, so the heavy kind+Keystone job does not slow the check gate (which is left unchanged).
  6. c3db56a Document the integration test harness

    • test/integration/README.md: prerequisites, the make targets, the environment overrides, what verify.py asserts, and the ephemeral-credentials handling.

Notes / divergences from the issue

  • Keystone-only deploy: the issue asks to "deploy Keystone on kind (forge)". The harness applies a Keystone-only ControlPlane rather than a full forge control plane, keeping the job as light as feasible for the single happy-path assertion.
  • load-balancer_member workaround: the only accommodation to the production code is pre-creating roles create.py expects (member, load-balancer_member) on the bare Keystone. create.py is intentionally left untouched; the workaround lives entirely in deploy_keystone.sh.
  • CI placement: the job runs in periodic-daily, not the check pipeline, so PRs are not gated on the multi-minute kind+Keystone bring-up.
  • The verifier checks not just the project but also the per-project group of the same name that create.py provisions, giving the happy path a slightly stronger assertion than the issue's minimal "create a project".

Implemented by planwerk-agent 84299e2 with Claude:claude-opus-4-8

@berendt berendt force-pushed the implement/issue-293-keystone-integration-test branch 8 times, most recently from e9961b9 to 3c6f403 Compare June 30, 2026 15:58
berendt added 5 commits June 30, 2026 19:17
Stand up a kind cluster with a Keystone-only forge ControlPlane and
emit an ephemeral clouds.yaml for the admin cloud. forge is pinned to a
known-good commit and its own installer provides the pinned kind/kubectl
toolchain.

create.py indexes its role cache directly, so it aborts on a bare
Keystone that lacks load-balancer_member (Octavia normally provisions
it). Pre-create the roles create.py expects, keeping create.py itself
untouched.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Christian Berendt <berendt@osism.tech>
run.sh deploys Keystone, runs the real `tox -e create` path as admin
with --nodomain-name-prefix so the project is named exactly
opm-integration-test, then asserts it with verify.py. A trap tears down
the kind cluster on every exit path but only when this run created it,
dumping diagnostics first on failure.

verify.py is a standalone openstacksdk script -- not named test_*.py --
so `tox -e test` does not collect it. tox passes OS_CLIENT_CONFIG_FILE
through to the create env, and the generated clouds.yaml is gitignored.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Christian Berendt <berendt@osism.tech>
integration runs the full provision/create/verify/teardown cycle;
integration-up deploys Keystone and leaves the cluster running for local
debugging; integration-down deletes the kind cluster.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Christian Berendt <berendt@osism.tech>
pre-integration.yml prepares an IPv6-only Zuul node: the ensure-pip /
ensure-pipenv / ensure-docker roles, the libldap/libsasl build deps the
create env needs, and the accept_ra=2 sysctl fix that keeps RAs honoured
once kind enables IPv6 forwarding. The pinned kind/kubectl toolchain
comes from forge's install-test-deps, invoked by deploy_keystone.sh.

test-integration.yml installs tox and openstacksdk into a venv and runs
make integration from the checkout.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Christian Berendt <berendt@osism.tech>
openstack-project-manager-integration runs on ubuntu-noble with a 2400s
timeout. It is added to the check pipeline so the kind+Keystone job runs
on every change.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Christian Berendt <berendt@osism.tech>
@berendt berendt force-pushed the implement/issue-293-keystone-integration-test branch from 3c6f403 to 2693744 Compare June 30, 2026 17:19
@berendt berendt moved this from New to In progress in Human Board Jul 1, 2026
@berendt berendt force-pushed the implement/issue-293-keystone-integration-test branch from 2693744 to 19ae5ea Compare July 3, 2026 06:35
@berendt berendt marked this pull request as ready for review July 3, 2026 18:28
berendt added 2 commits July 3, 2026 20:34
Cover prerequisites, the make targets, the environment overrides, what
verify.py asserts, and the ephemeral-credentials handling.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Christian Berendt <berendt@osism.tech>
The Keystone integration job OOM-killed its Keystone pod (exit 137,
CrashLoopBackOff). Two causes: the CR left
infrastructure.{database,cache}.replicas unset, so the CRD default of 3
brought up a 3-node Galera MariaDB plus three Memcached pods; and Docker
Desktop's containerd ships LimitNOFILE=infinity, so uWSGI pre-allocated
multiple GiB per worker from the inherited ~1e9 fd limit and died within
seconds of spawning (more memory does not help).

Pin database.replicas and cache.replicas to 1 -- a single-instance
non-Galera MariaDB and one Memcached pod -- and bump FORGE_REF to
a828d6a, whose deploy-infra caps each kind node's containerd
RLIMIT_NOFILE to 1048576 before any workload is scheduled. Together they
let Keystone come up at its default 512Mi/2-worker footprint.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Christian Berendt <berendt@osism.tech>
@berendt berendt force-pushed the implement/issue-293-keystone-integration-test branch from 6659858 to 3ffd3c1 Compare July 3, 2026 18:35
forge made two changes the harness has to follow:

- database.storageSize is now a spec field (forge 1468088); pin it to
  512Mi like forge's kind Quick Start CR so the projected MariaDB stops
  requesting the 100Gi CRD default that a CI run never fills.

- Managed-mode ControlPlanes now default to Dynamic (engine-issued) DB
  credentials and the static per-ControlPlane seed was retired
  (forge 49fb87c..e3b90b5). The engine path only works once
  setup-database-tenant.sh has onboarded the per-tenant OpenBao
  database-engine connection and role, and deploy-infra runs that hook
  only when it applies the bundled CR itself (WITH_CONTROLPLANE_CR=true).
  This harness applies its own CR, so deploy_keystone.sh now mirrors the
  hook: wait for the projected MariaDB to be Ready, then run the
  idempotent tenant script with the OpenBao root token. Without it,
  DBCredentialsReady never turns True and the Ready wait times out.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Christian Berendt <berendt@osism.tech>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: In progress

Development

Successfully merging this pull request may close these issues.

Add integration test: deploy Keystone on kind (forge) + create a project via the admin account

2 participants