Skip to content

Prevent deploy from conflicting with in-progress pipeline runs #3

@retlehs

Description

@retlehs

Problem

When a deploy runs while a pipeline (wpcomposer pipeline) is in progress, there are potential race conditions:

  1. The deploy restarts the web server, which is harmless on its own
  2. But if the pipeline is in its build/deploy phase, both processes could call deploy.Promote and swap storage/repository/current concurrently
  3. Two concurrent builds could write conflicting rows to the builds table

The pipeline binary stays in memory (Linux keeps the old inode), so it won't crash — but the results could be unpredictable.

Possible solutions

  • Lock file — pipeline acquires a lock (e.g. storage/pipeline.lock) before starting; deploy task checks for it and skips the service restart or waits
  • Deploy skips restart if pipeline is activesystemctl is-active wpcomposer-pipeline check in the Ansible deploy task
  • Pipeline guards its own deploy step — only Promote if no other build was promoted since the pipeline started

Context

The pipeline runs on a timer via wpcomposer-pipeline.service (oneshot). Deploys are triggered manually via Ansible. Both use shared storage and the same SQLite database.

Metadata

Metadata

Assignees

No one assigned

    Labels

    infrastructureServer, deploy, CI/CD, and ops

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions