Skip to content

Conversation

@hannahlanzrath
Copy link
Contributor

@hannahlanzrath hannahlanzrath commented Feb 12, 2026

Summary

This PR introduces a change to the output repository push logic to support parallel result pushes from multiple machines / instances of the from the same ProjectRepo plus corresponding tests and test changes.

The new test simulates two independent clones of the same CADET-RDM project pushing results to the same remote output repository.

The root cause of the original failures was that all runs update shared files in the output repository main branch, especially log.tsv and the run_history directory. When multiple machines attempted to push these updates concurrently, Git rejected the push due to non-fast-forward conflicts.

Additionally, output branch names were previously only based on timestamp, project branch, and project commit hash, which could lead to branch name collisions in fast CI runs or concurrent executions. This is why a random suffix was added to the branch name.


Changes introduced in this PR

1. Integration test for parallel pushes

A new integration test was added that:

  • creates bare repositories to simulate remote project and output remotes
  • initializes a CADET-RDM project and pushes it
  • clones the project twice to simulate two independent machines
  • runs and pushes results from both clones
  • verifies that concurrent pushes succeed without manual conflict resolution

This test reproduces the parallel push issue and ensures the push logic works correctly.


2. Unique output branch names

Output branches now include a short random suffix:

<timestamp>_<project-branch>_<project-hash>_<6-hex>

This prevents branch name collisions.


3. Safe output repository push logic

The output push sequence was redesigned to safely handle parallel updates:

  • Result branches are pushed first, since they are uniquely named and do not conflict.
  • Updates to the output repository main branch are applied using:
fetch → reset to origin/main → apply update → commit → push

This ensures updates are always based on the latest remote state.


4. Retry logic for concurrent updates

If pushing the output main branch fails due to a non-fast-forward conflict, the operation is retried automatically. This allows concurrent pushes from multiple machines to eventually succeed without manual intervention.


5. Removal of implicit git pull during push

The previous implementation relied on git pull, which can fail when the remote has changed. The new implementation uses explicit fetch and controlled updates.

@hannahlanzrath hannahlanzrath force-pushed the feat/allow_parallel_runs_of_projects branch 12 times, most recently from 300ef33 to bac99fe Compare February 12, 2026 19:07
@hannahlanzrath hannahlanzrath force-pushed the feat/allow_parallel_runs_of_projects branch from bac99fe to 9044edd Compare February 12, 2026 19:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant