Skip to content

Pipeline Transition Plan #4381

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
paulmedynski wants to merge 1 commit into
base: main
Choose a base branch
from
+106 −0
Open

Pipeline Transition Plan #4381

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
106 changes: 106 additions & 0 deletions eng/pipelines/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
# Pipeline Layout Transition

This directory is transitioning from a legacy layout to a new structure. Both layouts coexist during
the migration.

## Target Layout

```text
eng/pipelines/

@cheenamalhotra cheenamalhotra Jun 23, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We also need docs generation, localization pipelines to be included.

├── ci/ # CI pipelines (package, test, stress, kerberos, …)
│ ├── kerberos/

@cheenamalhotra cheenamalhotra Jun 23, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would recommend moving kerberos and stress pipelines to test\ directory. I would imagine test to contain all testing related yamls (expandable in future). We don't need separate directories for 1 yaml file each.

│ │ └── sqlclient-ci-kerberos-pipeline.yml
│ ├── package/
│ │ └── sqlclient-ci-package-pipeline.yml
│ ├── shared/ # Templates shared across CI pipelines only

@cheenamalhotra cheenamalhotra Jun 23, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why does CI need its own shared directory? Can we aim for only 1 shared directory to avoid duplication of CI artifacts with PR/test/official pipelines, etc.

│ ├── stress/
│ │ └── sqlclient-ci-stress-pipeline.yml
│ └── test/
│ └── sqlclient-ci-test-pipeline.yml
├── github/ # GitHub sync pipeline
│ └── sqlclient-github-sync-pipeline.yml
├── official/ # Official / release pipelines (renamed from onebranch/)
│ ├── sqlclient-official-pipeline.yml
│ └── sqlclient-non-official-pipeline.yml
├── pr/ # Pull-request validation pipelines
│ └── sqlclient-pr-pipeline.yml
├── shared/ # Templates shared across pr/, ci/, etc
└── README.md
```

Every leaf directory above (each pipeline group with entry files, plus the `shared/` directories)
also contains the standard template subdirectories, omitted above for brevity:
Comment on lines +31 to +32

```text
<group>/
├── …entry pipeline yml(s)… # omitted for shared/ directories
├── jobs/
├── scripts/
├── stages/
├── steps/
└── variables/
```

### Naming Convention

Pipeline entry files follow the pattern:

```text
sqlclient-<pipeline-name>-pipeline.yml
```

Each directory may contain multiple entry files when there are distinct pipelines in the same
logical group (e.g. `official/` has separate official and non-official pipelines). In `ci/`, each
pipeline (package, test, stress, kerberos) lives in its own subdirectory with its own `jobs/`,

@benrr101 benrr101 Jun 19, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I personally don't think this is necessary, but maybe there will be a mess of mixed up steps, etc without it

@cheenamalhotra cheenamalhotra Jun 23, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree, multiple directories can be avoided. Things should be parameterized to re-use core execution scripts that when modified/updated, updates all pipelines together.

`stages/`, `steps/`, `variables/`, and `scripts/`.

### `shared/` Directories

Templates that are consumed by multiple top-level pipelines live in the top-level
`shared/`. A template that is only used within a single pipeline group stays in that group's own

@cheenamalhotra cheenamalhotra Jun 23, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's debatable for CI - since most of the CI steps are similar to PR pipeline, if you can parameterize them well.

subdirectory, for example `ci/shared/`

## Legacy Layout (being removed)

```text
eng/pipelines/
├── dotnet-sqlclient-ci-core.yml
├── dotnet-sqlclient-ci-package-reference-pipeline.yml
├── dotnet-sqlclient-ci-project-reference-pipeline.yml
├── sqlclient-pr-package-ref-pipeline.yml
├── sqlclient-pr-project-ref-pipeline.yml
├── github-sync-pipeline.yml # ⚠ Legacy — moving to github/
├── common/ # ⚠ Legacy — shared templates
│ ├── templates/

@benrr101 benrr101 Jun 19, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think its worth noting that the files in this folder should be going away entirely.

In other words:

  • common/templates -> old, old "common" yaml (that really wasn't common at all)
  • common/[everything else] -> "legacy" common yaml, that can be moved directly to shared

│ ├── steps/
│ └── variables/
├── jobs/ # ⚠ Legacy — job templates
├── stages/ # ⚠ Legacy — stage templates
├── libraries/ # ⚠ Legacy — variable templates
├── kerberos/ # ⚠ Legacy — moving to ci/
├── stress/ # ⚠ Legacy — moving to ci/
└── onebranch/ # ⚠ Legacy — renaming to official/
```

## Migration Status

@cheenamalhotra cheenamalhotra Jun 23, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please include other remaining pipelines as well.

Also, could you add phase-based approach as well? What is the step-wise approach here to get this migration completed?

And finally, also provide clarity about how much progress has been made for the items 'In Progress'.

e.g. Official pipelines need to run all tests as package pipeline too, should be captured as one of the pending tasks for the official/non-official pipelines.


| Legacy Location | Target Location | Status |
| --------------- | --------------- | ------ |
| Root entry YAMLs (`dotnet-sqlclient-ci-*.yml`, `sqlclient-pr-*.yml`) | `ci/`, `pr/` | In progress |
| `github-sync-pipeline.yml` | `github/` | Not started |
| `common/` | `shared/` (most files to be deleted) | In progress |
| `jobs/` | `pr/jobs/`, `ci/jobs/`, `shared/jobs/` | In progress |
| `stages/` | `pr/stages/`, `ci/stages/`, `shared/stages/` | In progress |
| `libraries/` | `shared/variables/` | Not started |
| `kerberos/` | `ci/` | Not started |
| `stress/` | `ci/` | Not started |
| `onebranch/` | `official/` | Not started |

## Guidelines

1. **New pipelines** go in `pr/`, `ci/`, or `official/`.
2. **New shared templates** go in `shared/`.
3. **Do not add files** to `common/`, root-level `jobs/`, `stages/`, `libraries/`, or `onebranch/`.
4. When moving a template, update all `@self` references that point to it.
5. Legacy files will be deleted once all references are migrated. Most legacy shared templates are
not expected to survive the migration.
Loading