Add script to compile Changelog from individual entries

.cursor/commands/pr-create.md

@@ -8,18 +8,25 @@ Use the template found in `.github/pull_request_template.md` to create a pull re
 - In the description and code changes sections, do not include information about code formatting or general code clean up changes.
 - Do not include running tests as part of the Steps to Confirm, those will run automatically as part of the PR.
 - The issue can usually be derived from the current branch name, it's a Jira ticket with prefix "ENG-" (eg. `Ticket [ENG-1234]`)
-- If `acli` is available, use `acli jira workitem view [key]` cli command to get more information about the issue.
-- If `gh` is available, use `gh pr create -d --title "[title]" --body "[body]"` cli command to create the PR in draft mode. The body should be in Markdown code format, patterned after the template.
+- If `acli` is available, use `acli jira workitem view [key]` cli command to get more information about the issue. Full permissions must be used for the acli credentials to work, even when in sandbox mode.
+- If `gh` is available, use `gh pr create -d --title "[title]" --body "[body]"` cli command to create the PR in draft mode. The body should be in Markdown code format, patterned after the template.  Full permissions must be used for the `gh` credentials to work, even when in sandbox mode.
 - If `gh` is not available, provide a link to a GitHub URL with the body of the PR template and title as query parameters.
   - The format is: `https://github.com/ethyca/fides/compare/main...{branch}?quick_pull=1&title={title}&body={body}`. Make this link clickable, not copyable. Take care to ensure the URL encoding doesn't get mangled in the link. Be sure to include backticks encoded as %60 instead of stripping them from the description.
     - Key parameters:
       - quick_pull=1 - Opens the PR creation form
       - title= - URL-encoded PR title
       - body= - URL-encoded PR description (supports markdown with %0A for newlines)
 
-## Add the changelog
-In addition to returning that markdown response, provide a concise, one-liner summary for the CHANGELOG no longer than about 85 characters or about 15 words max.
-- It's ok to suggest multiple entries for large PRs, but each entry should be a short one-liner.
-- Do not prefix the changelog entry with anything, start with a verb in the past tense (eg. "Added...", "Updated..." etc.)
-- Add the changelog to the appropriate spot in the CHANGELOG.md file in the root. Make sure it always ends up in the "Unreleased" section.
-- If `gh` is used to create the PR, include a link to it following the same format as other changelog entries. If a URL was created instead, do not put placeholders or anything for the PR link, it will be added manually once the PR is created.
+## Add the changelog entry
+In addition to returning that markdown response, create a changelog fragment file for the PR.
+
+- Create a YAML file in the `changelog/` directory (use `.yaml` or `.yml` extension)
+- Name the file using the PR number (e.g., `1234.yaml` or `pr-1234.yaml`) or a descriptive slug (e.g., `add-feature-name.yaml`)
+- Use `changelog/TEMPLATE.yaml` as a reference for the format
+- Provide a concise, one-liner description no longer than about 85 characters or about 15 words max
+- Start the description with a verb in the past tense (e.g., "Added...", "Updated...", "Fixed..." etc.)
+- Choose the appropriate `type` from: Added, Changed, Developer Experience, Deprecated, Docs, Fixed, Removed, Security
+- If `gh` is used to create the PR, include the PR number in the `pr` field. If a URL was created instead, leave `pr` empty or omit it (it can be added later)
+- Add `labels` if applicable: `["high-risk"]` for high-risk changes, `["db-migration"]` for database migrations, or `["high-risk", "db-migration"]` for both
+- The changelog entry will be automatically compiled into `CHANGELOG.md` during the release process
+- It's ok to suggest multiple fragment files for large PRs, but each entry should be a short one-liner

.github/workflows/check_changelog.yml

@@ -0,0 +1,89 @@
+name: Check Changelog Entry
+
+on:
+  pull_request:
+    types: [opened, synchronize, reopened]
+    branches:
+      - main
+
+jobs:
+  check-changelog:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check for changelog entry
+        id: check-changelog-files
+        uses: actions/github-script@v7
+        with:
+          script: |
+            const pr = context.payload.pull_request;
+
+            // Fetch all files changed in the PR
+            const files = await github.paginate(github.rest.pulls.listFiles, {
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              pull_number: pr.number,
+            });
+
+            // Check for changelog entry files (excluding TEMPLATE.yaml)
+            const changelogFiles = files.filter(file => {
+              const filename = file.filename;
+              // Check if file is in changelog/ directory
+              if (!filename.startsWith('changelog/')) {
+                return false;
+              }
+              // Check if it's a YAML file
+              if (!filename.endsWith('.yaml') && !filename.endsWith('.yml')) {
+                return false;
+              }
+              // Exclude TEMPLATE.yaml
+              if (filename === 'changelog/TEMPLATE.yaml' || filename === 'changelog/TEMPLATE.yml') {
+                return false;
+              }
+              // Only count added or modified files (not deleted)
+              return file.status === 'added' || file.status === 'modified';
+            });
+
+            if (changelogFiles.length === 0) {
+              core.setFailed(
+                '❌ No changelog entry file found!\n\n' +
+                'Please add a changelog entry file in the changelog/ directory.\n' +
+                'See changelog/README.md for instructions on creating changelog entries.\n'
+              );
+            } else {
+              const fileNames = changelogFiles.map(f => f.filename).join(', ');
+              console.log(`✅ Found changelog entry file(s): ${fileNames}`);
+              // Set output for validation step
+              core.setOutput('changelog_files', fileNames);
+            }
+
+      - name: Checkout repository
+        if: success()
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        if: success()
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.10"
+
+      - name: Install nox
+        if: success()
+        run: pip install nox>=2022
+
+      - name: Cache Nox virtual environment
+        if: success()
+        uses: actions/cache@v4
+        with:
+          path: .nox/
+          key: ${{ runner.os }}-nox-changelog-${{ hashFiles('noxfiles/changelog_nox.py') }}
+          restore-keys: |
+            ${{ runner.os }}-nox-changelog-
+
+      - name: Install Dev Requirements
+        if: success()
+        run: pip install -r dev-requirements.txt
+
+      - name: Validate changelog entries
+        if: success()
+        run: |
+          nox -s "changelog(validate)" -- --files "${{ steps.check-changelog-files.outputs.changelog_files }}"

changelog/README.md

@@ -0,0 +1,37 @@
+# Changelog Fragments
+
+This directory contains changelog entry fragments that are compiled into `CHANGELOG.md` during releases.
+
+## Creating an Entry
+
+1. Copy `TEMPLATE.yaml` to a new file (use `.yaml` or `.yml` extension)
+2. Name it using your PR number (e.g., `1234.yaml`) or a descriptive slug (e.g., `add-feature-name.yaml`)
+3. Fill in the required fields:
+   - `type`: One of: Added, Changed, Developer Experience, Deprecated, Docs, Fixed, Removed, Security
+   - `description`: Short description (start with past-tense verb, e.g., "Added...", "Fixed...")
+   - `pr`: PR number
+   - `labels`: Optional list: `["high-risk"]`, `["db-migration"]`, or both
+
+## Compiling Entries
+
+**Preview (dry run):**
+```sh
+nox -s "changelog(dry)" -- --release 2.77.0
+```
+
+**Compile and create release section:**
+```sh
+nox -s "changelog(write)" -- --release 2.77.0
+```
+
+**For patch releases (only specific PRs):**
+```sh
+nox -s "changelog(write)" -- --release 2.76.2 --prs 1234,5678
+```
+
+The script will:
+- Compile fragments into a new version section
+- Delete processed fragment files
+- Update compare links in `CHANGELOG.md`
+
+**Note:** The `write` action always requires the `--release VERSION` flag.

changelog/TEMPLATE.yaml

@@ -0,0 +1,7 @@
+# Copy this file and rename it (e.g., pr-number.yaml or feature-name.yaml)
+# Fill in the required fields and delete this comment block
+
+type: Added # One of: Added, Changed, Developer Experience, Deprecated, Docs, Fixed, Removed, Security
+description: Short description of the change
+pr: 1234 # PR number
+labels: [] # Optional: ["high-risk", "db-migration"]

changelog/add-changelog-script.yaml

@@ -0,0 +1,4 @@
+type: Developer Experience
+description: Added a nox script to compile changelog fragments into CHANGELOG.md
+pr: 7177 # PR number
+labels: [] # Optional: ["high-risk", "db-migration"]

dev-requirements.txt

@@ -16,6 +16,7 @@ pytest-mock==3.14.0
 pytest-rerunfailures==14.0
 pytest-xdist==3.6.1
 pytest==7.2.2
+pyyaml==6.0.1
 requests-mock==1.10.0
 setuptools>=64.0.2
 sqlalchemy-stubs==0.4

docs/fides/docs/development/releases.md

@@ -20,15 +20,55 @@ When releasing, a new tag is created on the release branch and the release proce
 
 In the case of patches, a branch is created from the relevant tag. Commits are then cherry-picked into this branch, and a new patch version tag is created.
 
+## Changelog Entries
+
+To avoid merge conflicts in `CHANGELOG.md`, we use a fragment-based system where each PR adds a YAML file describing the change.
+
+### Creating a Changelog Entry
+
+When creating a PR, add a changelog entry file:
+
+1. Copy `changelog/TEMPLATE.yaml` to a new file in the `changelog/` directory
+2. Name the file using your PR number (e.g., `7137.yaml`) or a descriptive slug (e.g., `add-gpc-fields.yaml`)
+3. Fill in the required fields:
+   - `type`: One of: Added, Changed, Developer Experience, Deprecated, Docs, Fixed, Removed, Security
+   - `description`: Short description of the change (without the leading `-`)
+   - `pr`: PR number (optional if not yet known)
+   - `labels`: Optional list of labels: `high-risk`, `db-migration`
+
+Example (`changelog/7137.yaml`):
+
+```yaml
+type: Added
+description: Added editable GPC translation fields to privacy experience configuration
+pr: 7137
+labels: []
+```
+
+The changelog entry will be automatically compiled into `CHANGELOG.md` during the release process.
+
 ## Release Steps
 
-We use GitHub’s `release` feature to tag releases that then get automatically deployed to DockerHub and PyPi via GitHub Actions. We also use a `CHANGELOG.md` to mitigate surprises to our users and help them plan updates accordingly. The release steps are as follows:
+We use GitHub's `release` feature to tag releases that then get automatically deployed to DockerHub and PyPi via GitHub Actions. We also use a `CHANGELOG.md` to mitigate surprises to our users and help them plan updates accordingly. The release steps are as follows:
 
 ### Major and Minor
 
-1. Open a PR that is titled the version of the release (i.e. `1.6.0`)
-    * Rename the `Unreleased` section of `CHANGELOG.md` to the new version number and put a date next to it
-    * Update the `compare` links for both the new version and for the new `Unreleased` section
+1. Compile changelog fragments into `CHANGELOG.md`:
+   ```sh
+   # Preview what will be generated (dry run)
+   nox -s "changelog(dry)"
+
+   # Compile fragments and finalize for release
+   nox -s "changelog(write)" -- --release 2.76.0
+   ```
+   This will:
+   - Compile all YAML fragments from `changelog/` into the Unreleased section
+   - Create a new version section with all Unreleased content
+   - Leave the `Unreleased` section empty at the top (ready for new entries)
+   - Update the `compare` links for both the new version and the `Unreleased` section
+   - Delete the processed fragment files
+
+1. Open a PR that is titled the version of the release (i.e. `2.76.0`) with the updated `CHANGELOG.md`
 1. Once approved, merge the PR
 1. Create a new release, ensuring that the last PR to get merged is the aforementioned `CHANGELOG.md` update PR
 1. Add the new version as the tag (i.e. `1.6.0`)
@@ -131,10 +171,23 @@ It may be necessary for a patch release to contain only select commits to the `m
     git cherry-pick <commit>...
     ```
 
-1. Copy the `Unreleased` section of `CHANGELOG.md` and paste above the release being patched
-    1. Rename `Unreleased` to the new version number and put a date next to it
-    1. Cut and paste the documented changes that are now included in the patch release to the correct section
-    1. Commit these changes
+1. Compile changelog fragments for the patch release:
+   ```sh
+   # Preview what will be generated (dry run)
+   nox -s "changelog(dry)" -- --prs 1234,5678
+
+   # Compile fragments and finalize for patch release (only selected PRs)
+   nox -s "changelog(write)" -- --release 1.2.4 --prs 1234,5678
+   ```
+   This will:
+   - Compile only the YAML fragments matching the specified PR numbers (comma-separated list)
+   - Create a new version section from those entries
+   - Leave the `Unreleased` section empty at the top
+   - Delete only the processed fragment files (matching PRs)
+   - Leave other fragment files in `changelog/` for future releases
+   - Update compare links appropriately
+
+   **Note:** For patch releases, you typically only want to include specific PRs. Use the `--prs` flag to specify which PR numbers to include. Fragments without PR numbers will be skipped when using this flag.
 1. Open a pull request to incorporate any cherry-picked commits and the `CHANGELOG.md` changes into the release branch
     1. Set the base branch of this pull request to the release branch
     1. Once approved, merge the pull request into the release branch

noxfile.py

@@ -14,6 +14,7 @@
 
 sys.path.append("noxfiles")
 # pylint: disable=unused-wildcard-import, wildcard-import, wrong-import-position
+from changelog_nox import *
 from ci_nox import *
 from dev_nox import *
 from docker_nox import *