Filter out dataset-field and spec fields for correct download

[pooleycodes]

Description

Please replace this line with a brief description of the changes made.

What type of PR is this? (check all applicable)

• Refactor
• Feature
• Bug Fix
• Optimization
• Documentation Update

Related Tickets & Documents

• Closes #

QA Instructions, Screenshots, Recordings

Before

Before screenshot here

After

After screenshot here

Added/updated tests?

We encourage you to keep the code coverage percentage at 80% and above.

• Yes
• No, and this is why: Please replace this line with details on why tests have not been included
• I need help with writing tests

QA sign off

• Code has been checked and approved
• Design has been checked and approved
• Product and business logic has been checked and proved

[optional] Are there any post-deployment tasks we need to perform?

[optional] Are there any dependencies on other PRs or Work?

Summary by CodeRabbit

• Bug Fixes

  • Broadened system-field suppression so additional internal fields are now consistently excluded, improving data handling accuracy.

• Documentation

  • Added a clarifying comment to explain the filtering behaviour.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 728e73ac-1a37-4ca9-95be-b646dad6648b

📥 Commits

Reviewing files that changed from the base of the PR and between 672473a and d0bdc5b.

📒 Files selected for processing (1)

• src/middleware/common.middleware.js

---

Walkthrough

Expanded system-field suppression: filterOutSystemFields now also excludes fields whose datasetField value appears in the systemFields list; a clarifying comment was added to the filter logic. (50 words or fewer)

Changes

Cohort / File(s)	Summary
System field filter update src/middleware/common.middleware.js	filterOutSystemFields widened to exclude items where datasetField is in systemFields as well as where field is in systemFields. Added a clarifying comment explaining the filter logic. (+3/-2 lines)

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Suggested reviewers

• Ben-Hodgkiss

Poem

🐰 I hop through code with nimble paws,
I tuck away the system flaws,
Now dataset names get screened anew,
A tiny note to guide me through,
Hooray — the fields are tidy too!

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately describes the main change: filtering out dataset-field and spec fields from downloads, which aligns with the code modification that widens suppression criteria for system fields.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

• 📝 Generate docstrings (stacked PR)
• 📝 Generate docstrings (commit on current branch)

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch 2316-remove-organisation-entity-field-from-download-of-alternative-source-data

📝 Coding Plan

• Generate coding plan for human review comments

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

Tip

You can disable poems in the walkthrough.

Disable the reviews.poem setting to disable the poems in the walkthrough.

[github-actions]

Coverage Report

Status	Category	Percentage	Covered / Total
🔵	Lines	66.13%	6748 / 10203
🔵	Statements	66.13%	6748 / 10203
🔵	Functions	63.95%	275 / 430
🔵	Branches	79.23%	912 / 1151

File Coverage

File	Stmts	Branches	Functions	Lines	Uncovered Lines
Changed Files
src/middleware/common.middleware.js	65.88%	89.6%	37.09%	65.88%	31-40, 44, 58-78, 91-92, 94-95, 107-109, 113, 129-136, 150-154, 179-194, 209-210, 230-231, 253-267, 325-338, 344-366, 450-459, 475-488, 553, 562-564, 594-596, 644-685, 725-729, 734-737, 749-753, 781-795, 802-816, 822-830, 834-850, 919-921, 994-1001, 1016-1071, 1102-1103, 1106-1109, 1112-1125, 1150-1155, 1165-1168, 1177

Generated in workflow #1352 for commit d0bdc5b by the Vitest Coverage Report Action

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@src/middleware/common.middleware.js`:
- Line 469: The logging call currently dereferences req.dataset.dataset
unconditionally in logger.info, which can throw if req.dataset is undefined;
update the middleware to guard access to req.dataset (e.g., check if req.dataset
exists or use optional chaining when building the log payload) and log a safe
value (null/undefined or an empty object/string) for dataset when absent; ensure
you update the specific logger.info invocation that references
req.uniqueDatasetFields and req.dataset.dataset so unit tests no longer throw.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 86c5ebad-ef10-44aa-9264-af0dbad07bcf

📥 Commits

Reviewing files that changed from the base of the PR and between 2f0a224 and 672473a.

📒 Files selected for processing (1)

• src/middleware/common.middleware.js