Feat/sensor data source filtering by BelhsanHmida · Pull Request #2083 · FlexMeasures/flexmeasures

BelhsanHmida · 2026-04-02T07:31:11Z

Description

Add account filtering to GET /api/v3_0/sensors/<id>/data so sensor data can be filtered by the account linked to a data source.
Keep source as the existing data-source ID filter for backward compatibility.
Fix the OpenAPI/Swagger docs for the endpoint so the actual GET query parameters are exposed correctly.
Clarify the API docs for supported source filtering on this endpoint.
Add test coverage for account-based source filtering.
Keep the GET sensor data query fields shared between request parsing and OpenAPI docs via a mixin.
Add source_account_ids to shared belief-search schemas so they stay aligned with Sensor.search_beliefs(...).
Updated documentation/changelog.rst

Look & Feel

Swagger/OpenAPI for GET /api/v3_0/sensors/<id>/data now shows resolution, source, and account as query parameters.

Example request:

GET /api/v3_0/sensors/1/data?start=2021-05-02T00:00:00+02:00&duration=PT1H20M&unit=m%C2%B3%2Fh&resolution=PT20M&account=2

How to test

Run:
pytest flexmeasures/api/v3_0/tests/test_sensor_data.py

Also run:
pytest flexmeasures/data/schemas/tests/test_input_schema.py

Optionally inspect Swagger/OpenAPI and verify GET /api/v3_0/sensors/<id>/data shows:

resolution
source
account

Related Items

Closes Improve source filtering in GET endpoint for sensor data #2051
Depends on Add account_id to data sources #2047

Sign-off

I agree to contribute to the project under Apache 2 License.
To the best of my knowledge, the proposed patch is not based on code under GPL or other license that is incompatible with FlexMeasures

- Add account_id column (FK to account.id, nullable) to DataSource - Add account relationship (backref 'data_sources' on Account) - Update __table_args__ UniqueConstraint to include account_id - Set self.account from user.account in __init__ when user is provided

- Add account_id column (nullable, FK to account.id) - Data migration: populate account_id from user's account where user_id IS NOT NULL - Update UniqueConstraint to include account_id column - down_revision: 8b62f8129f34

…backfill

…ng sensor data Context: - DataSource.__init__ now sets self.account = user.account when a user is provided - The POST sensor data endpoint calls get_or_create_source(current_user) - Need to verify account_id is populated correctly on the resulting data source Change: - Added test_post_sensor_data_sets_account_id_on_data_source - Posts sensor data as test_supplier_user_4@seita.nl - Asserts data_source.account_id == user.account_id

… changelog entry Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com> Agent-Logs-Url: https://github.com/FlexMeasures/flexmeasures/sessions/7969297e-88c2-4ace-99c9-ec74109f03c8

… unflushed users Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com> Agent-Logs-Url: https://github.com/FlexMeasures/flexmeasures/sessions/7969297e-88c2-4ace-99c9-ec74109f03c8

…d invariants Context: - PR #2058 added account_id FK to DataSource table - No migration checklist existed for reviewing/writing Alembic migrations - New domain invariant (user-type DataSources now have account_id) was undocumented - Agent failed to self-update during the PR session Change: - Added Alembic migration checklist with safe backfill pattern - Added DataSource entity to Domain Knowledge with account_id invariant - Documented invariants #5 and #6 for DataSource UniqueConstraint and account_id - Added Lessons Learned section with PR #2058 case study

…essons Context: - PR #2058 added a test for account_id on DataSource after API POST - No guidance existed for testing data source properties after API calls - Agent failed to self-update during the PR session Change: - Added 'Testing DataSource Properties After API Calls' section with pattern - Added Lessons Learned section documenting PR #2058 self-improvement failure

…Coordinator failure Context: - PR #2058 session: Coordinator was not invoked (3rd session in a row) - API Specialist was not engaged despite endpoint behavior change - No agents updated their instructions (same failure as 2026-02-06) Change: - Added 'Agent Selection Checklist' mapping code change types to required agents - Documented PR #2058 as lessons learned (3 distinct failures) - Reinforced that Coordinator must ALWAYS be last agent in every session - Clarified that endpoint behavior changes → API Specialist must be engaged

…lure pattern Context: - PR #2058 repeated the same 3 failures from sessions 2026-02-06 and 2026-02-08 - Coordinator not invoked, no agent self-updates, API Specialist not engaged - Same failures now documented 3 times in Review Lead instructions with no change in behavior Change: - Added 'Additional Pattern Discovered (2026-03-24)' with root cause analysis - Documented missed API Specialist engagement for endpoint behavior change - Flagged governance escalation if Coordinator invocation fails a 4th time - Noted code observation: or-pattern for account_id and empty Initial plan commit

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com> Agent-Logs-Url: https://github.com/FlexMeasures/flexmeasures/sessions/7969297e-88c2-4ace-99c9-ec74109f03c8

…a data source before POSTing sensor data Signed-off-by: F.N. Claessen <claessen@seita.nl>

…l time (UTC+1) Signed-off-by: F.N. Claessen <claessen@seita.nl>

Signed-off-by: F.N. Claessen <claessen@seita.nl>

…lineage preservation Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com> Agent-Logs-Url: https://github.com/FlexMeasures/flexmeasures/sessions/004f0bb9-133d-45c1-9e06-2fedbf25b258

…n delete tests Context: - PR #2058 dropped FK constraints on DataSource.user_id and account_id to preserve lineage when users/accounts are deleted - The new tests only checked one of the two lineage fields per test Change: - test_delete_user: also assert data_source.account_id is preserved after user deletion (user deletion should not nullify account_id) - test_delete_account: also save and assert data_source.user_id per data source; account deletion internally calls delete_user so both user_id and account_id lineage must survive the full deletion chain

…eservation Context: - The existing entry only mentioned adding account_id to data_source - PR #2058 also drops FK constraints on user_id and account_id to preserve historical references when users/accounts are deleted (data lineage) Change: - Changelog now states both the column addition AND the behavior change: 'historical user/account IDs are no longer nullified when users or accounts are deleted'

…s lesson Context: - PR #2058 dropped FK constraints on DataSource.user_id and account_id for data lineage preservation (a deliberate design decision) - The original changelog entry omitted the FK drop and behavior change - No existing agent pattern covered 'intentional FK removal for lineage' Change: - Added checklist for no-FK lineage pattern (passive_deletes, tests, changelog) - Added lesson: changelog must describe behavior changes, not just schema changes - Documented FK drop design decision for future reviewers

…plementation The note previously described the 'or' anti-pattern as a concern, but the final code already uses 'if account_id is not None' correctly. Clarified to point to the implemented solution.

Signed-off-by: F.N. Claessen <claessen@seita.nl>

…d-to-data-source

…either is deleted Signed-off-by: F.N. Claessen <claessen@seita.nl>

Signed-off-by: F.N. Claessen <claessen@seita.nl>

…-data-source-filtering

Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>

…7450113f6) Remove the intermediate a1b2c3d4e5f6 migration that was dropping the account_id FK (only created within this PR) and the user_id FK. Instead: - Don't create the account_id FK in 9877450113f6 in the first place - Absorb the user_id FK drop into 9877450113f6 - Update b2c3d4e5f6a7 to chain off 9877450113f6 directly - Delete a1b2c3d4e5f6 Agent-Logs-Url: https://github.com/FlexMeasures/flexmeasures/sessions/4d9d8a92-d4fd-4635-bebf-1965c35667b3 Co-authored-by: nhoening <1042336+nhoening@users.noreply.github.com>

Copilot

Pull request overview

Adds support for filtering sensor data by the account linked to its data source, and updates the API documentation/specs so query parameters are correctly exposed in Swagger/OpenAPI.

Changes:

Add account-based data-source filtering to sensor data retrieval (plumbed through belief search/query utilities).
Update OpenAPI docstrings and generated OpenAPI JSON to expose resolution and source-related filters as query parameters.
Add/adjust documentation and tests around the new filtering behavior.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
flexmeasures/ui/static/openapi-specs.json	Updates generated OpenAPI output to show additional query parameters for GET sensor data.
flexmeasures/data/schemas/reporting/init.py	Extends reporting search config schema with `source_account_ids`.
flexmeasures/data/schemas/io.py	Extends reporter input schema with `source_account_ids`.
flexmeasures/data/queries/utils.py	Adds account-based source criterion and threads it through source-criteria building.
flexmeasures/data/models/time_series.py	Threads `source_account_ids` through `search_beliefs` / `TimedBelief.search` calls.
flexmeasures/api/v3_0/tests/test_sensor_data.py	Adds test coverage for account-based source filtering on GET sensor data.
flexmeasures/api/v3_0/sensors.py	Updates endpoint docstring to reference a query schema so Swagger exposes query parameters.
flexmeasures/api/common/schemas/sensor_data.py	Introduces shared GET-filter schema mixin; adds account-based filtering and a dedicated query schema for docs.
documentation/changelog.rst	Adds changelog entry describing the feature/doc improvements.
documentation/api/notation.rst	Documents source filtering options specific to GET sensor data.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

flexmeasures/api/common/schemas/sensor_data.py

flexmeasures/data/schemas/io.py

flexmeasures/data/schemas/reporting/__init__.py

flexmeasures/data/queries/utils.py

flexmeasures/ui/static/openapi-specs.json

flexmeasures/api/v3_0/tests/test_sensor_data.py

flexmeasures/api/v3_0/sensors.py

documentation/api/notation.rst

Context: - shared search schemas were deserializing source_account_ids to Account objects - downstream source filtering expects integer account ids for SQL criteria

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Mohamed Belhsan Hmida <149331360+BelhsanHmida@users.noreply.github.com>

Signed-off-by: F.N. Claessen <claessen@seita.nl>

…d-to-data-source

Signed-off-by: F.N. Claessen <claessen@seita.nl>

…main) Change 9877450113f6.down_revision from 8b62f8129f34 to e26d02ed1621 so the PR's two real migrations (9877450113f6, b2c3d4e5f6a7) chain linearly after the tip of the main branch, making the two merge-only files (b2e07f0dafa1 and 2c6a4c8127c1) completely obsolete. Delete b2e07f0dafa1_merge.py and 2c6a4c8127c1_merge.py. Agent-Logs-Url: https://github.com/FlexMeasures/flexmeasures/sessions/94c3070e-a701-4a97-a895-9ea1064e3c20 Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>

Signed-off-by: F.N. Claessen <claessen@seita.nl>

…ource' into feat/sensor-data-source-filtering

Copilot AI and others added 30 commits March 24, 2026 22:04

Initial plan

4b37779

migration: add account_id to data_source table (9877450113f6)

6745921

- Add account_id column (nullable, FK to account.id) - Data migration: populate account_id from user's account where user_id IS NOT NULL - Update UniqueConstraint to include account_id column - down_revision: 8b62f8129f34

migration: replace N+1 loop with single UPDATE...FROM for account_id …

78fb04f

…backfill

migration: use correlated subquery for account_id data migration; add…

85778e4

… changelog entry Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com> Agent-Logs-Url: https://github.com/FlexMeasures/flexmeasures/sessions/7969297e-88c2-4ace-99c9-ec74109f03c8

data_sources: fix account_id assignment to avoid SAWarning and handle…

06bbec9

… unflushed users Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com> Agent-Logs-Url: https://github.com/FlexMeasures/flexmeasures/sessions/7969297e-88c2-4ace-99c9-ec74109f03c8

data_sources: use explicit is not None check for account_id assignment

4191ae3

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com> Agent-Logs-Url: https://github.com/FlexMeasures/flexmeasures/sessions/7969297e-88c2-4ace-99c9-ec74109f03c8

feat: improve test by using a user that is not already registered as …

38db656

…a data source before POSTing sensor data Signed-off-by: F.N. Claessen <claessen@seita.nl>

fix: set approximate create datetime based on git info, assuming loca…

67eba6e

…l time (UTC+1) Signed-off-by: F.N. Claessen <claessen@seita.nl>

style: flake8

b7a4a5c

Signed-off-by: F.N. Claessen <claessen@seita.nl>

feat: add migration check to architecture agent instructions

44cff24

Signed-off-by: F.N. Claessen <claessen@seita.nl>

docs: add db upgrade warning and update agent instructions accordingly

95653f1

Signed-off-by: F.N. Claessen <claessen@seita.nl>

data_sources: drop FK constraints on user_id and account_id for data …

4e22e43

…lineage preservation Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com> Agent-Logs-Url: https://github.com/FlexMeasures/flexmeasures/sessions/004f0bb9-133d-45c1-9e06-2fedbf25b258

coordinator: clarify account_id fallback pattern references actual im…

ebfba1d

…plementation The note previously described the 'or' anti-pattern as a concern, but the final code already uses 'if account_id is not None' correctly. Clarified to point to the implemented solution.

feat: add test covering CLI command to delete user

c64b4ee

Signed-off-by: F.N. Claessen <claessen@seita.nl>

Merge remote-tracking branch 'origin/main' into copilot/add-account-i…

531cb74

…d-to-data-source

feat: let audit log entries retain the account ID and user IDs after …

57c1f8e

…either is deleted Signed-off-by: F.N. Claessen <claessen@seita.nl>

docs: rewrite changelog entry

e1d6cdc

Signed-off-by: F.N. Claessen <claessen@seita.nl>

fix: dropped foreign key name

7ec2cb1

Signed-off-by: F.N. Claessen <claessen@seita.nl>

fix: when deleting a user, still log the ID of the deleted user

0fefb2e

Signed-off-by: F.N. Claessen <claessen@seita.nl>

feat: allow tying a newly CLI-created data source to an account

6bcf9c6

Signed-off-by: F.N. Claessen <claessen@seita.nl>

fix: add missing drop_constraints

faa876f

Signed-off-by: F.N. Claessen <claessen@seita.nl>

BelhsanHmida added 10 commits April 7, 2026 02:29

Merge branch 'copilot/add-account-id-to-data-source' into feat/sensor…

d0209b3

…-data-source-filtering

docs: fix pr number in changelog entry

3adfed6

Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>

feat: rename sensor data account filter parameter

c865dbb

feat: align sensor data docs with account filter parameter

2734a97

test: update sensor data account filter request

62de0cb

feat: regenerate OpenAPI specs for account filter parameter

5f57b72

feat: use kwargs in source criteria call

5457e44

feat: add metadata to sensor data query schema

20cca6a

Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>

feat: share sensor data query filters in a mixin

28c0d94

Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>

feat: add source account ids to search schemas

9b61895

Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>

BelhsanHmida requested a review from Flix6x April 7, 2026 08:24

Flix6x mentioned this pull request Apr 8, 2026

Improve source filtering in GET endpoint for sensor data #2051

Open

BelhsanHmida linked an issue Apr 8, 2026 that may be closed by this pull request

Improve source filtering in GET endpoint for sensor data #2051

Open

BelhsanHmida requested a review from Copilot April 12, 2026 18:56

Copilot started reviewing on behalf of BelhsanHmida April 12, 2026 18:56 View session

Copilot AI reviewed Apr 12, 2026

View reviewed changes

BelhsanHmida and others added 2 commits April 13, 2026 01:52

feat: load source account ids as integers

e26d199

Context: - shared search schemas were deserializing source_account_ids to Account objects - downstream source filtering expects integer account ids for SQL criteria

Update flexmeasures/data/queries/utils.py

880efae

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Mohamed Belhsan Hmida <149331360+BelhsanHmida@users.noreply.github.com>

Flix6x added the API label Apr 13, 2026

Flix6x added this to the 0.32.0 milestone Apr 13, 2026

Flix6x and others added 8 commits April 13, 2026 11:48

docs: make this lengthy comment only once

1d39eae

Signed-off-by: F.N. Claessen <claessen@seita.nl>

docs: add explanation for defining foreign keys in the db.relationship

2503d95

Signed-off-by: F.N. Claessen <claessen@seita.nl>

Merge remote-tracking branch 'origin/main' into copilot/add-account-i…

40ab6ae

…d-to-data-source

fix: merge db revisions

d43b41d

Signed-off-by: F.N. Claessen <claessen@seita.nl>

style: flake8

74ed3dc

Signed-off-by: F.N. Claessen <claessen@seita.nl>

style: punctuation / double spaces

e7139da

Signed-off-by: F.N. Claessen <claessen@seita.nl>

Merge remote-tracking branch 'origin/copilot/add-account-id-to-data-s…

d9a7304

…ource' into feat/sensor-data-source-filtering

Base automatically changed from copilot/add-account-id-to-data-source to main April 13, 2026 11:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feat/sensor data source filtering#2083

Feat/sensor data source filtering#2083
BelhsanHmida wants to merge 69 commits intomainfrom
feat/sensor-data-source-filtering

BelhsanHmida commented Apr 2, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

BelhsanHmida commented Apr 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Look & Feel

How to test

Related Items

Sign-off

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

BelhsanHmida commented Apr 2, 2026 •

edited

Loading