Skip to content

feat: add account_id filter for sources in search_beliefs#2065

Draft
Copilot wants to merge 60 commits intomainfrom
copilot/add-account-id-filter-sources
Draft

feat: add account_id filter for sources in search_beliefs#2065
Copilot wants to merge 60 commits intomainfrom
copilot/add-account-id-filter-sources

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented Mar 26, 2026

Sensor.search_beliefs and related methods could filter sources by id or type, but not by the account that owns the source. This adds account_id as a first-class filter parameter throughout the belief query stack.

Core changes

  • data/queries/utils.py: New source_account_criterion() function; account_id: int | list[int] | None added to get_source_criteria(). Accepts both integer IDs and Account model instances. Empty list [] at the search_beliefs level matches nothing (consistent with SQL IN () semantics).
  • data/models/time_series.py: account_id parameter threaded through Sensor.search_beliefs() and TimedBelief.search()
  • data/models/generic_assets.py: account_id parameter added to GenericAsset.search_beliefs()
  • data/schemas/account.py: New AccountIdOrListField class that natively accepts int | list[int], validates each ID against the DB, rejects empty lists, and always deserializes to list[Account]. AccountIdField._deserialize updated to use @with_appcontext_if_needed() instead of @with_appcontext, allowing the field to be used outside CLI contexts. The normalization is encapsulated in the field definition (rather than a schema-level @pre_load hook) so future OpenAPI generation can emit a oneOf schema for it.
  • data/schemas/reporting/__init__.py: account_id = AccountIdOrListField() added to BeliefsSearchConfigSchema — enables account filtering in reporter/scheduler/forecaster configurations. Both 42 and [42] are accepted; empty lists are rejected at the field level.
  • data/schemas/io.py: Same field added to Input schema, which is what PandasReporter and AggregatorReporter actually deserialize for their input list entries.

Usage

# Filter beliefs to only those from sources owned by a specific account
bdf = sensor.search_beliefs(account_id=42)

# Or multiple accounts
bdf = sensor.search_beliefs(account_id=[42, 99])

# In a reporter input config (both forms accepted)
{"sensor": 1, "source_types": ["user"], "account_id": 42}
{"sensor": 1, "source_types": ["user"], "account_id": [42, 99]}

Note: account_id filtering only matches data sources with an explicit account association. DataSources created by reporters, schedulers, and forecasters have account_id=None and will not be matched by this filter — it is most useful for filtering input data by the account of the user who recorded it.

Original prompt

This section details on the original issue you should resolve

<issue_title>Reporters/schedulers/forecasters can filter by source account</issue_title>
<issue_description>Depends on #2047

These actuators can filter by data source id/type, but we most likely would like to filter by the account of the source (as well, probably in combination with type).

  • add the account_id parameter for filtering sources (search_beliefs)
  • enhance/create a test</issue_description>

<agent_instructions>- Create a PR to tackle issue #2046, using a new branch branched off of copilot/add-account-id-to-data-source (using that as the base branch for the PR).

  • Prompt subagents while reminding them to update their own instruction files, making atomic commits and running precommit hooks.
  • Use the architecture specialist to create the search filter.
  • Use the test specialist to create the tests.
  • Run the documentation specialist to update documentation, e.g. regarding forecasting, scheduling, reporting and plugins.
  • Run the coordinator.</agent_instructions>

Comments on the Issue (you are @copilot in this section)

Custom agent used: review-lead
Orchestrates team of specialist agents for comprehensive code reviews and synthesizes their findings into unified recommendations


📱 Kick off Copilot coding agent tasks wherever you are with GitHub Mobile, available on iOS and Android.

Copilot AI and others added 23 commits March 24, 2026 22:04
- Add account_id column (FK to account.id, nullable) to DataSource
- Add account relationship (backref 'data_sources' on Account)
- Update __table_args__ UniqueConstraint to include account_id
- Set self.account from user.account in __init__ when user is provided
- Add account_id column (nullable, FK to account.id)
- Data migration: populate account_id from user's account where user_id IS NOT NULL
- Update UniqueConstraint to include account_id column
- down_revision: 8b62f8129f34
…ng sensor data

Context:
- DataSource.__init__ now sets self.account = user.account when a user is provided
- The POST sensor data endpoint calls get_or_create_source(current_user)
- Need to verify account_id is populated correctly on the resulting data source

Change:
- Added test_post_sensor_data_sets_account_id_on_data_source
- Posts sensor data as test_supplier_user_4@seita.nl
- Asserts data_source.account_id == user.account_id
…d invariants

Context:
- PR #2058 added account_id FK to DataSource table
- No migration checklist existed for reviewing/writing Alembic migrations
- New domain invariant (user-type DataSources now have account_id) was undocumented
- Agent failed to self-update during the PR session

Change:
- Added Alembic migration checklist with safe backfill pattern
- Added DataSource entity to Domain Knowledge with account_id invariant
- Documented invariants #5 and #6 for DataSource UniqueConstraint and account_id
- Added Lessons Learned section with PR #2058 case study
…essons

Context:
- PR #2058 added a test for account_id on DataSource after API POST
- No guidance existed for testing data source properties after API calls
- Agent failed to self-update during the PR session

Change:
- Added 'Testing DataSource Properties After API Calls' section with pattern
- Added Lessons Learned section documenting PR #2058 self-improvement failure
…Coordinator failure

Context:
- PR #2058 session: Coordinator was not invoked (3rd session in a row)
- API Specialist was not engaged despite endpoint behavior change
- No agents updated their instructions (same failure as 2026-02-06)

Change:
- Added 'Agent Selection Checklist' mapping code change types to required agents
- Documented PR #2058 as lessons learned (3 distinct failures)
- Reinforced that Coordinator must ALWAYS be last agent in every session
- Clarified that endpoint behavior changes → API Specialist must be engaged
…lure pattern

Context:
- PR #2058 repeated the same 3 failures from sessions 2026-02-06 and 2026-02-08
- Coordinator not invoked, no agent self-updates, API Specialist not engaged
- Same failures now documented 3 times in Review Lead instructions with no change in behavior

Change:
- Added 'Additional Pattern Discovered (2026-03-24)' with root cause analysis
- Documented missed API Specialist engagement for endpoint behavior change
- Flagged governance escalation if Coordinator invocation fails a 4th time
- Noted code observation: or-pattern for account_id and empty Initial plan commit
…a data source before POSTing sensor data

Signed-off-by: F.N. Claessen <claessen@seita.nl>
…l time (UTC+1)

Signed-off-by: F.N. Claessen <claessen@seita.nl>
Signed-off-by: F.N. Claessen <claessen@seita.nl>
Signed-off-by: F.N. Claessen <claessen@seita.nl>
Signed-off-by: F.N. Claessen <claessen@seita.nl>
…n delete tests

Context:
- PR #2058 dropped FK constraints on DataSource.user_id and account_id
  to preserve lineage when users/accounts are deleted
- The new tests only checked one of the two lineage fields per test

Change:
- test_delete_user: also assert data_source.account_id is preserved
  after user deletion (user deletion should not nullify account_id)
- test_delete_account: also save and assert data_source.user_id per
  data source; account deletion internally calls delete_user so both
  user_id and account_id lineage must survive the full deletion chain
…eservation

Context:
- The existing entry only mentioned adding account_id to data_source
- PR #2058 also drops FK constraints on user_id and account_id to preserve
  historical references when users/accounts are deleted (data lineage)

Change:
- Changelog now states both the column addition AND the behavior change:
  'historical user/account IDs are no longer nullified when users or
  accounts are deleted'
…s lesson

Context:
- PR #2058 dropped FK constraints on DataSource.user_id and account_id
  for data lineage preservation (a deliberate design decision)
- The original changelog entry omitted the FK drop and behavior change
- No existing agent pattern covered 'intentional FK removal for lineage'

Change:
- Added checklist for no-FK lineage pattern (passive_deletes, tests, changelog)
- Added lesson: changelog must describe behavior changes, not just schema changes
- Documented FK drop design decision for future reviewers
…plementation

The note previously described the 'or' anti-pattern as a concern,
but the final code already uses 'if account_id is not None' correctly.
Clarified to point to the implemented solution.
…ource' into copilot/add-account-id-filter-sources
@read-the-docs-community
Copy link
Copy Markdown

read-the-docs-community bot commented Mar 26, 2026

Documentation build overview

📚 flexmeasures | 🛠️ Build #32234024 | 📁 Comparing 4859a78 against latest (45db49a)

  🔍 Preview build  

Show files changed (9 files in total): 📝 9 modified | ➕ 0 added | ➖ 0 deleted
File Status
changelog.html 📝 modified
genindex.html 📝 modified
_autosummary/flexmeasures.data.models.generic_assets.html 📝 modified
_autosummary/flexmeasures.data.models.time_series.html 📝 modified
_autosummary/flexmeasures.data.queries.utils.html 📝 modified
_autosummary/flexmeasures.data.schemas.account.html 📝 modified
_autosummary/flexmeasures.data.services.data_sources.html 📝 modified
api/v3_0.html 📝 modified
features/reporting.html 📝 modified

Copilot AI added 2 commits March 26, 2026 15:01
Add an account_id parameter (int or list[int]) to:
- get_source_criteria() / source_account_criterion() in queries/utils.py
- Sensor.search_beliefs() and TimedBelief.search() in models/time_series.py
- GenericAsset.search_beliefs() in models/generic_assets.py
- BeliefsSearchConfigSchema in schemas/reporting/__init__.py

Allows reporters/schedulers/forecasters to filter beliefs by the
account_id of their data source, implementing issue #2046.
Context:
- Issue #2046 added account_id filtering to Sensor.search_beliefs and TimedBelief.search
- No tests existed for the new account_id parameter

Change:
- Added setup_sources_with_accounts fixture in conftest.py
  Creates a sensor with beliefs from three sources: one per Prosumer/Supplier
  account and one without an account, enabling account_id filter testing
Copilot AI requested a review from Flix6x March 26, 2026 19:24
Flix6x and others added 28 commits March 27, 2026 13:38
Signed-off-by: F.N. Claessen <claessen@seita.nl>
…either is deleted

Signed-off-by: F.N. Claessen <claessen@seita.nl>
Signed-off-by: F.N. Claessen <claessen@seita.nl>
Signed-off-by: F.N. Claessen <claessen@seita.nl>
Signed-off-by: F.N. Claessen <claessen@seita.nl>
Signed-off-by: F.N. Claessen <claessen@seita.nl>
Signed-off-by: F.N. Claessen <claessen@seita.nl>
Signed-off-by: F.N. Claessen <claessen@seita.nl>
Signed-off-by: F.N. Claessen <claessen@seita.nl>
…-account-id-filter-sources

# Conflicts:
#	documentation/changelog.rst
Signed-off-by: F.N. Claessen <claessen@seita.nl>
…ource' into copilot/add-account-id-filter-sources
- Resolve conflicts in agent documentation files
- Keep both DataSource and Annotation documentation sections
- Accept deletion of lead.md from origin/main (governance consolidated)
- Integrate annotation API implementation from main branch
- All conflict markers resolved
Update all agent documentation to use 'Lead' instead of 'Review Lead',
reflecting the consolidated governance model where lead.md has been
removed and its responsibilities are now documented in AGENTS.md.
Changes:
- AGENTS.md: All 31 'Review Lead' references updated to 'Lead'
- coordinator.md: Updated monitoring patterns to reference Lead
- test-specialist.md: Updated coordination examples to reference Lead
- tooling-ci-specialist.md: Updated coordination examples to reference Lead
This ensures consistent terminology across all agent instructions.
Update b2e07f0dafa1_merge.py migration to specify both parent revisions
that are being merged:
- 3f4a6f9d2b11 (increase_audit_event_length_to_500)
- b2c3d4e5f6a7 (Drop FK constraint from audit_log.active_user_id)
This resolves the multiple heads issue and creates a proper mergepoint
in the migration history. The migration now correctly shows as a
(mergepoint) in the history output.
Clean up leftover git merge conflict markers (<<<<<<< HEAD, =======, >>>>>>> origin/main)
that were not properly resolved. Both pattern sections (2026-03-24 and 2026-02-10)
have been preserved as they document complementary learning from different sessions.
Signed-off-by: F.N. Claessen <claessen@seita.nl>
Co-authored-by: Nicolas Höning <nicolas@seita.nl>
Signed-off-by: Felix Claessen <30658763+Flix6x@users.noreply.github.com>
…7450113f6)

Remove the intermediate a1b2c3d4e5f6 migration that was dropping the account_id FK
(only created within this PR) and the user_id FK. Instead:
- Don't create the account_id FK in 9877450113f6 in the first place
- Absorb the user_id FK drop into 9877450113f6
- Update b2c3d4e5f6a7 to chain off 9877450113f6 directly
- Delete a1b2c3d4e5f6

Agent-Logs-Url: https://github.com/FlexMeasures/flexmeasures/sessions/4d9d8a92-d4fd-4635-bebf-1965c35667b3

Co-authored-by: nhoening <1042336+nhoening@users.noreply.github.com>
Signed-off-by: F.N. Claessen <claessen@seita.nl>
Signed-off-by: F.N. Claessen <claessen@seita.nl>
Signed-off-by: F.N. Claessen <claessen@seita.nl>
Signed-off-by: F.N. Claessen <claessen@seita.nl>
…main)

Change 9877450113f6.down_revision from 8b62f8129f34 to e26d02ed1621 so
the PR's two real migrations (9877450113f6, b2c3d4e5f6a7) chain linearly
after the tip of the main branch, making the two merge-only files
(b2e07f0dafa1 and 2c6a4c8127c1) completely obsolete.

Delete b2e07f0dafa1_merge.py and 2c6a4c8127c1_merge.py.

Agent-Logs-Url: https://github.com/FlexMeasures/flexmeasures/sessions/94c3070e-a701-4a97-a895-9ea1064e3c20

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>
Signed-off-by: F.N. Claessen <claessen@seita.nl>
…ource' into copilot/add-account-id-filter-sources

# Conflicts:
#	.github/agents/architecture-domain-specialist.md
#	.github/agents/coordinator.md
#	flexmeasures/data/schemas/account.py
Base automatically changed from copilot/add-account-id-to-data-source to main April 13, 2026 11:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Reporters/schedulers/forecasters can filter by source account

2 participants