Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
60 commits
Select commit Hold shift + click to select a range
4b37779
Initial plan
Copilot Mar 24, 2026
64f791a
data_sources: add account_id field to DataSource model
Copilot Mar 24, 2026
6745921
migration: add account_id to data_source table (9877450113f6)
Copilot Mar 24, 2026
78fb04f
migration: replace N+1 loop with single UPDATE...FROM for account_id …
Copilot Mar 24, 2026
65efbdd
tests/sensor_data: verify account_id is set on data source when posti…
Copilot Mar 24, 2026
85778e4
migration: use correlated subquery for account_id data migration; add…
Copilot Mar 24, 2026
06bbec9
data_sources: fix account_id assignment to avoid SAWarning and handle…
Copilot Mar 24, 2026
8b02034
agents/architecture: add migration checklist and DataSource account_i…
Copilot Mar 24, 2026
d7f7120
agents/test-specialist: add DataSource property testing pattern and l…
Copilot Mar 24, 2026
20a4e15
agents/review-lead: add agent selection checklist; document repeated …
Copilot Mar 24, 2026
fff71b2
agents/coordinator: document PR #2058 persistent self-improvement fai…
Copilot Mar 24, 2026
4191ae3
data_sources: use explicit is not None check for account_id assignment
Copilot Mar 24, 2026
38db656
feat: improve test by using a user that is not already registered as …
Flix6x Mar 24, 2026
67eba6e
fix: set approximate create datetime based on git info, assuming loca…
Flix6x Mar 24, 2026
b7a4a5c
style: flake8
Flix6x Mar 24, 2026
44cff24
feat: add migration check to architecture agent instructions
Flix6x Mar 24, 2026
95653f1
docs: add db upgrade warning and update agent instructions accordingly
Flix6x Mar 24, 2026
4e22e43
data_sources: drop FK constraints on user_id and account_id for data …
Copilot Mar 25, 2026
5c29f56
tests/data-lineage: also assert account_id and user_id preservation i…
Copilot Mar 25, 2026
97efde0
docs: expand PR #2058 changelog entry to cover FK drop and lineage pr…
Copilot Mar 25, 2026
6abc8d9
coordinator: document no-FK lineage pattern and changelog completenes…
Copilot Mar 25, 2026
ebfba1d
coordinator: clarify account_id fallback pattern references actual im…
Copilot Mar 25, 2026
8ad1866
Initial plan
Copilot Mar 26, 2026
688e812
Merge remote-tracking branch 'origin/copilot/add-account-id-to-data-s…
Copilot Mar 26, 2026
5cf63d2
Add account_id filter for source in search_beliefs
Copilot Mar 26, 2026
867c46c
tests/data: add fixture for sources with account IDs
Copilot Mar 26, 2026
7267db4
tests/schemas: add BeliefsSearchConfigSchema account_id field tests
Copilot Mar 26, 2026
162dbd9
docs: add account_id source filtering to changelog and reporting docs
Copilot Mar 26, 2026
32bf40d
agents: document schema-parity and model-coverage gaps from PR #2065
Copilot Mar 26, 2026
a090571
fix: add account_id to Input schema, GenericAsset test, and edge case…
Copilot Mar 26, 2026
2b3105b
fix: use AccountIdField, support single int, reject empty list, fix c…
Copilot Mar 26, 2026
ddf576d
refactor: introduce AccountIdOrListField, restore parametrized tests
Copilot Mar 26, 2026
c64b4ee
feat: add test covering CLI command to delete user
Flix6x Mar 27, 2026
531cb74
Merge remote-tracking branch 'origin/main' into copilot/add-account-i…
Flix6x Mar 27, 2026
57c1f8e
feat: let audit log entries retain the account ID and user IDs after …
Flix6x Mar 27, 2026
e1d6cdc
docs: rewrite changelog entry
Flix6x Mar 27, 2026
7ec2cb1
fix: dropped foreign key name
Flix6x Mar 27, 2026
0fefb2e
fix: when deleting a user, still log the ID of the deleted user
Flix6x Mar 27, 2026
6bcf9c6
feat: allow tying a newly CLI-created data source to an account
Flix6x Mar 27, 2026
faa876f
fix: add missing drop_constraints
Flix6x Mar 27, 2026
317c687
chore: minimize diff
Flix6x Mar 27, 2026
4401355
style: flake8
Flix6x Mar 27, 2026
16e252b
Merge branch 'copilot/add-account-id-to-data-source' into copilot/add…
Flix6x Mar 27, 2026
d7a4ec1
fix: test
Flix6x Mar 27, 2026
2965547
Merge remote-tracking branch 'origin/copilot/add-account-id-to-data-s…
Flix6x Mar 27, 2026
36989e3
Merge origin/main into copilot/add-account-id-to-data-source
Flix6x Apr 2, 2026
8bb5ebe
agents: rename Review Lead to Lead across all agent instructions
Flix6x Apr 2, 2026
86d3ebb
db: fix merge migration to properly resolve conflicting heads
Flix6x Apr 2, 2026
7d5fa60
agents: remove merge conflict markers from coordinator.md
Flix6x Apr 2, 2026
4724959
style: flake8
Flix6x Apr 2, 2026
60a4b50
docs: clarify why we add a creation audit log record in a test
Flix6x Apr 5, 2026
bf5bf75
migration: merge drop-FK migration into add-account-id migration (987…
Copilot Apr 10, 2026
1d39eae
docs: make this lengthy comment only once
Flix6x Apr 5, 2026
2503d95
docs: add explanation for defining foreign keys in the db.relationship
Flix6x Apr 5, 2026
40ab6ae
Merge remote-tracking branch 'origin/main' into copilot/add-account-i…
Flix6x Apr 13, 2026
d43b41d
fix: merge db revisions
Flix6x Apr 13, 2026
74ed3dc
style: flake8
Flix6x Apr 13, 2026
16655e4
migration: rebase add-account-id migration onto e26d02ed1621 (tip of …
Copilot Apr 13, 2026
e7139da
style: punctuation / double spaces
Flix6x Apr 13, 2026
4859a78
Merge remote-tracking branch 'origin/copilot/add-account-id-to-data-s…
Flix6x Apr 13, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions .github/agents/architecture-domain-specialist.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,7 @@ Marshmallow schemas define the canonical format for parameter dictionaries. All
- [ ] **Dictionary access**: Verify code uses dict keys from `data_key`, not Python attributes
- [ ] **Parameter modification**: Check `pop()`, `del`, assignment operations use correct keys
- [ ] **Storage consistency**: Ensure DataSource.attributes, job.meta use schema format
- [ ] **Schema parity**: When adding a filter/parameter to `Sensor.search_beliefs`, verify it is added to BOTH `Input` (io.py) AND `BeliefsSearchConfigSchema` (reporting/__init__.py). These two schemas serve overlapping purposes but are distinct classes — omitting one creates a silent gap where documented features silently fail at schema validation time.

**Domain Pattern: Schema Format Migrations**

Expand Down Expand Up @@ -588,3 +589,18 @@ After each assignment:
Change:
- Added guidance on <specific topic>
```

### Lessons Learned

**Session 2026-03-24 (PR #2058 — add account_id to DataSource)**:

- **New domain invariant**: User-type DataSources now have `account_id` populated. Document new FK relationships and invariants in the Domain Knowledge section immediately.
- **Migration checklist**: Added an explicit Alembic migration checklist after reviewing the migration for this PR. Key patterns: correlated subquery for bulk backfill, SQLAlchemy Core stubs (no ORM imports), `batch_alter_table` for all ALTER operations, exact constraint name matching.
- **Missed API Specialist coordination**: The PR changed endpoint behavior (POST sensor data sets account_id on the created data source). The API Specialist should have been engaged to verify backward compatibility. When domain model changes affect how endpoints behave or what they return, flag for API Specialist review.
- **Self-improvement failure**: Despite having explicit self-improvement requirements, no agent updated its instructions during this PR session. This was caught by the Coordinator post-hoc. The agent must update its own instructions as the LAST step of every assignment, not skip it.

**Session 2026-04 (PR #2065 — add account_id filter to search_beliefs)**:

- **Schema parity gap**: The PR added `account_id` to `BeliefsSearchConfigSchema` but not to `Input` (io.py). These two schemas both expose `Sensor.search_beliefs` parameters; omitting a parameter from one creates a silent gap. The architecture agent must check both schemas on any search_beliefs parameter addition.
- **Documentation vs. implementation mismatch**: The `reporting.rst` docs stated reporters can filter by `account_id`, but this only works if `Input` also has the field. Docs that outrun schema support mislead users. Always verify the full schema chain before documenting a feature.
- **DataSource account_id=None for non-user sources**: The existing invariant (reporters/schedulers/forecasters have `account_id=None`) limits the usefulness of `account_id` filtering: it only matches user-type sources. PRs adding `account_id` filters should either document this limitation explicitly or reconsider the invariant.
151 changes: 112 additions & 39 deletions .github/agents/coordinator.md

Large diffs are not rendered by default.

56 changes: 40 additions & 16 deletions .github/agents/test-specialist.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,11 +118,11 @@ If ANY test fails during full suite execution:
**Click context errors**:
- Check IdField decorators (`@with_appcontext` vs `@with_appcontext_if_needed()`)
- Compare against SensorIdField pattern
- See Review Lead's Click context error pattern
- See Lead's Click context error pattern

### Integration with Review Lead
### Integration with Lead

The Test Specialist MUST provide evidence of full test suite execution to Review Lead.
The Test Specialist MUST provide evidence of full test suite execution to Lead.

**Required evidence format:**
```
Expand All @@ -134,14 +134,14 @@ Full test suite execution:
- Coverage: 87.2% (unchanged)
```

**Review Lead verification:**
Review Lead's session close checklist includes:
**Lead verification:**
Lead's session close checklist includes:
- [ ] Test Specialist confirmed full test suite execution
- [ ] All tests pass (100%)
- [ ] Test output captured and reviewed

**Enforcement:**
Review Lead cannot close session until Test Specialist provides evidence of full test suite execution with 100% pass rate.
Lead cannot close session until Test Specialist provides evidence of full test suite execution with 100% pass rate.


## Testing Patterns for flexmeasures
Expand Down Expand Up @@ -880,6 +880,30 @@ flask db current
- **Assertions**: Use descriptive assertion messages for failures
- **Mocking**: Use pytest fixtures and mocking when testing external dependencies

### Testing DataSource Properties After API Calls

When writing tests that verify data source properties (e.g. `account_id`, `user`, `type`) after an API call:

1. **Use `fresh_db` fixture** — tests that POST data and then query the resulting data source are modifying the DB and must use the function-scoped `fresh_db` fixture. Place these tests in a `_fresh_db` module.

2. **Query by user, not just name** — data sources created by the same user across test runs may collide; use `filter_by(user=user)` or `filter_by(user_id=user.id)` for precision.

3. **Pattern** (from `test_post_sensor_data_sets_account_id_on_data_source`):
```python
# Fetch the user that made the request
user = db.session.execute(
select(User).filter_by(email="test_supplier_user_4@seita.nl")
).scalar_one()
# Fetch the data source created for that user
data_source = db.session.execute(
select(Source).filter_by(user=user)
).scalar_one_or_none()
assert data_source is not None
assert data_source.account_id == user.account_id
```

4. **Check both existence and value** — don't just assert `data_source is not None`; also assert the specific field value you're testing.

## Understanding Test Design Intent (CRITICAL)

**Before changing a test, understand WHY it's designed that way.**
Expand Down Expand Up @@ -1060,14 +1084,14 @@ After each assignment:
- Added guidance on <specific topic>
```

Example:
### Lessons Learned

```
agents/test-specialist: learned to verify claims with actual test runs
Context:
- Session #456 claimed tests passed but they were never actually run
- Led to bug slipping through to production
Change:
- Added "Actually Run Tests" section with verification steps
- Emphasized checking test output and coverage
```
**Session 2026-03-24 (PR #2058 — add account_id to DataSource)**:

- **Self-improvement failure**: Despite having explicit instructions to update this agent file after each assignment, no update was made during this PR session. This was caught by the Coordinator post-hoc. The agent must treat instruction updates as the LAST mandatory step of any assignment.
- **DataSource property testing**: Added guidance in "Testing DataSource Properties After API Calls" above. When testing properties set by the API on a data source (like `account_id`), use `fresh_db`, query by user to avoid ambiguity, and assert both existence and the specific field value.

**Session 2026-04 (PR #2065 — add account_id filter to search_beliefs)**:

- **Model layer coverage gap**: The PR added `account_id` parameter to three model methods: `Sensor.search_beliefs`, `TimedBelief.search`, and `GenericAsset.search_beliefs`. Tests covered only the first two. When a parameter is added to multiple model-layer methods, each method must be tested independently — especially when they use different code paths (e.g., `GenericAsset.search_beliefs` delegates through `Sensor.search_beliefs` but may not if the delegation chain changes). Add a checklist item: "When a filter/param is added to `search_beliefs` across multiple models, ensure at least one test exercises each model class."
- **Empty-list edge case**: `account_id=[]` is schema-valid but semantically means "match nothing" (SQL `IN ()` returns zero rows). Tests should include this edge case and assert the expected empty result rather than letting it pass as a valid no-op.
12 changes: 6 additions & 6 deletions .github/agents/tooling-ci-specialist.md
Original file line number Diff line number Diff line change
Expand Up @@ -163,12 +163,12 @@ Code that bypasses pre-commit:

**Who runs pre-commit:**
- **During code changes**: Agent making changes runs pre-commit before committing
- **Before PR close**: Review Lead verifies pre-commit execution
- **Before PR close**: Lead verifies pre-commit execution
- **In PR review**: Tooling & CI Specialist validates config matches CI

**Enforcement:**
- Review Lead's session close checklist includes pre-commit verification
- Review Lead cannot close session without pre-commit evidence
- Lead's session close checklist includes pre-commit verification
- Lead cannot close session without pre-commit evidence
- If pre-commit fails, agent must fix all issues before proceeding

#### Common Failures and Fixes
Expand Down Expand Up @@ -206,9 +206,9 @@ black .
ci/run_mypy.sh
```

#### Integration with Review Lead
#### Integration with Lead

**Review Lead checklist items:**
**Lead checklist items:**
- [ ] Pre-commit hooks installed
- [ ] All hooks pass: `pre-commit run --all-files`
- [ ] Zero failures from flake8, black, mypy
Expand All @@ -219,7 +219,7 @@ ci/run_mypy.sh
- Or confirm: "Pre-commit verified: all hooks passed"

**Enforcement:**
Review Lead MUST verify pre-commit execution before closing session.
Lead MUST verify pre-commit execution before closing session.

### Agent Environment Setup

Expand Down
Loading
Loading