Add "Delete data" action to sensor page with filtered DELETE endpoint

[Copilot]

Adds a "Delete data" panel to the sensor page sidebar, backed by a filtered DELETE endpoint, allowing users to delete sensor data by source, start time, and/or end time.

Changes Made

• Delete data panel: Added a sidebar panel with source dropdown, "From" and "Until" datetime inputs, a "Select all data" link, and a Delete button (visible to users with delete permission).
• Filtered DELETE endpoint: Extended DELETE /api/v3_0/sensors/<id>/data to accept optional source, start, and until body parameters for scoped deletion.
• Source dropdown populated from stats: The source dropdown reuses data already fetched by loadSensorStats() to avoid a duplicate /stats API call.
• Inputs pre-filled from graph time range: The "From" and "Until" inputs are initialised to the time range currently shown in the graph. They stay in sync when the user changes the date picker via a graphDateRangeChanged custom event dispatched from graphs.html.
• "Select all data" link: A link on the left of the Delete button sets the "From" and "Until" inputs to the sensor's earliest first event start and latest last event end, respectively, using data already available from the statistics panel. The link is revealed once stats are loaded.
• Statistics table refreshed after deletion: The statistics table and source dropdown are automatically updated after a successful data deletion, keeping the page in sync without requiring a manual refresh.
• Stats endpoint returns Cache-Control: no-store: The GET /api/v3_0/sensors/<id>/stats endpoint now sends Cache-Control: no-store so browsers never cache stats responses, ensuring the refreshed stats after deletion always reflect the current state.
• Snapshot values passed to deleteData: Input values are read at click time and passed as arguments to deleteData(sourceValue, startValue, untilValue), ensuring the confirmation message and the DELETE request use the same values.
• Toast bug fix: Removed the redundant manual btn-close click listener in toasts.html that caused a TypeError: Cannot read properties of null (reading 'classList') error.

[read-the-docs-community]

Documentation build overview

📚 flexmeasures | 🛠️ Build #32245822 | 📁 Comparing 344f5ed against latest (7dee659)

  🔍 Preview build  

Show files changed (17 files in total): 📝 16 modified | ➕ 0 added | ➖ 1 deleted

File	Status
changelog.html	📝 modified
configuration.html	📝 modified
genindex.html	📝 modified
index.html	📝 modified
_autodoc/STATE_OF_CHARGE.html	📝 modified
_autosummary/flexmeasures.api.v3_0.sensors.html	📝 modified
_autosummary/flexmeasures.data.models.charts.belief_charts.html	📝 modified
_autosummary/flexmeasures.data.models.generic_assets.html	📝 modified
_autosummary/flexmeasures.data.models.planning.storage.html	📝 modified
_autosummary/flexmeasures.data.schemas.utils.html	📝 modified
_autosummary/flexmeasures.data.services.data_sources.html	📝 modified
_autosummary/flexmeasures.data.services.time_series.html	📝 modified
api/v3_0.html	📝 modified
dev/scripting.html	➖ deleted
features/scheduling.html	📝 modified
host/installation.html	📝 modified
tut/forecasting_scheduling.html	📝 modified

[joshuaunity]

This works as expected, I've gone through the code and i also think the approach used is on point.

[joshuaunity]

Hi @nhoening, this code is needed, as it's being used to populate the dropdown of data sources for the form.

[nhoening]

Hi @nhoening, this code is needed, as it's being used to populate the dropdown of data sources for the form.

As I said earlier, the statistics panel on this page already makes this call, I believe.

• the information alreay exists on the page
• the call is expensive (computes statistics), so we should not make it twice
• one other way out is to have a simple endpoint GET /sensors/ID/sources to get the sources which have data on that sensor, but it seems this page already has the info so that should be done.

@copilot investigate re-using the source information from the statistics panel for this button.

[Copilot]

Hi @nhoening, this code is needed, as it's being used to populate the dropdown of data sources for the form.

...

Done in e2b9bc5. The separate loadDeleteDataSources() function and its /stats API call have been removed. Instead, loadSensorStats() in flexmeasures.js now also populates the deleteDataSource dropdown using the stats data it already fetched, so no duplicate call is made.

[nhoening]

This works as expected, I've gone through the code and i also think the approach used is on point.

@joshuaunity did you test manually also?

[joshuaunity]

Hi @nhoening, this code is needed, as it's being used to populate the dropdown of data sources for the form.

...

Done in e2b9bc5. The separate loadDeleteDataSources() function and its /stats API call have been removed. Instead, loadSensorStats() in flexmeasures.js now also populates the deleteDataSource dropdown using the stats data it already fetched, so no duplicate call is made.

It was in a separate file... I see.

[joshuaunity]

This works as expected, I've gone through the code and i also think the approach used is on point.

@joshuaunity did you test manually also?

yes i did, but ill pull this upate and test again

[joshuaunity]

Something I noticed, when all sources are deleted, it reflects immediately on the graph, but it takes some time before it reflects on the page itself. That is, the statistics and the dropdown still show the data source.

But there doesn't seem to be any drawback or issues caused by this.

@nhoening

[joshuaunity]

@Flix6x i have tested again adn it works fine.

it took a while, but i also fixed a bug relating to the toast, which i noticed on this PR.

[nhoening]

@copilot please initalize the datepicker for the delete-data dialogue to the start and end time shown in the graph

[Copilot]

@copilot please initalize the datepicker for the delete-data dialogue to the start and end time shown in the graph

Done in 6be0e99.

Two things now happen automatically:

1. On page load with URL time params – the "From" and "Until" inputs are pre-filled from event_starts_after / event_ends_before (the same values the graph is initially queried with).
2. When the user changes the date picker – graphs.html dispatches a graphDateRangeChanged custom event after the new storeStartDate/storeEndDate are set; the index.html listener calls syncDeleteInputsWithGraphRange() to update both inputs to match the new graph window.

[nhoening]

@copilot also, add a "select all data" link on the left of the Delete button, which will set the start and end time to the first and last event time, respectively. These datetimes can be found in the statistics table.

[Copilot]

@copilot also, add a "select all data" link on the left of the Delete button, which will set the start and end time to the first and last event time, respectively. These datetimes can be found in the ...

Done in fdaa8c5.

A "Select all data" link now appears on the left of the Delete button. It is initially hidden and is revealed once the stats data has loaded. When clicked, it sets the "From" input to the earliest "First event start" and the "Until" input to the latest "Last event end" across all sources, using the data already fetched by loadSensorStats().

Implementation:

• flexmeasures.js: after populating the delete source dropdown, dispatches a sensorDataRangeAvailable custom event with { firstEventStart, lastEventEnd } computed from the stats.
• index.html: listens for sensorDataRangeAvailable to store the dates and reveal the link; the link's click handler calls syncDeleteInputsWithGraphRange() to fill both inputs.

[nhoening]

@copilot good. but actually, the last recorded event time is the beginning of the event, so for the end time (when the "Select all data" link is clicked, we should add the sensor resolution.

Also, the statistics table should be updated upon sensor data deletion.

[Copilot]

@copilot good. but actually, the last recorded event time is the beginning of the event, so for the end time (when the "Select all data" link is clicked, we should add the sensor resolution.

Also, ...

Done in d58c513.

Two changes made:

1. Resolution added to end time – in index.html, when sensorDataRangeAvailable fires, sensorResolutionMs = {{ sensor.event_resolution.total_seconds() | int }} * 1000 is added to lastEventEnd before storing it. So clicking "Select all data" now sets "Until" to the actual end of the last event (start + resolution).

2. Stats table refreshed after deletion – in graphs.html, a new dataDeleted listener calls loadSensorStats() (respecting whether the stats checkbox is checked and the current date range), so the statistics table and source dropdown update automatically after a successful deletion.

[nhoening]

@copilot the code looks okay but both additions from d58c513 are not working properly yet.

• The resolution is not added in the selection.
• the stats are updated, but don't often seem to reflect the new state - might this be a caching issue?

Finally, syncDeleteInputsWithGraphRange should now be called syncDeleteInputsWithRange - as we also use a range different from the graph.

[Copilot]

@copilot the code looks okay but both additions from d58c513 are not working properly yet.

• The resolution is n...

Fixed in 9857109.

Three issues addressed:

1. Resolution double-count removed – The stats endpoint (sensors.py) already computes "Last event end" = max_event_start + sensor.event_resolution, so the event end is correct out of the box. The previous commit was incorrectly adding the resolution a second time. That extra addition has been removed; allDataEnd now uses e.detail.lastEventEnd directly.

2. Caching fixed – Added cache: 'no-cache' to the fetch call in loadSensorStats() so the browser always retrieves fresh stats from the server after a deletion.

3. Rename done – syncDeleteInputsWithGraphRange → syncDeleteInputsWithRange everywhere it appears.

[nhoening]

@copilot the endpoint called by loadSensorStats() does not seem to respect the cache: 'no-cache' header. Maybe it is possible to add that behavior.

[Copilot]

@copilot the endpoint called by loadSensorStats() does not seem to respect the cache: 'no-cache' header. Maybe it is possible to add that behavior.

Fixed in 344f5ed. The GET /<id>/stats endpoint now returns Cache-Control: no-store in its response headers. This instructs browsers (and any intermediary proxies) to never store the response in a cache, ensuring every call to loadSensorStats() always fetches fresh data from the server.