Use stdout for datapoint exports #9
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Ekhorn
force-pushed
the
enhancement/use-stdout-for-datapoint-exports
branch
from
4e2addf to
8d288e1
Compare
Ekhorn
force-pushed
the
enhancement/use-stdout-for-datapoint-exports
branch
from
8d288e1 to
57f349c
Compare
Author
|
The backend tests pass locally. 
And so does your frontend test 😉 
I do need approval to run the pipeline 😄 |
Ekhorn
commented
Dec 24, 2025
Comment on lines
-60
to
-61
| public static final String OR_DATA_POINTS_EXPORT_LIMIT = "OR_DATA_POINTS_EXPORT_LIMIT"; | ||
| public static final int OR_DATA_POINTS_EXPORT_LIMIT_DEFAULT = 1000000; |
Author
There was a problem hiding this comment.
@richturner Is it okay to get rid of this? The default CSV exports with 10million datapoints seems not to have a big impact. The other CSV format queries that @Hackerberg43 added take longer (and the request times out by axios on the frontend).
Owner
There was a problem hiding this comment.
The above was added quite recently I remember, as otherwise you would just get a general from the timeout, not letting you know what the problem is.
The other formats do some database manipulation first, so will take longer yes.
Hackerberg43
merged commit d184d26
into
Hackerberg43:enhancement/dataExportPage
1 check passed
3 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Changes the datapoint exports to write to
stdoutrather than the filesystem and uses piped streams so data can be streamed to the client without having to hold all datapoints in memory.I did some tests to ensure the memory footprint stays reasonable. With 10 million datapoints exported multiple times in parallel (4 active streams) the memory usage spikes from ~150MB to ~350MB. The zip files that are downloaded each are ~131MB and the
datapoints.csvfiles are ~620MB.Screencast.From.2025-12-24.14-32-01.webm
Checklist