Feature Request: Efficient Server-Side Filtering & Historical Data Retrieval

[tyronechrisharris]

Enhancement/Feature Request

Description

The current filtering implementation relies on loading all available occupancy and event data into the client-side memory to perform filtering operations. On deployments with high data volume (e.g., 50 lanes, millions of occupancy records), this causes the frontend to become unresponsive, freeze, or crash.

We need to transition to a server-side query model where the client requests specific subsets of data from the database based on user criteria, rather than filtering a massive pre-loaded table.

Use Case

• Operational Stability: Operators need the dashboard to run continuously without performance degradation or crashing due to memory overload from historical data.
• Historical Analysis: An operator needs to find specific old alarms (e.g., "all alarming occupancies between Nov 22nd and Nov 24th") without loading the entire database history.
• Workflow Management: Users need to efficiently retrieve and view all "unadjudicated" alarms to process them, without the interface bogging down.

Expected Behavior

1. Dashboard Default View: To maintain continuous stability, the Dashboard and main views should default to loading a "rolling window" of recent data (e.g., the last 24 to 48 hours). This window duration should ideally be user-configurable.
2. Server-Side Querying: The client must be able to execute specific queries against the database for data outside the default window.
3. Filtering Capabilities: The system must support server-side filtering for:
   • Date Range: Retrieving events between specific start and end dates.
   • Alarm Status: Filtering by Alarming vs. Non-Alarming.
   • Adjudication Status: Specifically filtering for "Unadjudicated" items to facilitate workflow.

Current Behavior

• The client attempts to load all available data into the table/browser memory upon initialization or navigation.
• On large systems, the application becomes unresponsive or fails to load new alarms properly.
• The current "Event Log" filtering mechanism only applies to the data page currently loaded in the view, meaning it does not accurately search the entire history.

Additional Information

• Context: This issue was identified as a critical deliverable for Phase 3.5.
• Database Optimization: The backend should utilize Postgres capabilities (such as indexing or time-based partitioning extensions) to execute these queries efficiently without requiring the entire dataset to be resident in RAM.
• Performance: This change is necessary to support requirements for high-volume sites (e.g., 50 lanes) where the database may contain hundreds of millions of records.

WebID Information

The spectrum "file" can be in various formats like JSON, CSV, TXT, or XML.
The WebID team recommends producing a proper N42 format.
If you prefer not to create a file on the filesystem, you can include the "file" contents directly in the request body using multipart/form-data without saving the file. Most programming languages support attaching an in-memory blob to the request.

Get list of options, and all available detectors

curl https://full-spectrum.sandia.gov/api/v1/info

Analyze foreground+background, with each spectrum being in seperate files, and explicitly specifying DRF

curl -v -F "options={"drf": "IdentiFINDER-NGH"}" -F "foreground=@./foreground.n42" -F "background=@./background.n42" https://full-spectrum.sandia.gov/api/v1/analysis

Similar, but specify DRF as an argument in the URL

curl -v -F "foreground=@./foreground.n42" -F "background=@./background.n42" https://full-spectrum.sandia.gov/api/v1/analysis?drf=IdentiFINDER-NGH

Similar, but with foreground and background in same file (which is which will be detected)

curl -v -F "filesomething=@./foregroundWithBackground.n42" https://full-spectrum.sandia.gov/api/v1/analysis?drf=IdentiFINDER-NGH

Have detector type be automatically detected

curl -v -F "file=@./someFile.n42" https://full-spectrum.sandia.gov/api/v1/analysis

Analyze a foreground only spectrum (background will be synthesized, and detector type automatically detected)

curl -v -F "options={"synthesizeBackground": true}" -F "foreground=@./foreground.n42" https://full-spectrum.sandia.gov/api/v1/analysis

Similar to previous, but specifying options as URL arguments, and explicitly specifying detector system

curl -v -F "foreground=@./foreground.n42" "https://full-spectrum.sandia.gov/api/v1/analysis?synthesizeBackground=true&drf=Falcon%205000"