feat: local AI analytics (ONNX object detection) by keyldev · Pull Request #23 · OpenIPC/viewer

keyldev · 2026-06-19T20:33:44Z

Summary

Adds on-device object detection for cameras: per-camera detection
with class filtering, bounding-box + counter overlay on the live grid,
auto-record on detection with a smart auto-stop, detection events, and an AI
control center. Everything runs locally via ONNX Runtime in-process — no
cloud, no Python sidecar. Inference never leaves the machine.

Detection core (15.1/15.2) — new OpenIPC.Viewer.Analytics project on
Microsoft.ML.OnnxRuntime. Pure, testable YOLOX post-processor (grid/stride
decode + per-class NMS) + letterbox mapping. Execution-provider selection with
a mandatory CPU fallback — the app never crashes because a GPU/NPU EP is
missing.

Sampling pipeline (15.3) — one shared detector + a single inference worker
behind a bounded drop-oldest queue. Frames are thinned to analyticsFps
(default 3) and downscaled off the decode thread, so memory/CPU stay bounded
regardless of camera count.
Per-camera settings (15.4) — enable, class selection (curated COCO
subset), confidence threshold, FPS, auto-record + post-event seconds. Migration
010_ai_analytics.sql. Global "Force CPU" toggle in Settings.
Overlay (15.5) — DetectionOverlay draws normalized boxes + per-class
colors/labels and an object counter on the live tile. Analytics pauses with
Smart Pause (Phase 12) — hidden/suspended tiles stop inferring.
Auto-record (15.6) — detection starts recording and keeps it running for
postEventSeconds after the last hit (each hit extends the window); graceful
stop via RecordingService (no corrupted MP4); cooldown against flapping.
Manual recordings are never auto-stopped.
Detection events (15.7) — new EventKind.Detection flowing through the
existing Phase 7 ingestion path (reuses debounce / quiet-close / live feed).
AI control center (15.7) — engine status + diagnostics (active cameras,
frames processed/dropped, queue depth, latency), analytics-enabled cameras,
and recent detections. Includes an engine status line
(Not started → Downloading model… → Loading… → Ready / Failed).

Architecture

Contracts (IObjectDetector, IAnalyticsEngine, IModelProvider,
ModelSpec, Detection, AnalyticsSettings, AutoRecordCoordinator, and the
pure helpers) live in Core; the ONNX-backed implementation lives in
Analytics and is wired via SharedComposition. The App → Core only rule
is preserved (App talks to interfaces, never to ONNX Runtime).
Analytics is opt-in per camera and the engine initializes lazily on the
first enabled live tile, so registering it on every head is boot-safe.

Model & privacy

Model: YOLOX-tiny (Apache-2.0) — deliberately not Ultralytics YOLO (AGPL),
to stay MIT/store-compatible. The post-processor is pluggable via ModelSpec.
The model file (~20 MB) is downloaded once on first enable, cached in
%LOCALAPPDATA%/OpenIPC.Viewer/models/, and SHA-256 verified. After that
it runs fully offline. An OPENIPC_DETECTION_MODEL env var overrides with a
local file. The repo ships no binaries.

Platforms / execution providers

CPU works everywhere (guaranteed fallback).
Android: NNAPI → XNNPACK → CPU (ORT android native ships in the package;
XA0141 16 KB-page advisory suppressed on the Android head).
Desktop GPU (DirectML/CUDA/OpenVINO) is deferred — needs dedicated per-head
packages; Windows/Linux run on CPU for now.
Verified end-to-end on Windows and Android.

Tests

Pure unit tests (Core.Tests): YOLOX decode/NMS/class-filter/threshold, inverse
letterbox, frame sampler rate gate, auto-record window (extend/quiet-stop/
cooldown), detection summary.
OpenIPC.Viewer.Analytics.Tests: engine attach/detach wiring with a fake
detector; a SkippableFact real-model ONNX integration test gated on
OPENIPC_DETECTION_MODEL.
Solution builds with 0 warnings across all five heads.

How to try

Edit a camera → enable AI detection, pick classes (e.g. person, car).
Open Live — on first enable the model downloads (watch the status on the
AI page: "Downloading model…" → "Ready"); requires internet once.
Boxes + counter appear on the tile; the AI page shows live diagnostics.

Known limitations / follow-ups

Analytics (and detection-based auto-record) runs while the Live grid has the
camera's tile attached; a headless/background detection mode is out of scope.
Offline-first delivery (bundle the model into release artifacts, or a
"pick model file" button) is deferred to the packaging phase.
Desktop GPU EP packages (DirectML/CUDA) are a follow-up.

Type

Checklist

Builds with 0 warnings (TreatWarningsAsErrors=true).
Tests pass (dotnet test); new Core logic has unit tests.
No layering violation — App references Core only (Infrastructure / Video / Devices wired via DI in a head).
Scope stays within one phase (didn't pull work from a later phase's "Не входит").
README / docs updated if public commands, options, or setup changed.

Platforms tested

Screenshots / notes

- New OpenIPC.Viewer.Analytics project on ONNX Runtime (no Python sidecar) - Core contracts: IObjectDetector, ModelSpec, Detection, FrameBuffer, ExecutionProvider - Execution-provider selection with mandatory CPU fallback (never crash on GPU) - Pure YOLOX post-processor (grid/stride decode + per-class NMS) + letterbox map - Unit tests for decode, NMS, class filter, threshold, and inverse letterbox

- IAnalyticsEngine + ObjectDetectionEngine: shared detector, one inference worker - FrameSampler thins decoder FPS to analyticsFps; bounded drop-oldest channel - Full-frame downscale-copy off the decoder thread (buffer is session-owned) - IModelProvider/ModelProvider: download-on-first-enable, AppData cache, SHA-256 - ModelCatalog pins Apache-2.0 YOLOX-tiny; AnalyticsSettings + diagnostics - FrameSampler + settings unit tests

- Migration 010: Ai* columns on Cameras (enabled, classes CSV, threshold, fps, auto-record) - Camera/New/UpdateCameraRequest carry AnalyticsSettings; repo maps flat columns - CameraDirectoryService.SetAnalyticsAsync + thread analytics through add/update - Global AiAcceleration user setting (auto/force-cpu) via IUserSettingsAccessor - Register IModelProvider/IObjectDetector/IAnalyticsEngine (boot-safe, opt-in) - Suppress Android XA0141 (ORT native 16KB page-size advisory, upstream)

- AutoRecordWindow: pure start/extend/stop state machine with cooldown - AutoRecordCoordinator: detections -> RecordingService start, 1Hz quiet-stop tick - Only stops recordings it started; manual recordings are never auto-stopped - Graceful stop via RecordingService (avoids corrupt MP4); Failed event for logging - AutoRecordWindow unit tests (extend, quiet-stop, cooldown)

- New EventKind.Detection; MotionTick carries optional Kind + Summary - EventIngestionService threads kind/summary into opened + finalized events - AnalyticsMotionEventSource bridges engine results -> motion ticks (debounce, quiet-close, live observable reused) - Detection summary "person ×2, car ×1"; summary unit tests

- DetectionOverlay control: scales normalized boxes to tile, per-class colors + labels - CameraTileViewModel attaches/detaches frames per session; exposes Detections + counter - Analytics pauses with Smart Pause (suspended/non-playing tile -> isActive false) - AnalyticsBootstrap: lazy engine init + auto-record start on first enabled tile (best-effort) - GridPage: overlay above video + bottom-center object counter

- Camera editor: enable, class toggles (curated COCO subset), threshold, fps, auto-record + post-event - Hydrates/persists AnalyticsSettings through New/UpdateCameraRequest - Settings: "Force CPU for AI analytics" toggle wired to AiAcceleration - EN/RU localization for the new strings

- AnalyticsPage: engine status + diagnostics, analytics-enabled cameras, recent detection events - 1Hz diagnostics poll bound to the view's visual lifetime - Nav entry (desktop sidebar) + VM registration + DataTemplate; EN/RU strings

- OpenIPC.Viewer.Analytics.Tests: engine attach/detach wiring with a fake detector - Skippable real-model ONNX integration test gated on OPENIPC_DETECTION_MODEL - Mark Phase 15 done in docs/ROADMAP.md

- IconRadar is a stroke-designed Lucide path; PathIcon filled its open arcs into a blob - Use Path.lucide (Fill=Transparent + stroke) per the icon convention

- GridPage rebuilds a live tile when its analytics settings change, so enabling detection in the editor takes effect without an app restart (tile held a stale Camera snapshot, so it never attached to the engine) - BottomNavBar: add the AI entry (6 columns) so Android can reach the control center

- Confirmed the upstream 0.1.1rc0 asset downloads (~20MB); pin its SHA-256 so the cached/downloaded model is integrity-checked

- New AnalyticsEngineStatus (NotStarted/Preparing/Loading/Ready/Failed) on IAnalyticsEngine - Engine sets it through init: Preparing (model cache/download) -> Loading -> Ready, Failed on error - AI page Overview shows a localized Status row (e.g. "Downloading model…", "Ready", "Failed (see logs)") - EN/RU strings

keyldev added 13 commits June 19, 2026 21:43

feat(analytics): AI control center page (15.7)

2998818

- AnalyticsPage: engine status + diagnostics, analytics-enabled cameras, recent detection events - 1Hz diagnostics poll bound to the view's visual lifetime - Nav entry (desktop sidebar) + VM registration + DataTemplate; EN/RU strings

test(analytics): engine tests + integration scaffold; roadmap (15.8)

d2f2599

- OpenIPC.Viewer.Analytics.Tests: engine attach/detach wiring with a fake detector - Skippable real-model ONNX integration test gated on OPENIPC_DETECTION_MODEL - Mark Phase 15 done in docs/ROADMAP.md

fix(analytics): render AI nav icon as Lucide stroke, not filled PathIcon

270c6e3

- IconRadar is a stroke-designed Lucide path; PathIcon filled its open arcs into a blob - Use Path.lucide (Fill=Transparent + stroke) per the icon convention

chore(analytics): pin verified YOLOX-tiny SHA-256

265ffc0

- Confirmed the upstream 0.1.1rc0 asset downloads (~20MB); pin its SHA-256 so the cached/downloaded model is integrity-checked

keyldev merged commit c8340e5 into main Jun 19, 2026
6 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: local AI analytics (ONNX object detection)#23

feat: local AI analytics (ONNX object detection)#23
keyldev merged 13 commits into
mainfrom
feat/phase-15-ai-analytics

keyldev commented Jun 19, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

keyldev commented Jun 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Related

Architecture

Model & privacy

Platforms / execution providers

Tests

How to try

Known limitations / follow-ups

Type

Checklist

Platforms tested

Screenshots / notes

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

keyldev commented Jun 19, 2026 •

edited

Loading