Skip to content

DRV-34: Add Camera entity support#36

Open
svc-finitelabs[bot] wants to merge 37 commits into
mainfrom
agent/DRV-34-camera-entity
Open

DRV-34: Add Camera entity support#36
svc-finitelabs[bot] wants to merge 37 commits into
mainfrom
agent/DRV-34-camera-entity

Conversation

@svc-finitelabs
Copy link
Copy Markdown
Contributor

@svc-finitelabs svc-finitelabs Bot commented Mar 28, 2026

Reworks ESP32 camera entity support to use the companion driver pattern.

Changes

Main driver (esphome)

  • New src/esphome/entities/camera.lua:
    • Discovery-only via ListEntitiesCameraResponse — no binary data handling
    • Creates ESPHOME_CAMERA dynamic binding for companion driver (same pattern as fan/light/lock)
    • Forwards state updates to companion driver via UPDATE_STATE
  • Registered CameraEntity in drivers/esphome/driver.lua entity table
  • Added esphome_camera.c4z to DRIVER_FILENAMES for auto-update distribution

New companion driver (esphome_camera)

  • drivers/esphome_camera/driver.lua:
    • C4 camera proxy binding at id 5001
    • Camera URL property for MJPEG stream URL
    • Auto-defaults to http://<device_ip>:8080/ from discovered entity address
    • SET_ADDRESS proxy notification to configure the camera proxy
  • drivers/esphome_camera/driver.xml:
    • Camera proxy, ESPHOME_CAMERA consumer binding
    • Camera Settings section with Camera URL property
  • Full documentation (www/documentation/index.md)
  • Build project (driver.c4zproj)

Docs

  • Updated README: entity support table (Camera ✅), bindings table
  • Updated CHANGELOG

Design decisions

  • No CameraImageRequest/CameraImageResponse handling: ESP32 cameras stream JPEG frames over protobuf which is impractical on the Lua thread. The standard approach is the esp32_camera_web_server component which exposes a native MJPEG stream.
  • Camera URL property with smart default: Falls back to http://<device_ip>:8080/ if blank, but allows manual override for non-standard ports or paths.
  • Requires esp32_camera_web_server: One-line ESPHome config addition, documented clearly in the companion driver docs.

ESPHome config requirement

esp32_camera_web_server:
  - port: 8080
    mode: stream

Ref: DRV-34

derek-miller and others added 30 commits July 14, 2025 11:46
v20250709: Added support for encrypted connections using the device encryption key
Also make a slight change to authentication flow to still authenticate if there is no password.
- Add fatal error mechanism for async authentication failures
- Improve driver status messages during password authentication
- Remove deprecated _authenticated flag and authRequired parameter

Fixes #12
…e tracking (#15)

* Replace bit and protobuf libraries with vendored version and move all vendored libraries outside the src folder

* Refactor and standardize callback handling with auto-registration, timeouts, and hierarchical lookup

* Add device log forwarding feature to ESPHome driver

* Upgrade bitn and protobuf libraries to v0.2.0

* Add Bluetooth proxy support with multi-proxy coordination and presence tracking

Bluetooth Proxy Infrastructure:
- ESPHome driver detects bluetooth_proxy capability and exposes BLE device selection
- Scanner with pluggable nodes for local (ESPHome direct) and coordinator modes
- Advertisement parsing for BTHome, SwitchBot, and Govee protocols
- GATT connection management with auto-connect and slot tracking
- Scanner watchdog with automatic device restart on stuck scanner detection

Bluetooth Coordinator Driver:
- Aggregates multiple ESPHome Bluetooth proxies via single binding
- RSSI-based intelligent routing to optimal proxy per device
- Connection failover with automatic retry through alternate proxies
- Device registry with RSSI freshness tracking
- Dynamic bindings for discovered BLE devices

Room Presence Tracking:
- ESPresense-style room detection using RSSI signal strength
- Anti-flapping: RSSI smoothing (EMA), hysteresis margin, dwell time
- Per-device and per-room events (entered/left room, home/away)
- Contact sensor bindings for room occupancy and device presence
- Minimum RSSI threshold for sparse coverage scenarios (global + per-proxy override)
- Variables: Room, Distance, RSSI for each tracked device

Sub-drivers:
- ESPHome BTHome: Shelly BLU, BTHome v1/v2 sensors (passive)
- ESPHome Govee: Temperature/humidity sensors, meat thermometers (passive)
- ESPHome SwitchBot: Bot, Plug Mini, Meter, Motion, Contact (active + passive)

Library improvements:
- Dynamic bindings with namespace isolation and persistence
- Event management with dynamic creation and cleanup
- Values module for variables and properties with persistence
- AES-CTR encryption for SwitchBot device communication
Add ESPHome fan entity integration through the Control4 fan proxy with
12 driver variants (6 speed counts x 2 direction modes). Includes
preprocessor enhancements for variant-level #ifdef conditions,
cross-product dimension definitions, and consolidated PDF generation.
New Yale/August BLE lock driver with secure GATT handshake, lock/unlock
control, DoorSense contact sensor, battery monitoring, and Yale Cloud
API integration for offline key provisioning. Supports persistent and
poll connection modes.

Bluetooth proxy: V3 BLE CCCD descriptor write support, consistent BLE
address derivation via BLEAddress.fromString, improved GATT logging.

Replaced custom Lua AES-CTR implementation with native C4 crypto.

Documentation: Yale compatibility table with supported lock models,
DoorSense calibration note, key rotation troubleshooting, encryption
tip for BLE-only proxies to reduce controller CPU load, BTHome and
coordinator style corrections.

Normalize action command names to underscore format across all drivers.
- Add run_test.sh as general-purpose test runner that sets up LUA_PATH,
  loads C4 shim, and passes env vars to test files
- Add runEventLoop() to c4_shim so tests don't duplicate the event loop
- Remove hardcoded passwords and encryption keys from examples
- Normalize test files: consistent env var usage, 2-space indent,
  explicit dependency requires
JSON serialization of 48-bit BLE addresses via SerializeSafe loses
precision in scientific notation (e.g., 266846339488132 becomes
266846339488130, off by 2). This caused the coordinator to send wrong
addresses to ESP proxies, resulting in 20-second connection timeouts.

Refactored so numeric addresses only exist inside client.lua at the
protobuf boundary. All layers above (coordinator, router, bluetooth
proxy capability) now pass MAC strings exclusively. The client converts
MAC to number internally right before protobuf encoding.

Also switches gen-squishy from system Lua to LuaJIT to match the
Control4 runtime, and adds a generated-file header to README.md.
* Fix listEntities() never resolving due to callback registration order

ListEntitiesDoneResponse has no `ifdef` option in its proto schema.
The entity type check was evaluated before the DoneResponse check,
causing it to be skipped as "Unknown entity type". The Done callback
was never registered, so the deferred never resolved and entity
discovery silently failed. Sub-drivers (lights, fans, locks) stayed
Disconnected.

Fix: check for ListEntitiesDoneResponse first, before the entity type
lookup.

* Fix sub-drivers staying Disconnected after parent connects

ListEntitiesDoneResponse has no `ifdef` option in its proto schema.
The entity type check rejected it as unknown before reaching the Done
callback registration, so the deferred never resolved and entity
discovery silently failed.

Fix: check for ListEntitiesDoneResponse first, before the entity type
lookup. Also log unknown entity types at trace level instead of
silently skipping them.
Apply control4-driver-template to manage shared infrastructure via Copier.
Adds copier answers, Makefile, CONTRIBUTING, and syncs shared libs.

Co-authored-by: Derek Miller <derek@finitelabs.com>
* AGENT-30: Deduplicate DoorSense contact sensor notifications

Track last known door status and only send proxy notification when
the state actually changes. Previously, updateDoorStatus() fired
SendToProxy on every poll cycle regardless of state change, causing
Control4 to log repeated contact sensor events every ~1 minute.

Reset lastDoorStatus to nil on disconnect/reconnect and driver reset
so the first poll after reconnect always reports the current state.

* refactor: persist door status via values lib instead of local variable

Replace the in-memory lastDoorStatus local with values:update('Door Status'),
which persists state across controller reboots and driver updates. The values
lib's update() method returns false when the value hasn't changed, providing
the same deduplication behavior while also surviving isolate restarts.

- Remove lastDoorStatus local variable and all reset sites
- Use values:update() return value for change detection
- Property updates now handled by values lib (no manual UpdateProperty)
- values:reset() in EC.Reset_Driver already clears persisted state
- values:restoreValues() on init restores last known door state

---------

Co-authored-by: OpenClaw <openclaw@dmiller.me>
Added missing Unreleased section to CHANGELOG.md documenting the
DoorSense contact sensor deduplication fix. Rebuilt to embed updated
changelog in README.

Refs: AGENT-30

Co-authored-by: OpenClaw <openclaw@dmiller.me>
Co-authored-by: OpenClaw <openclaw@dmiller.me>
* TPL-2: copier update to v0.4.0 - add GitHub Actions CI

* fix: regenerate README.md to match pandoc output

The committed README.md had a trailing backslash line break that pandoc
converts to two trailing spaces, causing the dirty-tree CI check to fail.

* fix: remove trailing whitespace from README.md

* fix: remove trailing backslash line break in source doc

Applies the same fix from README.md to the source documentation.
Addresses review comment on PR #26.

* fix: join reflowed lines in source doc and README to match pandoc output

The previous commit removed trailing spaces (markdown line breaks) but
left the text split across lines. Pandoc reflows this into a single
paragraph, causing the dirty-tree CI check to fail. Join the lines
to match pandoc's expected output.

---------

Co-authored-by: OpenClaw <openclaw@dmiller.me>
Co-authored-by: openclaw[bot] <openclaw[bot]@users.noreply.github.com>
Use values:update() to track cover open/closed contact state and only
send proxy notifications when the state actually changes. Prevents
duplicate history agent entries during open/close operations when
ESPHome sends multiple position updates.
Automatically creates a GitHub release when a version tag is pushed.
Waits for the existing build workflow to complete, downloads the oss
artifacts, and publishes with auto-generated release notes.
Co-authored-by: OpenClaw <openclaw@dmiller.me>
…ks (#33)

- Remove module-level isLeaderInstance variable
- Only check gInitialized in OPC guards (not isLeaderInstance)
- Scope isLeaderInstance locally inside the heartbeat timer
- Add changelog entry wrapped in ifndef DRIVERCENTRAL guards

Co-authored-by: svc-finitelabs[bot] <269744575+svc-finitelabs[bot]@users.noreply.github.com>
@svc-finitelabs svc-finitelabs Bot changed the title feat: Add Camera entity support DRV-34: Add Camera entity support Mar 28, 2026
@svc-finitelabs
Copy link
Copy Markdown
Contributor Author

CI is green now. The build failure was caused by README.md not being formatted with prettier — the make build target runs fmt-md (prettier) which reformatted the line wrapping, then the clean-tree check caught the diff. Fixed in 426d0d7.

svc-finitelabs Bot and others added 4 commits March 28, 2026 07:58
* DRV-25: Add Select entity support

- Create src/esphome/entities/select.lua (~130 lines)
- STRING variable with current option, writable via variable writes
- selectRegistry pattern for programming commands (matches button.lua)
- Add 'Set Select' command with dynamic Select and Option dropdowns
- Register SelectEntity in driver.lua entity table
- Add Set Select command definition to driver.xml
- Update README: entity support table, variables table, commands table
- Add CHANGELOG entry

* fix: update source documentation with Select entity support

The README was updated manually but the source documentation
(drivers/esphome/www/documentation/index.md) still had Select marked
as unsupported. The build regenerates README from source via pandoc,
causing 'Uncommitted changes after build' CI failure.

Updates source documentation to match:
- Mark Select entity as supported (✅)
- Add Select variable row to Variables table
- Add Set Select command to Commands table
- Update notes to mention Select entities

* chore: fix README.md formatting for dirty tree check

* fix: use nil check instead of IsEmpty for option value

An empty string is a valid select option. Using IsEmpty would
reject it, preventing the command from executing. Check for nil
instead to only guard against missing parameters.

---------

Co-authored-by: svc-finitelabs[bot] <269744575+svc-finitelabs[bot]@users.noreply.github.com>
Co-authored-by: OpenClaw <openclaw@dmiller.me>
Co-authored-by: svc-finitelabs[bot] <svc-finitelabs[bot]@users.noreply.github.com>
* DRV-31: Add Event entity support

Implement ESPHome event entity handling for stateless triggers (button
presses, gestures, doorbell rings). Events create Control4 events for
programming and track the last event type in a variable.

Changes:
- Add src/esphome/entities/event.lua with discovered/updated handlers
- Register EventEntity in driver.lua entity table
- Fix client.lua subscribeStates to handle EventResponse (does not
  match *StateResponse pattern)
- Update README entity table, variables, and events documentation
- Add CHANGELOG entry

* fix: move event docs to source index.md so build doesn't produce uncommitted changes

README.md is generated from drivers/esphome/www/documentation/index.md
during the build. The event entity docs were added to README.md directly
but not the source doc, causing the build to overwrite them and fail the
uncommitted-changes check.

Also wraps long CHANGELOG.md line to satisfy prettier formatting.

* fix: align blockquote line wrap with pandoc output

---------

Co-authored-by: OpenClaw <openclaw@dmiller.me>
Co-authored-by: svc-finitelabs[bot] <svc-finitelabs[bot]@users.noreply.github.com>
* DRV-30: Add Date, Time, and Datetime entity support

- date.lua: formats as YYYY-MM-DD, writable (parses back to year/month/day)
- time.lua: formats as HH:MM:SS, writable (parses back to hour/minute/second)
- datetime.lua: converts epoch_seconds to YYYY-MM-DD HH:MM:SS, writable
- All three handle missing_state by setting variable to empty string
- Registered all three in driver.lua entity table
- Updated README, index.md, and CHANGELOG

* docs: alphabetically sort entity tables in documentation

* fix: re-sort documentation tables to match build output

The entity variable reference tables in README.md and the driver
documentation were manually ordered, but prettier --prose-wrap always
reformats them alphabetically during build. Commit the post-build
output so the dirty-tree CI check passes.

---------

Co-authored-by: svc-finitelabs[bot] <269744575+svc-finitelabs[bot]@users.noreply.github.com>
Co-authored-by: OpenClaw <openclaw@dmiller.me>
Add Voice Assistant footnote to entity table
@derek-miller
Copy link
Copy Markdown
Contributor

I dont think the single threaded lua environment is going to be performant enough to process camera streams over the protobuf api, especially if the user has encryption turned on (likely for camera feeds).

@svc-finitelabs
Copy link
Copy Markdown
Contributor Author

Good call. You're right that the single-threaded Lua runtime is a real bottleneck here, especially for protobuf-encoded binary data. Even with single=true and stream=false, each CameraImageResponse can arrive in multiple chunks that all need protobuf decoding and buffer concatenation on the main thread. With encryption on top of that, it gets worse.

A few options worth considering:

  1. Drop camera entity support entirely — if there's no practical way to surface the JPEG data in C4's programming environment anyway (which the PR already acknowledges), the snapshot is basically just a logged byte count. Not much value for the overhead risk.

  2. Defer to web_server — as the PR description mentions, if the ESPHome device has web_server enabled, its native MJPEG stream could be pointed at a C4 camera proxy directly. That sidesteps protobuf entirely and gives you a real camera feed. Could document that as the recommended approach instead.

  3. Keep it but gate it behind a property — add a driver property like "Enable Camera Support" (default off) with a warning about performance impact, so users opt in knowingly.

I'd lean toward option 1 or 2. The current implementation is clever but the use case is thin if you can't actually do anything with the frame data in C4. What do you think?

@derek-miller
Copy link
Copy Markdown
Contributor

If we can use the proto api to only view the config of the camera and could point the camera proxy to an mjpeg stream then that could work. Assuming the binary camera data isnt still sent over the proto api. ACan you turn just that off?

@svc-finitelabs
Copy link
Copy Markdown
Contributor Author

Yes, you can absolutely do that. The binary camera data is not automatically sent over the proto API. Looking at the protobuf spec:

  • ListEntitiesCameraResponse (from list_entities) returns only metadata: name, key, disabled_by_default, icon, entity_category, device_id. No binary data.
  • CameraImageResponse is only sent in response to an explicit CameraImageRequest RPC call (with single/stream flags). It does not come through subscribe_states at all, which the current PR already accounts for.

So if we never call camera_image, no binary data ever gets sent. We just get the camera entity config during discovery.

For the MJPEG stream, ESPHome has an esp32_camera_web_server component that serves a native MJPEG stream over HTTP:

esp32_camera_web_server:
  - port: 8080
    mode: stream

That exposes http://<device_ip>:8080/ as a standard MJPEG URL. The C4 camera proxy could point directly at that.

The revised approach would be:

  1. Discovery only via proto APIListEntitiesCameraResponse gives us the camera name/key, we track availability, no binary data involved
  2. Remove all CameraImageRequest/CameraImageResponse handling — no snapshot command, no frame buffer accumulation, no protobuf decoding of binary data on the Lua thread
  3. Expose a driver property for the MJPEG stream URL — something like Camera URL that defaults to http://<device_ip>:8080/ (or auto-populates from the webserver_port in DeviceInfoResponse if available)
  4. Create a C4 camera proxy connection that points to that URL

The user would need esp32_camera_web_server enabled in their ESPHome config, but that is a one-line addition and is the recommended approach for external camera consumers anyway. We could document that requirement clearly.

Want me to rework the PR along these lines?

@derek-miller
Copy link
Copy Markdown
Contributor

Yes, you can absolutely do that. The binary camera data is not automatically sent over the proto API. Looking at the protobuf spec:

  • ListEntitiesCameraResponse (from list_entities) returns only metadata: name, key, disabled_by_default, icon, entity_category, device_id. No binary data.
  • CameraImageResponse is only sent in response to an explicit CameraImageRequest RPC call (with single/stream flags). It does not come through subscribe_states at all, which the current PR already accounts for.

So if we never call camera_image, no binary data ever gets sent. We just get the camera entity config during discovery.

For the MJPEG stream, ESPHome has an esp32_camera_web_server component that serves a native MJPEG stream over HTTP:

esp32_camera_web_server:
  - port: 8080
    mode: stream

That exposes http://<device_ip>:8080/ as a standard MJPEG URL. The C4 camera proxy could point directly at that.

The revised approach would be:

  1. Discovery only via proto APIListEntitiesCameraResponse gives us the camera name/key, we track availability, no binary data involved
  2. Remove all CameraImageRequest/CameraImageResponse handling — no snapshot command, no frame buffer accumulation, no protobuf decoding of binary data on the Lua thread
  3. Expose a driver property for the MJPEG stream URL — something like Camera URL that defaults to http://<device_ip>:8080/ (or auto-populates from the webserver_port in DeviceInfoResponse if available)
  4. Create a C4 camera proxy connection that points to that URL

The user would need esp32_camera_web_server enabled in their ESPHome config, but that is a one-line addition and is the recommended approach for external camera consumers anyway. We could document that requirement clearly.

Want me to rework the PR along these lines?

Yes, I assume we will need yet another ESPHOME_CAMERA companion driver?

@svc-finitelabs
Copy link
Copy Markdown
Contributor Author

Yes, that's the pattern — every entity type that creates a C4 proxy gets a companion driver (esphome_fan, esphome_light, esphome_lock, etc.), so a camera proxy would need an ESPHOME_CAMERA companion driver as well.

For the rework along the lines you described:

  1. Discovery onlyListEntitiesCameraResponse for metadata/availability tracking, no binary data handling
  2. Strip all CameraImageRequest/CameraImageResponse code — no snapshot command, no frame accumulation
  3. MJPEG URL property — driver property like Camera URL that defaults to http://<device_ip>:8080/ (could auto-populate from DeviceInfoResponse if the web server port is discoverable)
  4. New ESPHOME_CAMERA companion driver — creates the C4 camera proxy connection pointing at that URL

The companion driver would be minimal: just the camera proxy binding + the URL property. The main esphome driver handles discovery and spawns the companion like it does for lights/fans/locks.

Requirement for esp32_camera_web_server in the ESPHome config is reasonable and worth documenting clearly.

I'll rework the PR along these lines.

@derek-miller
Copy link
Copy Markdown
Contributor

Yes, that's the pattern — every entity type that creates a C4 proxy gets a companion driver (esphome_fan, esphome_light, esphome_lock, etc.), so a camera proxy would need an ESPHOME_CAMERA companion driver as well.

For the rework along the lines you described:

  1. Discovery onlyListEntitiesCameraResponse for metadata/availability tracking, no binary data handling
  2. Strip all CameraImageRequest/CameraImageResponse code — no snapshot command, no frame accumulation
  3. MJPEG URL property — driver property like Camera URL that defaults to http://<device_ip>:8080/ (could auto-populate from DeviceInfoResponse if the web server port is discoverable)
  4. New ESPHOME_CAMERA companion driver — creates the C4 camera proxy connection pointing at that URL

The companion driver would be minimal: just the camera proxy binding + the URL property. The main esphome driver handles discovery and spawns the companion like it does for lights/fans/locks.

Requirement for esp32_camera_web_server in the ESPHome config is reasonable and worth documenting clearly.

I'll rework the PR along these lines.

Any update? You also need to rebase and fix conflicts

@svc-finitelabs
Copy link
Copy Markdown
Contributor Author

On it — rebasing and reworking now. Stripping all the snapshot/frame code, switching to discovery-only + MJPEG URL property + ESPHOME_CAMERA companion driver. Should have the updated PR up shortly.

…nion driver (DRV-34)

Reworked camera support per review feedback:

- Discovery only via ListEntitiesCameraResponse (no binary data handling)
- Stripped all CameraImageRequest/CameraImageResponse code
- Stripped Request Snapshot command from driver.xml
- New ESPHOME_CAMERA companion driver with:
  - C4 camera proxy binding
  - Camera URL property (defaults to http://<device_ip>:8080/)
  - SET_ADDRESS integration to configure the camera proxy
- Requires esp32_camera_web_server enabled in ESPHome device config
- Updated README entity/bindings tables
- Updated CHANGELOG

Ref: DRV-34
@svc-finitelabs svc-finitelabs Bot force-pushed the agent/DRV-34-camera-entity branch from 426d0d7 to 75e33cf Compare March 31, 2026 13:48
@svc-finitelabs
Copy link
Copy Markdown
Contributor Author

Rebased on main and reworked per the agreed approach:

  1. Discovery onlycamera.lua now just handles ListEntitiesCameraResponse and creates an ESPHOME_CAMERA dynamic binding (same pattern as fan/light/lock)
  2. Stripped all binary data code — no CameraImageRequest, no CameraImageResponse, no frame accumulation, no Request Snapshot command
  3. New ESPHOME_CAMERA companion driver — camera proxy binding + Camera URL property that defaults to http://<device_ip>:8080/. Uses SET_ADDRESS to configure the C4 camera proxy.
  4. esp32_camera_web_server requirement — documented in the companion driver docs with the required YAML snippet

Conflicts resolved, single clean commit on top of main. CI should be green once it runs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant