Design proposal for V2 of ESPHome components

[kpfleming]

I mentioned to Sam recently that I'm motivated (and have time) to do some work on a redesign of the ESPHome components. I've worked on a number of other ESPHome components and contributed to various parts of the platform, and I also hang out in the 'dev' Discord channel so I'm familiar with the types of things that the core team likes and dislikes. While it may never be useful for the MITP components to be incorporated upstream, I'd personally like to see them be built in a way that they could be.

My goals for this redesign are:

1. Ensure that the Python (configuration) and C++ (implementation) code is internally consistent (naming, interfaces, etc.) and broken into logical pieces which can be reviewed on their own.
2. Remove the diagnostic sensors from the climate component since the core team prefers the climate component to be focused on its own functionality; these sensors should be made available by a separate component.
3. Remove bridging from the climate component for the same reasons; this functionality should be made available by a dedicated component.
4. Support a wider variety of configuration styles - for example some users may want to only monitor their heatpump/thermostat and not have a 'climate' component at all (avoiding any risk of setpoints being changed by HA or ESPHome).
5. Provide a mechanism to support my future 'mitp-failsafe' hardware design, where the heatpump and thermostat are not connected directly to ESP UARTs but instead are proxied by another MCU.
6. Provide a mechanism to support integration testing by incorporating simulated heatpumps/thermostats into the configuration.

MITP 'links'

The lowest layer of the configuration and implementation are MITP 'links' - communications channels where MITP packets are sent and received between the ESP and some other device.

An example configuration for MITP-over-UART could be:

mitp_link:
  - id: hp_link
    platform: mitp_link_uart
    uart: hp_uart
  - id: th_link
    platform: mitp_link_uart
    uart: th_uart

This would create two mitp_link components which operate by sending and receiving ITP packets; in the C++ code the interface to these components would be the packets defined in the itp-packet library. Since these are UART links, they would handle framing, encoding, decoding, etc. for the ITP-over-UART protocol.

Alternative implementations of mitp_link would have the same C++ interface but use a different implementation. Examples could include 'mock' components which simulate a heatpump or thermostat; 'TCP' components which support links over network connections; and my 'failsafe' component which would expose two links over one UART, using Protocol Buffers for the packet communication.

MITP 'devices'

The next layer are devices: heatpumps and thermostats.

An example configuration for these could be:

mitp_device:
  - id: hp
    type: heatpump
    mitp_link_id: hp_link
    (fan modes, operating modes, etc. go here)
  - id: th
    type: thermostat
    enhanced_mhk: true/false
    time_id: (id of time component, auto-detected in most cases)

The various sensors which are not related to the climate component would be exposed by these device components: It is possible during the implementation that it may be necessary to make separate mitp_heatpump and mitp_thermostat components instead of an mitp_device component with a type attribute.

MITP 'bridging'

If the user wants to use both a heatpump and thermostat, and bridge between them, they'd use a configuration like this:

mitp_bridge:
  - id: bridge
    mitp_heatpump_id: (auto-detected in most cases)
    mitp_thermostat_id: (auto-detected in most cases)
    (any bridge-specific attributes go here)

MITP 'climate'

Finally, there is the climate component. It would be able to operate off of either a heatpump or a bridge.

climate:
  - id: hp_direct
    platform: mitp_device (or mitp_heatpump)
    mitp_heatpump_id: (auto-detected in most cases)
    (climate component attributes go here)
  - id: bridged
    platform: mitp_bridge
    mitp_bridge_id: (auto-detected in most cases)
    (climate component attributes go here)

MITP 'sensor', 'binary_sensor', 'text_sensor', 'button', 'select'

All of these would look as they do today, except the platform names would change as they'd reference the new components.

Implementation plan

I would do this work in my own fork of the ESPHome core repository, lifting code from the esphome-components repository in this organization as needed. Initially I'd focus on the Python code, getting the schemas built and parsed properly, with very minimal to_code implementations that just instantiate the proper C++ classes but without any implementation code in those classes. Once that is ready for review I'll ask for testers and feedback, to get more people building example configurations that meet their needs and ensuring that the Python code will accept them.

Assuming that phase goes well, I'd then start copying around the C++ code into the new components, starting at the bottom layer and working my way up. I'll use my own systems for testing (carefully, of course!) and when it's ready for beta testing by others I'll ask for testers again.

Ongoing support

I'm happy to commit to being around for the long-term to support this work, as long as my health and other personal circumstances permit it. I've got a very long history in the open source community and have a number of my own projects that I support (including an ESPHome component for publishing data to InfluxDB) so I know what's required and I plan to do it for as long as I can.

[samphonic]

Some thoughts (obviously just my three cents, so this is not meant to be prescriptive), but in summary: I think these are pretty much all good improvements, but I'm not sure starting a new codebase is needed/best.

Ensure that the Python (configuration) and C++ (implementation) code is internally consistent (naming, interfaces, etc.) and broken into logical pieces which can be reviewed on their own.

Naming consistency could be better, but I think things are fairly broken up (with the exception of the sensors below). This one would certainly be easier to do with a fresh codebase, but worth doing either way.

Remove the diagnostic sensors from the climate component since the core team prefers the climate component to be focused on its own functionality; these sensors should be made available by a separate component.

MITP 'devices'

This is basically already the case; the Python for the sensors is all separate, and each sensor is just a listener that subscribes to updates from the packet handler (packet handling should probably be broken out though, see next point).

I do have mixed feelings about this as a user though. I don't really like when adding a single hardware device to Home Assistant creates a bunch of different "devices" that I have to then manually group back together if I want an overview for the status of a single thing. Having a separation of concerns in the code is good, but at least by default I would prefer if the sensors all remained part of the "climate" device in Home Assistant (even breaking out the thermostat information is questionably helpful I would say, but I could see that being more useful).

Remove bridging from the climate component for the same reasons; this functionality should be made available by a dedicated component.

MITP 'links'

Hard agree and: we should go a step further and make a generic UART packet handler bus. The queuing, parsing, routing of packets should be a separate concern from adding utility functions to say "this byte represents fan speed". I think like the itp-packet information was broken out, a UART-packet-protocol (or whatever) library would be useful, and then all that would be required in the MITP code is the binding that says how raw packets are assigned to concrete implementations.

The climate, sensor, and other components (thermostat) could just subscribe and publish to this bus, keeping them all well-separated.

Support a wider variety of configuration styles - for example some users may want to only monitor their heatpump/thermostat and not have a 'climate' component at all (avoiding any risk of setpoints being changed by HA or ESPHome).

We actually started out having this, but no one wanted it and so it was removed to reduce complexity. If anyone decides they actually want this, they can make the climate component internal in the config, and proxy the built-in sensors. If that's something that comes up a lot we could provide some example config.

Provide a mechanism to support my future 'mitp-failsafe' hardware design, where the heatpump and thermostat are not connected directly to ESP UARTs but instead are proxied by another MCU.

Definitely.

Provide a mechanism to support integration testing by incorporating simulated heatpumps/thermostats into the configuration.

This has actually been on the to-do list for a long time. I think the appropriate way to do this is a separate ESPHome component that simulates a heat pump. For testing it could be connected virtually in the config to the climate component, but it could also be installed on a separate board to simulate a heat pump over CN105.

MITP 'bridging'

This, I think, will be a challenge to do in a way that doesn't get overly complex; but is also making be think the current "bridge" is confusingly named. The current "bridge" component is a bridge between UART and ITP, not a bridge between the heat pump and thermostat.

Passing packets between the thermostat and heat pump is handled by the MITP component because it's not always straight forward. Some packets are only forwarded one-way, some are blocked from being forwarded, and the current state of the climate component may affect which action is taken for a give packet (SetRemoteTemperature is a prime example of these, though in the latest update things have been simplified by never forwarding those particular packets at all).

---

In medium-length summary, my thoughts are:

• UART packet queuing, routing, and parsing should its own component/library like ITP-packet.
• Support for your failsafe board should be fairly easy to incorporate by using the above.
• Simulated heat pumps / thermostats should probably be their own separate things.
• Logic around what gets sent to/from the thermostat/heat pump is not super straightforward. It's probably not impossible to create a generic component to connect these things and allow users to turn on and off different features, but some of the logic is closely linked to the climate component and that might make it simpler to just keep the logic in the core MITP component.

[kpfleming]

I do have mixed feelings about this as a user though. I don't really like when adding a single hardware device to Home Assistant creates a bunch of different "devices" that I have to then manually group back together if I want an overview for the status of a single thing. Having a separation of concerns in the code is good, but at least by default I would prefer if the sensors all remained part of the "climate" device in Home Assistant (even breaking out the thermostat information is questionably helpful I would say, but I could see that being more useful).

At the moment the standard configuration produces one 'device' with a large number of entities, and those entities represent separate logical devices. The new sub-device support in ESPHome is the solution for that - in my MUART configuration I have the 'bridge' device separate from the heatpump and the thermostat. In other words this can be done directly in the YAML, which results in explicit mapping to devices instead of implicit mapping, and this allows an advanced user to decide how they want the mapping done. The default will be what it is today, all entities connected to the 'base' device produced by ESPHome.

Hard agree and: we should go a step further and make a generic UART packet handler bus. The queuing, parsing, routing of packets should be a separate concern from adding utility functions to say "this byte represents fan speed". I think like the itp-packet information was broken out, a UART-packet-protocol (or whatever) library would be useful, and then all that would be required in the MITP code is the binding that says how raw packets are assigned to concrete implementations.

Yes! That's what I've been thinking too - the UART encoding/decoding needs to be separate from the protocol packets. Essentially layer 1 vs. layer 2.

This has actually been on the to-do list for a long time. I think the appropriate way to do this is a separate ESPHome component that simulates a heat pump. For testing it could be connected virtually in the config to the climate component, but it could also be installed on a separate board to simulate a heat pump over CN105.

Agreed again: I'm thinking the existing ITP 'simulator' could be wrapped up in a 'link' component, and it would just act as if that link was connected to a real heatpump. This would presumably even allow tests to be run using the 'host' architecture in ESPHome.

I'm off on vacation for the next week and will have at least an hour every morning between the time I wake up and when my partner wakes up, so I'll start noodling around with the Python parts of this.