P183 modbus RTU

[flashmark]

Initial version of a simple Modbus RTU plugin. This plugin handles a Modbus device as a set of holding registers. Up to 4 holding registers can be read into the 4 values of the plugin. The plugin number is 183 as agreed elsewhere.

[TD-er]

Can you look at the timing stats for this plugin you added?
I suspect the reading like this will take 100's of msec per register read.

Please take a look at what I did for the Eastron plugin, where I set a queue of registers to read.
This way the waiting time per call is nihil and thus not blocking the rest of the ESPeasy functionality.

My intention is to make all ESPEasy plugins using modbus use this approach to share access to the same Modbus bus.

Also the ESPEasySerial for (nearly) all plugins is using the same set of PCONFIG() for the serial config.
I did not check if you did this, so please make sure to use the same way as in the most recent plugins using a serial port.

[flashmark]

I have been busy for some time. Try to pickup the comments soon. Indeed the exchange takes some time and I will look into creating a queue. Eastron was my inspiration, but I did not look in most of the details as it was mainly device specific handling. Serial settings were copied from Eastron. To share the modbus I need some more insights how the sharing is intended. I have too little experience with some details in ESPEasy.

[flashmark]

A small introduction of myself. I started as embedded software developer using mainly C, but I am for quite some time software architect and not doing professional coding anymore. I am definitely not a C++ expert. I am working for a large company building complex machines. I like to pickup smaller embedded software projects and domotica. And I really like the way ESPEasy is set up.

I see some issues with the current P078 implementation. The modbus and Eastron device specific features are mixed over the various "layers". There is the "plugin" that takes care of the configuration and external data (variable, config, web representation). Then the "data_struct" to put the behavior of the plugin in a C++ friendly environment (away from .ino). And there is a "Eastron library" in SDM.cpp.

If understood you well the requirements are:

• Multiple plugin instances of different plugin types can share the same physical Modbus. (Use the same link)
• Multiple links should be possible in parallel
• Modbus access shall not hold the CPU during message exchange

The P078 implementation uses a queue that can manage multiple plugins in theory. For me it is unclear how it can differentiate between multiple links. The plugin defines the serial link, if there are multiple plugins connected it seems the last plugin that initializes defines the link properties. Is this desired behavior?

The queue knows it is handling SDM messages and delivers the received holding register directly into the uservar: UserVar.setFloat(it->taskIndex, it->taskVarIndex, value);
It is a fast way to handle the data, but it makes it impossible to retrieve other data from the Modbus device.

My proposal would be to split the code into the following classes:

• Plugin & optional data_struct to handle the user interaction as standard in ESPEasy
• A Modbus class that handles one link. This includes a queue and Modbus packet coding/decoding.
• A singleton Modbus "broker" that manages a list of Modbus links. The broker handles init/terminate requests from the plugins and tries to combine requests from multiple plugins in a single Modbus link.
• Existing ESPEasy serial link class responsible for the data transport over the selected serial link

Broker and link should be outside any plugin as they can be shared by multiple plugin classes. As they are both Modbus specific they can be in a single file sharing a header file.

I still need to think about the details how to exchange data and fit the queue neatly in the design. What has to be in and how does it return results. The Modbus link should be able to handle both the RTU binary and ASCII format. It should be able to handle other Modbus message types. Both option for future only :-)

One thing to consider:
Do we want to have the serial link specified through the plugin or add Modbus links as dedicated resources to the hardware page? My preference would be to keep it as is. Will make the broker a bit more complex, but we can manage that.

[TD-er]

Well hats off to you sir, as you seem to have a very good idea of the layers we have right now :)

Right now, I am working on another pull-request to do a complete rewrite for ESPEasy's WiFi/network implementation.
This does add a "Network" tab, just like the current "Devices" and "Controllers" tab we currently have.
And when I say "just like", it really is very similar.
So the first table is showing all network adapters and a short overview of its current state/config.

When clicking "Edit" on such an adapter, you get the specific settings, very similar to how controllers and tasks are being dealt with.

My next idea for a follow-up on this is to add a tab for "Interfaces" (or buses, name is not yet clear).
This way you can define interfaces like Modbus, SPI, I2C, 1-wire, etc. (maybe also extended GPIO pins)

Especially some of those interfaces which share a bus for multiple devices, need a handler to deal with all devices and pending transactions on the bus.
And also as you rightfully mentioned, some main (singleton) handler to handle all interfaces of the same type.
For example, when requesting a read or write on a modbus device, you must make sure you have completed this before something else is requested.

This idea is already implemented (in a bit too specific way) by keeping track of a list of registers to read and where to put the result.
This does work fine for Eastron devices, as it is all the same. You call for a register and interpret the result, then store the value somewhere.
However this already has some limitation as it only assumes float values. There are however some registers which do not return a float.

So to "fix" this, I imagine there might be some new PLUGIN_MODBUS_READ_RESULT call, which then should be implemented in those plugins which support Modbus communications.
Then in the event (ESPEasy EventStruct), there should be the taskindex set and probably some other values too and a pointer to the read data.
Those plugins then should request a read to the Modbus handler from the PLUGIN_READ call.
This way the read is already way more generic.

Then adding a main handler would probably be something for a next pull-request so we can think of a more generic way to manage modbus handlers.

The main advantage with this is that there is no longer a collision when accessing modbus devices on the same bus and there is no active blocking wait for a reply from the device, which may take quite some time.
For example the SenseAir S8 may take 200+ msec to reply and currently does actively wait.
The same for the ACU28x (or how those power meters are called...) and probably also for those PZEM meters.

[flashmark]

Looks good. By the way, I am not in any hurry to push my branch. Please continue the framework and let me know where I can contribute for something modbus specific. A suggesting is to remove the word MODBUS in the event and keep is a bit more abstract like BUS_READ_RESULT. This can then be any pending bus transaction. I think that I2C could also benefit.

If we add this callback trough an event and a central bus or link administration the singleton management layer will be very simple. The plugin know the bus type and index and the manager returns the associated bus object. Maybe do some admin to check how many active plugins are coupled to the bus; and do initialization termination when the first joins or the last leaves.

By the way, I saw a Modbus_RTU file. Is this where the new bus management stuff should grow?

[TD-er]

Not likely that this will remain the (file) structure.

For the WiFi/network rewrite, I'm introducing namespaces like ESPEasy/net/wifi

The idea of having a generic bus callback/event also seems OK, as long as the bus manager/handler does keep in mind which task may be expecting specific bus responses.
Like there are certain devices which can be used via various different bus types. For example the same sensor on I2C or SPI and/or UART.
But that's something to worry about later.

[flashmark]

Sorry it is still work in progress I wanted to set aside. I had some other duties and am just picking up this project. Will update it soon with a more crisp design. Keep in mind: I am not a C++ coder, any corrections and suggestions are welcomed.
Main design separation into several classes:

• Plugin: Unique for a device and implementing the interaction with ESPEasy.
• Modbus_device: Represents one modbus device and is coupled to a plugin. Does the modbus message (de)coding.
• Modbus_link: Represents a physical modbus link (serial link). One link can serve multiple modbus_devices.
• Modbus_mgr: Connects the devices and the links. This singleton object knows the devices and the links. Main task is to create a link when the first device wants to connect and connect a device to a link on request.
• Queue: Each link has a queue of pending modbus transactions.

Main issues to resolve:

• Serial port configuration is connected to a modbus_link. But I can only configure a plugin. Workaround: last device connecting to a link determines the properties. (If the ESPEasySerialPort type is different then it is a new link, still to think how to handle multiple "software serial")
• Transaction takes too much time to wait/poll. Instead we use a queue to prepare transactions (one message exchange). As a result we need some callback mechanism to return the results.
• For the device to link relation the is a real callback is implemented as a class function and the link knows the modbus_device that queued the transaction.
• For the plugin to device relation I don't know how to properly implement a mechanism that can feed back the uint16 registers from the modbus reply. Converting them to plugin output values blurs the responsibilities of the plugin and the modbus_device. This will make reuse much more difficult (like the P078 plugin).
• To keep the link busy it needs to poll the serial link for a complete reply. For now this is done by a ten_per_second trigger from each plugin. This is a bit heavy on CPU load. I am looking for a direct triggering of the link objects. The code should be prepared for that.

Is there a way to store design documentation with a plugin?

[TD-er]

Why not use some (weak) std::shared_ptr like structure for this?

[flashmark]

Right now, I am working on another pull-request to do a complete rewrite for ESPEasy's WiFi/network implementation. This does add a "Network" tab, just like the current "Devices" and "Controllers" tab we currently have. And when I say "just like", it really is very similar. So the first table is showing all network adapters and a short overview of its current state/config.

Especially some of those interfaces which share a bus for multiple devices, need a handler to deal with all devices and pending transactions on the bus. And also as you rightfully mentioned, some main (singleton) handler to handle all interfaces of the same type. For example, when requesting a read or write on a modbus device, you must make sure you have completed this before something else is requested.

This idea is already implemented (in a bit too specific way) by keeping track of a list of registers to read and where to put the result. This does work fine for Eastron devices, as it is all the same. You call for a register and interpret the result, then store the value somewhere. However this already has some limitation as it only assumes float values. There are however some registers which do not return a float.

So to "fix" this, I imagine there might be some new PLUGIN_MODBUS_READ_RESULT call, which then should be implemented in those plugins which support Modbus communications. Then in the event (ESPEasy EventStruct), there should be the taskindex set and probably some other values too and a pointer to the read data. Those plugins then should request a read to the Modbus handler from the PLUGIN_READ call. This way the read is already way more generic.

Then adding a main handler would probably be something for a next pull-request so we can think of a more generic way to manage modbus handlers.

The main advantage with this is that there is no longer a collision when accessing modbus devices on the same bus and there is no active blocking wait for a reply from the device, which may take quite some time. For example the SenseAir S8 may take 200+ msec to reply and currently does actively wait. The same for the ACU28x (or how those power meters are called...) and probably also for those PZEM meters.

I got the Modbus engine running. Plugins can use de Modbus_device to queue transactions to the Modbus_link object. The singleton Modbus_manager manages all links and implements toe link configuration web page. Needs some cleaning, but it works.

Next task is to get rid of plugins waiting for the transaction result. Currently this only works for the plugin_read and causes a delay of one full sample time. I want to implement a "callback" using the event mechanism. Can we design a generic event for responses from autonomous modules like the Modbus link? I can imagine there are other processes that may take time to generate a value. Using a PLUGIN_MODBUS_READ_RESULT sounds very specific for the Modbus.

[TD-er]

We currently have the PLUGIN_TASKTIMER_IN, which I guess is somewhat related.
This is typically called through scheduling (see setPluginTaskTimer ) which also makes it even more 'async' in behavior.

A typical work flow can be something like this:

• Task adds request to Modbus engine's queue. (giving some priority or not)
• Modbus engine sends out next request
• Modbus engine receives reply ('happy flow' here, no timeout) and schedules a PluginTaskTimer with the response
• Modbus engine sends out next request.
• Task processes scheduled PluginTaskTimer call + adds new request to Modbus engine's queue.

N.B. the task does 'know' it is a modbus task, so there is no ambiguity in what's intended with the PLUGIN_TASKTIMER_IN

[flashmark]

I will give it a try with PLUGIN_TASKTIMER_IN. Let's do some research about the event mechanism and structure.

[TD-er]

I will give it a try with PLUGIN_TASKTIMER_IN. Let's do some research about the event mechanism and structure.

You can start by looking at systemTimerStruct to see if you can fit everything to let the task know the read result.
If not, please let me know, so we can see what should/could be extended.

[TD-er]

The value may not always be a float.
So maybe we could have a look at how to deal with those.

However a 16-bit value can still be stored in a float without loosing bits.

[flashmark]

It is indeed a 16-bit unsigned integer. Not sure how to handle those as task values. I copied this from somewhere.

[TD-er]

I just approved running your GH Actions build, just to see if there are any build issues so far.

[TD-er]

OK, that didn't run for long....

Compiling .pio/build/minimal_core_312_ESP8266_1M_OTA_FHEM_HA/src/src/Controller_config/C023_AT_commands.cpp.o
src/src/Commands_tmp/__tmpfile.cpp:91368:10: fatal error: P183_data_struct.h: No such file or directory

**************************************************************************
* Looking for P183_data_struct.h dependency? Check our library registry!
*
* CLI  > platformio lib search "header:P183_data_struct.h"
* Web  > https://registry.platformio.org/search?q=header:P183_data_struct.h
*
**************************************************************************

91368 | #include "P183_data_struct.h"
      |          ^~~~~~~~~~~~~~~~~~~~

As you may not yet know... For ESP8266 builds, all .cpp files are concatenated into a single file (well all except the new src/ESPEasy/net/.... part)
This is for 2 reasons:

• The linker would other wise fail due to too long argument list with all .cpp.o files
• The linker is better at removing unused functions resulting in smaller builds.

However this has a major consequence, where you need to make the include path a bit longer as the concatenated file will be in a different directory from the .h files.

So the #include "P183_data_struct.h" should be changed into #include "../DataStructs/P183_data_struct.h"
And probably on other includes too.

[flashmark]

Sorry, Copilot did not do a good job here...
I still need help with the defines for the KVS storage definition. I don't get how these values
I will push a new baseline, but don't consider that final...

[TD-er]

idem, missing check for nullptr

[flashmark]

Here we can argue if we want to check the precondition at some cost of execution time and memory. This is an internal function and it is called by functions that already checked the pointer. I added the function as two functions create the same Modbus message content with different context.

[TD-er]

Wouldn't it make more sense then to have the function argument as a reference and not a pointer?

When handing a pointer as a function argument, the function's responsibility is to check for a nullptr.
When handing a reference as function argument, it is the caller's responsibility to check.

[flashmark]

I still have to learn proper C++

[TD-er]

This does look quite tricky as _userData is a void pointer.
So doing a static_cast to an uint16_t pointer does make quite some assumptions, especially since the pointer is pointing to some stack-allocated variable in the function where readHoldingRegister is called.
So if the data struct is having a longer life span than just this call, then you will for sure get a crash.

[flashmark]

Indeed the whole construction is a bit tricky. It was the first attempt to abstract a way to pass back values from the modbus_device to the plugin. I wanted to decouple the pointer storage in a Modbus_link defined queue element from the user of the pointer. It is the modbus_device object that defines what is pointed at. It is an uit16_t pointer provided by the plugin.

I also would like to discuss the anyhow. Since we have the callback through an event it is only used for read requests where the caller expects immediate result. I implemented a PLUGIN_GET_CONFIG_VALUE that expects the value from the device to be returned immediately. It calls readHoldingRegister with this uint16_t pointer. A very ugly construction. But I don't know what other solution we can offer. I can also remove this GET_CONFIG_VALUE.

If we see value in such a direct retrieval at the cost of stalling the CPU I can try to add some comments in the code to point to the potential issues.

[TD-er]

It is also possible to think of a local mirror of the registers of the device in the instance.
This way the PLUGIN_GET_CONFIG_VALUE can pick the last known value from this mirror which is way faster.

Quite a lot of Modbus devices also support reading a range of registers at once and some even support mirroring registers to get the most frequently read ones next to eachother so they can be read in a single call.
ESPEasy typically addresses only a single 'taskvalue' at once, like when parsing a [taskname#taskvaluename] format in rules or to be shown somewhere. If this would trigger a read from a device, then things will run much slower.
So I'm thinking about letting the Modbus thingy run in the background gathering the register data and then when accessing those, you simply take them from the 'mirror' in the ModbusDevice object.

This ModbusDevice object then must know what value is stored where in the mirror and has the responsibility to keep this updated. So it makes sense to have some idea of a refresh interval and that's what we also have in each Task, where you set an Interval.
However for the Eastron devices we continuously poll those registers and also keep track of min/max values for the 'stats' as we get updates way more frequent than the 'Interval' of the task.

[flashmark]

For known devices you are right, we can cache all known, interesting registers and select the ones requested by the task value or GET_CONFIG_VALUE. The generic modbus device has no knowledge what registers are of interest. Either the cache/mirror will very big or a preselection is required.

For known devices this would not be a problem. The mirror shall be in the plugin/plugin_data_struct as the Modbus_device is intended to be generic. Unless we add a generic cache/mirror which can be parameterized. I have to think about how we can fit this in best.

For now I need to get the code robust. I started with everything being triggered and configured from the plugins. With the new interfaces page this shifted to a decoupled link life cycle with a predefined set of links. Each link can now exist, but be decoupled from a serial port. And even be (de)coupled separately from the plugin life cycle. Some puzzles to solve... Once done I would like to baseline the plugin and framework. That should be the starting point to look into the requirements to get other plugins based on the framework.

[TD-er]

Can you also post the graphical rendering of the UML diagram?

[flashmark]

Can you also post the graphical rendering of the UML diagram?

Sure, Is it possible to generate some documentation with UML diagrams? I don't know much about document generator used by ESPEasy. And it should not be part of the user manual... With the callbacks the code is quite complex and if other developers want to build plugins on the Modbus facility they should be aware of the caveats.

[TD-er]

Can you also post the graphical rendering of the UML diagram?

Sure, Is it possible to generate some documentation with UML diagrams? I don't know much about document generator used by ESPEasy. And it should not be part of the user manual... With the callbacks the code is quite complex and if other developers want to build plugins on the Modbus facility they should be aware of the caveats.

Well it is at least much easier to 'read' when it is rendered into some diagram.
And since you were using quite a specific notation syntax, I thought maybe you also would use some tool yourself to convert it into an image. Maybe something like dot (or using dot).

[flashmark]

Can you also post the graphical rendering of the UML diagram?

Sure, Is it possible to generate some documentation with UML diagrams? I don't know much about document generator used by ESPEasy. And it should not be part of the user manual... With the callbacks the code is quite complex and if other developers want to build plugins on the Modbus facility they should be aware of the caveats.

Well it is at least much easier to 'read' when it is rendered into some diagram. And since you were using quite a specific notation syntax, I thought maybe you also would use some tool yourself to convert it into an image. Maybe something like dot (or using dot).

I am using PlantUML to specify UML diagrams. There is a plugin for VisualCode. I will upload SVG drawings generated by PlantUML. I will try to create some additional documentation in text.

[flashmark]

Question: how can I show the connected Modbus link on the device overview page? For I2C the port column shows "I2C

"

[tonhuisman]

how can I show the connected Modbus link on the device overview page?

There is a function PLUGIN_WEBFORM_SHOW_CONFIG that can be handled in your plugin, see P152 for a usable example, just insert <BR/> where you need a newline if you have multiple lines to show. Many other plugins use this function to show the serial configuration.
The I2C and SPI configuration is handled by the DevicesPage code, as they have a matching Device.Type attribute, but the Serial types aren't specific enough to include all info for specific setups, that's why most of the Serial type plugins have a custom implementation (though most are very similar).
For Modbus we don't (yet) have a Device type defined, we can think about introducing that though.