TCP error handling

[andig]

@gq0 in goburrow/modbus#25 (comment) you were so kind to help with tcp recovery parameters. I've switched to grid-x and am still having some problems getting a stable connection to SE9k. I'm using these parameters:

handler.Timeout = 300 * time.Millisecond
handler.ProtocolRecoveryTimeout = 10 * time.Second
handler.LinkRecoveryTimeout = 60 * time.Second

Still, recovery does only work sometimes:

17:23:33 modbus: send 18 fc 00 00 00 06 01 03 9c 95 00 02
17:23:33 modbus: send 18 fc 00 00 00 06 01 03 9c 95 00 02
17:23:38 modbus: send 18 fc 00 00 00 06 01 03 9c 95 00 02
17:23:48 Device 1 failed to retrieve opcode 0x9c95, error was: read tcp 192.168.178.36:34614->192.168.178.5:502: i/o timeout
17:23:48 Device 1 failed to respond - retry attempt 1 of 5
17:23:48 modbus: send 18 fd 00 00 00 06 01 03 9c 95 00 02
17:23:48 modbus: send 18 fd 00 00 00 06 01 03 9c 95 00 02
17:23:48 modbus: send 18 fd 00 00 00 06 01 03 9c 95 00 02
17:23:48 modbus: send 18 fd 00 00 00 06 01 03 9c 95 00 02
17:23:48 modbus: send 18 fd 00 00 00 06 01 03 9c 95 00 02
17:23:48 modbus: recv 18 fd 00 00 00 07 01 03 04 13 87 ff fe
17:23:48 Device 1 - Frequency: 49.99

Whereas often it does not work:

broken pipe:

03:46:43 modbus: send 12 f3 00 00 00 06 01 03 9c 8f 00 04
03:46:43 Device 1 failed to retrieve opcode 0x9c8f, error was: read tcp 192.168.178.36:60398->192.168.178.5:502: read: connection reset by peer
03:46:43 Device 1 failed to respond - retry attempt 1 of 5
03:46:43 modbus: send 12 f4 00 00 00 06 01 03 9c 8f 00 04
03:46:43 Device 1 failed to retrieve opcode 0x9c8f, error was: write tcp 192.168.178.36:60398->192.168.178.5:502: write: broken pipe
03:46:43 Device 1 failed to respond - retry attempt 2 of 5

i/o timeout:

17:24:24 modbus: send 19 0e 00 00 00 06 01 03 9c 9d 00 03
17:24:24 modbus: send 19 0e 00 00 00 06 01 03 9c 9d 00 03
17:24:24 modbus: send 19 0e 00 00 00 06 01 03 9c 9d 00 03
17:24:34 Device 1 failed to retrieve opcode 0x9c9d, error was: read tcp 192.168.178.36:34614->192.168.178.5:502: i/o timeout
17:24:34 Device 1 failed to respond - retry attempt 1 of 5
17:24:34 modbus: send 19 0f 00 00 00 06 01 03 9c 9d 00 03
17:24:44 Device 1 failed to retrieve opcode 0x9c9d, error was: read tcp 192.168.178.36:34614->192.168.178.5:502: i/o timeout
17:24:44 Device 1 failed to respond - retry attempt 2 of 5
17:24:44 modbus: send 19 10 00 00 00 06 01 03 9c 9d 00 03

I would appreciate any hints how to improve error handling- is there anything that needs to be done on client side? Are the parameter values ok?

[gq0]

Mmmm... ticky. Here is a list of things you may try:

• increase handler.Timeout to 1 second
• do not request more than 120 words (240 bytes) in one request
• time between request should be >= 64 ms
• filter on the timeout error err, ok := err.(*net.OpError); ok && err.Timeout() && err.Op == "read" and trigger a new ReadHoldingRegisters with a new transaction ID but consider the time between requests in the bullet point above
• based on the error (e.g. on a reset) you could completely close the connection and reconnect with a backoff strategy, e.g. cubic up to 10 minutes (a not proper closed TCP connection, missing RST or FIN, is kept by the OS' TCP stack and could lead to reconnecting issues, I am not sure what SolarEdge uses but in most cases a stalling TCP connection is dropped by the TCP stack after 2 minutes)

[gq0]

The tips above are with regards to using the lib. SolarEdge is really weird, especially their Modbus implementation and TCP stack. I am not sure how much of this weirdness should be handled by the lib 😄

[andig]

Let me start by saying thank you for the quick and in-depth response!

Looking at each of these none seem to apply afaics. I was wrong about handler.Timeout- its actually the default 10s, I had checked the RTU client. Read timeout Delay between reads (only in case of error though) is 100ms and no more than ~5 registers are read at once.

From looking through the code I assume that the connection is actually dead and closed. In that case- since the lib-internal connect() will only act on nil which then conn isn't- it will never reestablish a connection. Would that make sense?

If so- assuming a simple dropped connection (flaky dialup)- I could Close() and things might start working again but I'm really unsure what errors to test against. Up to debate if reestablishing a broken conn should be user or lib responsibility?

Update I feel it would be lib responsibility since its also the lib that establishes the connection in the first place?

Update the Close() approach is working in intial testing.

@gq0 what's your thought about having lib handle this at least in certain scenarios? It does open and own the connection anyway?

[gq0]

Hi @andig, this week I am a little bit busy and haven't had time to put my mind to it. But I will get back to you this week.

[andig]

Hi @gq0 did you have a chance to take a look? Closing the connection still works perfectly.

[andig]

ping @gq0 I'd still be interested in this

[andig]

ping @gq0 hope you're doing well and might find some time for this?

[gq0]

Hi @andig, I just noticed this ticket and totally forgot, sorry. I will check out your proposal this week.

[andig]

hi @gq0 just checking how your week went ;)

[andig]

@gq0 Almost a year going- looking forward to a Christmas present ;)

[gq0]

Ah, I am deeply sorry for my really delayed response.

I see your point in integrating this reconnect handing in the library. For us, it was not relevant to have in there because all the poll and reconnect handling is done Modbus-independent in our software stack. Hence, we don't have it on our roadmap to integrate it here. But this does not mean that it shouldn't be added.

You can add this reconnect handling also in your code-base if at some point your Modbus operations stop with an error, most likely an io operation or dialling timeout error, and then close and reinitiate the connection.

[andig]

Just now, one and a half years later I've run into the same issue again on a different project and totally forgot about the solution :O Would you accept a PR that simply closes the connection on error?

[e-cite]

This also affects modbus polling with tool "Influxdata/Telegraf". The connection is fairly regularly reset / broken every few minutes.

Any update on fixing this? Thanks!