Telnet and Web Server Crash

[SoulGit]

Running on an Atom 485 Stack.

After about 15 to 30 minutes, both the web server and telnet become unresponsive. The web site sends back no data, telnet connects and then disconnects: "Connection closed by foreign host." Not sure if this is related to my prior issue and running out of resources due to the continuous flood of errors.

[dacarson]

It might be crashing and restarting. It shouldn't be running out of resources as it is not storing the unknown packets.
If you are able to run it for a while, and send the last couple of lines you see before it disconnects, that would be helpful.
Also the last log messages you see from the web server.

[SoulGit]

I'll do that. I found this from my console history if it is helpful. The web server didn't show any last logs, just goes unavailable. The system also never comes back, I have to pull power for a hard restart.

Error parse_status_packet Unknown status packet type received.
Error print_buffer F7 13 50 52 10 03 40 00 04
Error parse_status_packet Unknown status packet type received.
Error print_buffer F7 02 50 51 10 22 41 00 00 01 00 67 62 54 00 00 00 00 00 00 01 00 00 20 02 04 45 21 6D 0A 0E 00
Error print_buffer 06 02 00 00 00 C2 00 00
Error parse_status_packet Unknown status packet type received.
Error print_buffer F7 13 50 52 10 03 40 00 04
Error parse_status_packet Unknown status packet type received.
Error print_buffer F7 02 50 51 10 22 41 00 00 01 00 67 62 54 00 00 00 00 00 00 01 00 00 20 02 04 45 21 6D 0A 0E 00
Error print_buffer 06 02 00 00 00 C2 00 00
Error parse_status_packet Unknown status packet type received.
Error print_buffer F7 13 50 52 10 03 40 00 04
Error parse_status_packet Unknown status packet type received.
Error print_buffer F7 02 50 51 10 22 41 00 00 01 00 67 62 54 00 00 00 00 00 00 01 00 00 20 02 04 45 21 6D 0A 0E 00
Error print_buffer 06 02 00 00 00 C2 00 00

Connection closed by foreign host.

[dacarson]

Are you able to attach a computer and see what the serial output is?

[SoulGit]

Haven't been able to attach a computer yet, but while capturing packets for the Unknown packet status issue, the web server stopped responding. The open telnet connection remained functional and for the time being I am able to disconnect and reconnect to telnet. I just realized that the homespan service also seems to have died.

I can open another issue for this, but will drop it here just in case it is related. In both homekit (and home assistant when importing the homekit device), I see the unit switching between heat and off multiple times a second. Almost seems as if it is flickering when viewed within homekit. Perhaps the frequent changes is causing Homespan to fall over?

[dacarson]

I checked to see if there are major differences between the "Atom 485 Stack" that you are running and the "WeMos D1 ESP32 Mini" that I am using. My WeMos may have more RAM.
To that end, I added a new command to Telnet, memory which will print out the available memory.
This is what I see:

> memory
Memory Details
  Free heap: 127568 bytes
  Max alloc block: 77812 bytes
>

The switching between heat and off is a different issue - I know what is up, and I am working on a fix for that.

[SoulGit]

Memory Details
Free heap: 129140 bytes
Max alloc block: 81908 bytes

[SoulGit]

I attempted to add the device to HomeAssistant via HomeKit integration and the website crashed right about the same time. Looks like there is enough memory:

Memory Details
Free heap: 105744 bytes
Max alloc block: 65524 bytes

[dacarson]

Agreed, it seems that there is enough memory. There must be something else.

[SoulGit]

Connected over serial and saw repeating messages of Unknown Water Packet. There were also a few messages about writing history to memory. After about 25 minutes, the web page crashed. Unfortunately, I didn't get to see if there was a console message as the buffer was lost when I disconnected the serial connection from the Arduino IDE (didn't expect that!). Let me know if you would like me to try again or if you think the repeating messages might be related.

[SoulGit]

I put a proper serial monitor in place and basically saw the same thing. Repeating messages of "unknown water packet" with a few "Updating History Status store.usedMemory 2056 store.lastEntry 2056" and "Navien current Temperature is 92.00 F." mixed in.

This however may be interesting to you: I was monitoring for about 30 minutes with nothing but the repeating messages. The water heater turned on due to demand and right afterwards the stack died. Console just stopped outputting messages. Only remedy was a reboot.

[dacarson]

Very very interesting.
So there is a thing built into ESP32 code, a WDT (Watch Dog Timer) which should reboot the device if it hangs. That should be kicking in on your device rather than requiring a hard reboot. Digging deeper, it seems that there are cases where the device is hung and the WDT doesn't fire. I reviewed the packet parsing code and there are instances where that could hang.

[dacarson]

Try it again with the updated Navien.cpp file and it will hopefully print out what the issue might be to the console without hanging.

[SoulGit]

This version won't compile . . .

Navien.cpp:102:42: error: cannot convert 'Navien::NAVIEN_STATE_GAS*' to 'Navien::NAVIEN_STATE*' in argument passing
102 | if (on_gas_packet_cb) on_gas_packet_cb(&(state.gas));
| ^~~~~~~~~~~~
| |
| Navien::NAVIEN_STATE_GAS*
exit status 1

[dacarson]

Sorry. Looks like I've made too many changes since you last pulled the code. You will need to update everything.