Firmware, GPAD_API, Rev 0.58, US0008 and US0014 Lost Broker: krakepubinv.cloud.shiftr.io

[ForrestErickson]

Describe the bug
A clear and concise description of what the bug is.
Short Summary of the expected behavior:
On 20260529 at 12:00 Lee published to all Rev2 Krakes the in discord message.

Short Summary of the buggy behavior:
Neither US0008 nor US0014 responded. Both show last message at 20260527.

US0014 is connected to VSCode terminal and shows power on reset.

--- Terminal on COM6 | 115200 8-N-1
--- Available filters and text transformations: debug, default, direct, esp32_exception_decoder, hexlify, log2file, nocontrol, printable, send_on_enter, time
--- More details at https://bit.ly/pio-monitor-filters
--- Quit: Ctrl+C | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H
ets Jul 29 2019 12:21:46

rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:1184
load:0x40078000,len:13232
load:0x40080400,len:3028
entry 0x400805e4
==== ESP23WiFiScaner, Ver: 0.0.14 ====
Built: May 25 2026 13:47:18
========================================

LittleFS mounted successfully.
Configuration profiles loaded from flash storage.
>>> Scanning environment for stored profiles... <<<
[ROAM] Candidate list built: 1 profiles
[ROAM] Attempt 1/1 -> SSID: VRX

>>> SUCCESS: Connected as a Station! <<<
SSID Connected: VRX
IP Address allocated: 192.168.1.247
========================================

Asynchronous Web Server with ElegantOTA active on port 80.

Neither devices show up in the PMD_Processing_MQTT console.

Request 'h' and 'i' of US0014

Typed into terminal

LCD Status

The LCD shows network and broker connection as normal.

Network Connection

Ping 192.168.1.230 and no response. Pinged another address and then .230 again after the 'h' and 'i' commands above and got response.

Note the broker is show as Broker: public.cloud.shiftr.io

[ForrestErickson]

Checking Broker URL

Accessing the embedded web pages
US0008

US0014

Conclusion regarding Broker

Both devices are no longer connecting to the broker
krakepubinv:DlDmkWjp4I4kgDcA@krakepubinv.cloud.shiftr.io

They were but are no longer connecting to the krakepubinv broker.

[ForrestErickson]

Other Krakes LB0098 and UK0001 are on line at other broker than LB0009

20260529_123523 Msg_recd: KRAKE_LB0009 - === KRAKE SYSTEM INFO ===
20260529_123523 Msg_recd: KRAKE_LB0009 - SN: KRAKE-C295D0
20260529_123523 Msg_recd: KRAKE_LB0009 - FW: 0.58.1 
20260529_123523 Msg_recd: KRAKE_LB0009 - IP: 192.168.10.234
20260529_123523 Msg_recd: KRAKE_LB0009 - MAC: F4650BC295D0
20260529_123523 Msg_recd: KRAKE_LB0009 - SSID: Cudy-2A60
20260529_123523 Msg_recd: KRAKE_LB0009 - Broker: krakepubinv.cloud.shiftr.io
20260529_123523 Msg_recd: KRAKE_LB0009 - MQTT: Connected
20260529_123523 Msg_recd: KRAKE_LB0009 -  online, RSSI:-60.00
20260529_123523 Msg_recd: KRAKE_UK0001 -  online, RSSI:-61.00
20260529_123523 Msg_recd: KRAKE_UK0001 - === KRAKE SYSTEM INFO ===
20260529_123523 Msg_recd: KRAKE_UK0001 - SN: KRAKE-C295EC
20260529_123523 Msg_recd: KRAKE_UK0001 - FW: 0.58 
20260529_123523 Msg_recd: KRAKE_UK0001 - IP: 192.168.0.50
20260529_123523 Msg_recd: KRAKE_UK0001 - MAC: F4650BC295EC
20260529_123523 Msg_recd: KRAKE_UK0001 - SSID: SKYNM46G
20260529_123523 Msg_recd: KRAKE_UK0001 - Broker: krakepubinv.cloud.shiftr.io
20260529_123523 Msg_recd: KRAKE_UK0001 - MQTT: Connected

[ForrestErickson]

Further investigation with Nagham's (@nk25719 ) help showed that the US0014 user settings showed that it was on the public.public shifter Broker.

Tentative Conclusion, Broker Was Over Written.

[ForrestErickson]

Reset US0014 through Terminal

At 20260529 1418 sent 'r' command to US0014.

Software reset requested.
ets Jul 29 2019 12:21:46

rst:0xc (SW_CPU_RESET),boot:0x17 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:1184
load:0x40078000,len:13232
load:0x40080400,len:3028
entry 0x400805e4

PubInv GPAD: enter command in form CDa (C is a char, D is a digit)
COM UART1: 9600, 8-N-1, flow=off
[  2739][E][vfs_api.cpp:105] open(): /littlefs/wifi.json does not exist, no permits for creation
*wm:AutoConnect
*wm:Connecting to SAVED AP: VRX
*wm:connectTimeout not set, ESP waitForConnectResult...
Attempting MQTT connection at: 4841 broker=krakepubinv.cloud.shiftr.io user=krakepubinv ip=192.168.1.230 rssi=-50 clientId=PubInv -F4650BC0B528 sub=F4650BC0B528_ALM pub=F4650BC0B528_ACK ... *wm:AutoConnect: SUCCESS
*wm:STA IP Address: 192.168.1.230
success at: 5710
Command: i
=== KRAKE SYSTEM INFO ===
SN: KRAKE-C0B528
FW: 0.58
IP: 192.168.1.230
MAC: F4650BC0B528
SSID: VRX
Broker: krakepubinv.cloud.shiftr.io
MQTT: Connected
Heap: 198416
Uptime: 00:00:13
=========================
currentlyMuted : 0
interpret Done
Muted: NO
LVL: 0
No Message.
Silent level: skipping DFPlayer playback.

[ForrestErickson]

Lee asked CODEX for analysis:

For a device running the GPAD_API code version 0.58 the device started with broker: krakepubinv.cloud.shiftr.io but at 60 hours was found to have changed to the other broker:public.cloud.shiftr.io

Why?

Codex came back with an explanation

Short answer
The broker did not change because of a 60-hour timer. In GPAD_API 0.58, the firmware has an automatic MQTT broker
 failover path: if the currently active MQTT broker fails to reconnect 5 times, the code advances 
activeBrokerIndex to the next broker in brokerOptions. Since the default/project broker is krakepubinv.cloud.shiftr.io 
and the next broker is public.cloud.shiftr.io, a temporary MQTT outage or repeated failed reconnects would switch 
the device from Krake PubInv to Public Shiftr. 

Why that specific switch happened
The broker list is ordered like this:
public.cloud.shiftr.io
krakepubinv.cloud.shiftr.io

The default selected broker index is 1, which means the device starts on krakepubinv.cloud.shiftr.io. 
When reconnecting, the code increments mqttFailCount every time client.connect(...) fails. 
Once mqttFailCount >= MQTT_FAILOVER_THRESHOLD, where the threshold is 5, it resets the fail count and advances:

Link to Codex Analysis

https://chatgpt.com/s/cd_6a19de030e888191b7d642de7e6b7ffc

[ForrestErickson]

Notes of 20260530.

As of 0400 ET
Only UK0001 is on the broker: krakepubinv.cloud.shiftr.io

Trying To ReConnect Broker krakepubinv.cloud.shiftr.io

US0008 has presumably chanted over to public.cloud.shiftr.io and US0014 has changed over to the public.cloud.shiftr.io broker.

Lets try resetting.

Power cycle of US008 did not bring it back to: krakepubinv.cloud.shiftr.io
Reset of US0014 did not bring ti back to: krakepubinv.cloud.shiftr.io

Navigation through the UI

On US0014, Lee rotated the encoder till the 'B' was blinking and pressed the encoder. The menu came up offering option for " 1 Krake PubInv" as where the cursor was blinking = selected, and another. See image below. Lee pressed the encoder knob to select broker 1. The Krake went back into a mode of displaying System OK. See second image, System OK image below further .
The Krake US0014 was now on the broker. See PMD_Processing_MQTT console below.

Broker Menu

System OK Image

PMD_Processing_MQTT console after reconnection of US0014

Krake US0008 back on line after selecting broker.

[ForrestErickson]

Broker Misrepresented on LCD

Additional Bug, The display of brokers in the Broker menu is misleading as to the exact broker.

Broker Menu

In the code the brokers