alarm.sleep_memory not preserved across microcontroller.reset()

[lzrd]

alarm.sleep_memory not preserved across microcontroller.reset()

Summary

On ESP32-S2 (and likely all Espressif targets), alarm.sleep_memory is zeroed by
microcontroller.reset() despite the hardware being capable of preserving RTC memory
across a software reset. This is because the backing storage uses RTC_DATA_ATTR
instead of RTC_NOINIT_ATTR, causing the ESP-IDF bootloader to zero-initialize it
on every boot.

There is currently no way for CircuitPython code to persist data across
microcontroller.reset() without writing to flash. The hardware supports it — the
fix is a one-line attribute change in the Espressif port.

Bug or Feature Request?

Feature request / documentation clarification. The current behavior is technically
correct per the docstring ("persists during deep sleep"), but the underlying hardware
can do more, and the current behavior is surprising and limits useful patterns.

Steps to Reproduce

import alarm
import microcontroller

# Write a value to sleep memory
alarm.sleep_memory[0] = 42
alarm.sleep_memory[1] = 0xAB
print("Before reset:", alarm.sleep_memory[0], alarm.sleep_memory[1])

# Reset the microcontroller
microcontroller.reset()

After reset, in a separate boot.py or code.py:

import alarm

# Expected: 42 and 0xAB (preserved across software reset)
# Actual: 0 and 0 (zeroed by bootloader)
print("After reset:", alarm.sleep_memory[0], alarm.sleep_memory[1])

Test Program

Save as code.py. Run on any Espressif board. The program uses a CRC32 integrity
check to distinguish valid data from uninitialized memory, then reboots twice via
microcontroller.reset() to verify persistence.

Power must be maintained throughout — do not remove USB or battery, and do not
press the reset button (which triggers a more thorough reset that clears all
memory regardless of this fix; see the behavior table below).

"""Test: does alarm.sleep_memory survive microcontroller.reset()?

Boot counter with CRC32 integrity check. Reboots twice, then reports.
If sleep_memory survives, the counter increments across resets.
If not, each boot sees invalid/zeroed data and reports FAIL.
"""
import alarm
import binascii
import microcontroller
import struct
import time

# Layout: [magic:2][count:1][padding:1][crc32:4] = 8 bytes at offset 0
OFFSET = 0
MAGIC = 0xBE01  # non-zero, unlikely in uninitialized SRAM
REBOOT_LIMIT = 3
FMT = "<HBx"  # little-endian: uint16 magic, uint8 count, 1 byte padding
DATA_SIZE = struct.calcsize(FMT)  # 4 bytes
TOTAL_SIZE = DATA_SIZE + 4  # + 4 bytes CRC32

def read_counter():
    """Read and validate the boot counter. Returns count or None."""
    raw = bytes(alarm.sleep_memory[OFFSET:OFFSET + TOTAL_SIZE])
    data = raw[:DATA_SIZE]
    stored_crc = struct.unpack_from("<I", raw, DATA_SIZE)[0]
    actual_crc = binascii.crc32(data) & 0xFFFFFFFF
    if stored_crc != actual_crc:
        return None  # integrity check failed
    magic, count = struct.unpack(FMT, data)
    if magic != MAGIC:
        return None  # not our data
    return count

def write_counter(count):
    """Write the boot counter with CRC32."""
    data = struct.pack(FMT, MAGIC, count)
    crc = struct.pack("<I", binascii.crc32(data) & 0xFFFFFFFF)
    for i, b in enumerate(data + crc):
        alarm.sleep_memory[OFFSET + i] = b

reason = microcontroller.cpu.reset_reason
count = read_counter()

print(f"Reset reason: {reason}")
print(f"Boot counter: {count} (None = invalid/uninitialized)")

if reason == microcontroller.ResetReason.POWER_ON:
    # Fresh power-on — initialize counter to 1
    print("Power-on reset. Initializing counter and rebooting...")
    write_counter(1)
    time.sleep(1)
    microcontroller.reset()
elif count is not None and count < REBOOT_LIMIT:
    # Counter valid — increment and reboot again
    print(f"Counter valid ({count}/{REBOOT_LIMIT}). Incrementing and rebooting...")
    write_counter(count + 1)
    time.sleep(1)
    microcontroller.reset()
elif count is not None and count >= REBOOT_LIMIT:
    # Reached limit — test passed
    print(f"PASS: sleep_memory preserved across {count} microcontroller.reset() calls")
    write_counter(0)  # clean up (magic still valid, count=0)
else:
    # Counter invalid after a non-power-on reset — sleep_memory was cleared
    print(f"FAIL: sleep_memory not preserved across microcontroller.reset()")
    print(f"  Reset reason was {reason}, but counter integrity check failed.")
    print(f"  This means the bootloader zeroed RTC memory on software reset.")
    print(f"  Raw bytes: {bytes(alarm.sleep_memory[OFFSET:OFFSET + TOTAL_SIZE])}")

Expected output across 4 boots (with fix):

Reset reason: POWER_ON             # boot 1: initialize
Power-on reset. Initializing counter and rebooting...
Reset reason: SOFTWARE             # boot 2: counter = 1
Counter valid (1/3). Incrementing and rebooting...
Reset reason: SOFTWARE             # boot 3: counter = 2
Counter valid (2/3). Incrementing and rebooting...
Reset reason: SOFTWARE             # boot 4: counter = 3
PASS: sleep_memory preserved across 3 microcontroller.reset() calls

Actual output (without fix):

Reset reason: POWER_ON             # boot 1: initialize
Power-on reset. Initializing counter and rebooting...
Reset reason: SOFTWARE             # boot 2: counter gone
FAIL: sleep_memory not preserved across microcontroller.reset()
  Reset reason was SOFTWARE, but counter integrity check failed.
  This means the bootloader zeroed RTC memory on software reset.
  Raw bytes: b'\x00\x00\x00\x00\x00\x00\x00\x00'

Root Cause

In ports/espressif/common-hal/alarm/SleepMemory.c, line 18:

static RTC_DATA_ATTR uint8_t _sleep_mem[SLEEP_MEMORY_LENGTH];

RTC_DATA_ATTR places the array in the .rtc.data section (or .rtc.bss since it
has no explicit initializer). The ESP-IDF second-stage bootloader initializes these
sections to zero on every boot, including after esp_restart().

ESP-IDF provides RTC_NOINIT_ATTR specifically for data that should survive software
resets. Variables with this attribute are placed in the .rtc_noinit section, which
the bootloader skips during initialization.

From the ESP-IDF documentation:

By default, the RTC Fast memory region is added to the heap allocator ... To place
uninitialized data into RTC fast memory which is not initialized during startup,
use RTC_NOINIT_ATTR.

Hardware Context (ESP32-S2 TRM, Chapter 6)

microcontroller.reset() calls esp_restart(), which triggers RTC_CNTL_SW_SYS_RST
(reset source code 0x03 in Table 6.1-1). Despite the name "Software system reset",
this is classified as a Core Reset in the TRM, which does NOT include the RTC
domain (see Figure 6.1-1). Per TRM Section 6.1.1:

"All reset types mentioned above (except Chip Reset) maintain the data stored in
internal memory."

The hardware preserves RTC SRAM across Core Reset. The data loss is purely a software
artifact of the bootloader initializing RTC_DATA_ATTR variables.

Behavior of other reset and reload types

For completeness, here is how sleep memory behaves across different restart mechanisms:

Mechanism	What happens	Sleep memory today	Sleep memory with fix
alarm.exit_and_deep_sleep_until_alarms()	Deep sleep wake	Preserved	Preserved
alarm.light_sleep_until_alarms()	Light sleep wake	Preserved	Preserved
supervisor.reload()	Python VM restart, no hardware reset	Preserved	Preserved
microcontroller.reset()	Core Reset via esp_restart()	Zeroed	Preserved
Watchdog / panic reset	Core Reset	Zeroed	Preserved
Reset button	Chip Reset (see note)	Zeroed	Zeroed (contents undefined)
Brown-out	System Reset (resets RTC domain)	Zeroed	Zeroed (cleared explicitly)
Removing and restoring power	Chip Reset	Zeroed	Zeroed (contents undefined)

Note on the reset button: the reset type depends on board wiring. On the Adafruit
MagTag (and most Espressif dev boards), the reset button pulls the EN (chip enable)
pin low, which triggers a Chip Reset — the most thorough reset type, clearing all
memory including RTC. This is distinct from microcontroller.reset() which triggers
a Core Reset via esp_restart().

Proposed Fix

Patch file: circuitpython-sleep-memory.patch

Two changes in ports/espressif/common-hal/alarm/SleepMemory.c:

Change 1: Use RTC_NOINIT_ATTR for storage

--- a/ports/espressif/common-hal/alarm/SleepMemory.c
+++ b/ports/espressif/common-hal/alarm/SleepMemory.c
@@ -15,7 +15,7 @@

 // Data storage for singleton instance of SleepMemory.
 // Might be RTC_SLOW_MEM or RTC_FAST_MEM, depending on setting of CONFIG_ESP32S2_RTCDATA_IN_FAST_MEM.
-static RTC_DATA_ATTR uint8_t _sleep_mem[SLEEP_MEMORY_LENGTH];
+static RTC_NOINIT_ATTR uint8_t _sleep_mem[SLEEP_MEMORY_LENGTH];

This places _sleep_mem in the .rtc_noinit section, which the bootloader does not
initialize. Sleep memory then persists across all reset types where the RTC domain
remains powered.

Change 2: Clear explicitly when contents are unreliable

+#include "esp_system.h"
+
 void alarm_sleep_memory_reset(void) {
-    // ESP-IDF build system takes care of doing esp_sleep_pd_config() or the equivalent with
-    // the correct settings, depending on which RTC mem we are using.
-    // https://docs.espressif.com/projects/esp-idf/en/latest/esp32s2/api-reference/system/sleep_modes.html#power-down-of-rtc-peripherals-and-memories
+    // With RTC_NOINIT_ATTR, the bootloader does not initialize sleep memory.
+    // Preserve contents on resets where the RTC domain stays powered (software
+    // reset, watchdog, panic, deep sleep wake). Clear on everything else —
+    // after power-on, SRAM contents are undefined; after brown-out, the RTC
+    // domain was reset by hardware (System Reset scope per TRM Figure 6.1-1).
+    esp_reset_reason_t reason = esp_reset_reason();
+    switch (reason) {
+        case ESP_RST_SW:        // microcontroller.reset() / esp_restart()
+        case ESP_RST_DEEPSLEEP: // deep sleep wake
+        case ESP_RST_PANIC:     // unhandled exception
+        case ESP_RST_INT_WDT:   // interrupt watchdog
+        case ESP_RST_TASK_WDT:  // task watchdog
+        case ESP_RST_WDT:       // other watchdog
+            // RTC domain was not reset — sleep memory is intact.
+            break;
+        default:
+            // Power-on, brown-out, unknown, or any other reason where
+            // RTC SRAM contents may be undefined. Clear to zero.
+            memset(_sleep_mem, 0, sizeof(_sleep_mem));
+            break;
+    }
 }

The whitelist approach defaults to clearing on unknown or unexpected reset reasons,
which is safer than a blacklist. If a new reset reason is added in a future ESP-IDF
version, sleep memory will be cleared (safe) rather than preserved (potentially
corrupt).

This preserves the existing guarantee ("if power is lost, the memory contents are
lost") while adding the new capability of surviving software resets.

Alternative: User-Controllable Clear via boot.py

If a fully opt-in approach is preferred, CircuitPython could expose a method to
explicitly clear sleep memory, allowing boot.py to decide:

# boot.py — clear sleep memory on power-on if desired
import microcontroller
import alarm

if microcontroller.cpu.reset_reason == microcontroller.ResetReason.POWER_ON:
    for i in range(len(alarm.sleep_memory)):
        alarm.sleep_memory[i] = 0

This approach works today as a workaround but doesn't fix the core issue — users
who want persistence across microcontroller.reset() currently have no way to
achieve it with alarm.sleep_memory.

Use Cases

1. OTA Firmware Updates (our use case)

A/B slot OTA system where the updater writes an ephemeral trial boot marker to
sleep memory and calls microcontroller.reset(). boot.py reads the marker,
validates the target slot, and boots it. The persistent slot preference lives in
a flash file (slot.txt); sleep memory only carries the transient trial request
that needs to survive a single reset cycle.

The marker uses distinct non-zero values (e.g., 0x11 = trial requested,
0x21 = trial acknowledged by boot.py) so that zeroed memory is unambiguously
"no trial active — use the persistent preference." Any unrecognized value,
including 0x00, falls through to slot.txt.

Current workaround: Write the trial marker to a flash file (/trial.txt)
instead of sleep memory. This works but adds flash wear and requires the
filesystem to be writable (which conflicts with USB mass storage mode).

2. Crash Counter / Watchdog Recovery

Track boot count across resets to detect boot loops. If the device has reset 3+ times
without reaching stable operation, enter safe mode or fall back to a known
configuration.

3. State Handoff Across Intentional Resets

Pass small amounts of state (configuration changes, pending operations, error codes)
across an intentional microcontroller.reset() without flash writes. Useful when
the reset is needed to apply hardware configuration changes that require a full
reboot (USB mode changes, clock reconfiguration, etc.).

4. Deep Sleep vs Reset Consistency

Currently, data written before alarm.exit_and_deep_sleep_until_alarms() is
preserved, but data written before microcontroller.reset() is lost. Both deep
sleep and software reset preserve the RTC domain at the hardware level — deep sleep
keeps RTC powered while the digital core is off, and Core Reset (from esp_restart())
does not include the RTC domain. The asymmetry in sleep memory behavior is entirely
a software artifact of the bootloader initializing RTC_DATA_ATTR variables. The
proposed fix aligns the software behavior with what the hardware already supports.

Impact Assessment

Positive

• Sleep memory becomes useful for a broader set of patterns
• Behavior aligns with hardware capability and user expectations
• No API changes — existing code continues to work
• Documentation becomes simpler: "persists across resets; lost on power loss"

Negative / Risks

• Stale data on first power-on: With RTC_NOINIT_ATTR, the initial contents of
  sleep memory are undefined after power is first applied (random SRAM values). The
  proposed alarm_sleep_memory_reset() change handles this by clearing on any reset
  reason that is not in the known-safe whitelist (software reset, watchdog, deep sleep
  wake). If the whitelist logic is wrong, users could see garbage data.

  Mitigation for application code: As a best practice, any application using
  alarm.sleep_memory across resets should protect its data with an integrity check.
  Define a structure with a known magic number and a hash or checksum over the
  contents. Validate before trusting, update the hash on every write. The ESP32-S2
  provides hardware-accelerated SHA (SHA-1, SHA-224, SHA-256, SHA-384, SHA-512) and
  HMAC via its cryptography accelerators (TRM Chapters 16 and 19), or applications
  can use a simpler CRC32 via binascii.crc32(). This protects against both
  uninitialized memory and corruption from edge-case resets.

• Behavioral change: Any code that relies on sleep memory being zeroed after
  microcontroller.reset() would break. However, no reasonable code should depend on
  this behavior since it's undocumented and the documented purpose is deep sleep
  persistence.

• Brown-out reset: Brown-out triggers a System Reset (code 0x0F in TRM Table
  6.1-1), which resets the entire digital system including the RTC domain. RTC SRAM
  contents are not guaranteed after a System Reset. The proposed fix correctly clears
  sleep memory on ESP_RST_BROWNOUT via the default branch of the switch statement.

• Scope: This fix is specific to the Espressif port
  (ports/espressif/common-hal/alarm/SleepMemory.c). Other ports (nrf52, RP2040,
  STM32) have their own SleepMemory implementations with different hardware
  constraints and are not affected.

• Other Espressif targets: The RTC_NOINIT_ATTR attribute is supported across
  all ESP-IDF targets. Testing should verify behavior on ESP32 (original), ESP32-S3,
  ESP32-C3, and ESP32-C6.

Docstring Update

If accepted, the SleepMemory class docstring should be updated:

-//|     """Store raw bytes in RAM that persists during deep sleep.
-//|     The class acts as a ``bytearray``.
-//|     If power is lost, the memory contents are lost.
+//|     """Store raw bytes in RAM that persists across deep sleep and software resets.
+//|     The class acts as a ``bytearray``.
+//|     Contents are preserved across ``microcontroller.reset()``, watchdog resets,
+//|     and deep sleep wake. Contents are lost when power is removed and restored,
+//|     or on brown-out reset.

Environment

• Board: Adafruit MagTag (ESP32-S2)
• CircuitPython version: 10.1.4
• ESP-IDF version: (bundled with CircuitPython build)

References

• ESP32-S2 Technical Reference Manual v1.3, Chapter 6 (Reset and Clock)
• ESP-IDF docs: RTC memory
• CircuitPython source: ports/espressif/common-hal/alarm/SleepMemory.c
• ESP-IDF RTC_NOINIT_ATTR: placed in .rtc_noinit linker section, skipped by bootloader init

[tannewt]

This is way more detailed than it needs to be. Please don't just copy and paste verbose LLM output. Instead, please open a PR with the proposed fix after you've reviewed all code written and tested it on a device to make sure it works. If you want an LLM pre-review, please do that in your fork before making a PR here. Thanks!