m1n1 crashed when sending P_HV_START_SECONDARY

[dgh05t]

I'm trying to use m1n1 as hypervisor, and run a guest macOS with it, try to debug macos kernel.
Guest OS is macos 15.5 (24F74), host running python3 proxyclient/tools/shell.py works fine.

but when I try to boot the kernel: python3 proxyclient/tools/run_guest.py ~/Desktop/to_host_24F74/dev.kc.macho.development -- "debug=0x14e serial=3 apcie=0xfffffffe -enable-kprintf-spam wdt=-1 clpc=0", system crashed.

crash log:

TTY> HV: Initializing secondary 1
TTY> Exception: SYNC
TTY> Exception taken from EL2h
TTY> Running in EL2
TTY> MPIDR: 0x80000001
TTY> Registers: (@0x806463dd0)
TTY>   x0-x3: 0000000000000001 0000000000000000 0000000000000000 0000000000000000
TTY>   x4-x7: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
TTY>  x8-x11: 0000000030901085 0000000000000001 a005a500f005f00f f005a500f00ff00f
TTY> x12-x15: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
TTY> x16-x19: 0000000000000000 0000000000000000 00000000fffffff5 0000000804215220
TTY> x20-x23: 0000000804205650 0000000804215000 0000000000000001 0000000000000000
TTY> x24-x27: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
TTY> x28-x30: 0000000000000000 0000000804215220 0000000804215220
TTY> PC:       0x804215220 (rel: 0xc1220)
TTY> SP:       0x806463ed0
TTY> SPSR:     0x10c9
TTY> FAR:      0x0
TTY> ESR:      0x2000000 (unknown)
TTY> L2C_ERR_STS: 0x11000ffc00000000
TTY> L2C_ERR_ADR: 0x0
TTY> L2C_ERR_INF: 0x0
TTY> E_LSU_ERR_STS: 0x0
TTY> E_FED_ERR_STS: 0x0
TTY> E_MMU_ERR_STS: 0x0
TTY> Unhandled exception, rebooting...
[cpu0] Python exception while handling guest exception:
Traceback (most recent call last):
  File "/opt/homebrew/lib/python3.13/site-packages/serial/serialposix.py", line 575, in read
    buf = os.read(self.fd, size - len(read))
OSError: [Errno 6] Device not configured

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/mac-test/sourcecodes/m1n1/proxyclient/m1n1/hv/__init__.py", line 971, in handle_exception
    handled = self.handle_vm_hook(ctx)
  File "/Users/mac-test/sourcecodes/m1n1/proxyclient/m1n1/hv/__init__.py", line 591, in handle_vm_hook
    wfunc(base, data.addr - base, d, 8 << data.flags.WIDTH, **kwargs)
    ~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mac-test/sourcecodes/m1n1/proxyclient/m1n1/hv/__init__.py", line 1554, in cpustart_wh
    self.start_secondary(die, cluster, i)
    ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/Users/mac-test/sourcecodes/m1n1/proxyclient/m1n1/hv/__init__.py", line 1595, in start_secondary
    self.p.hv_start_secondary(index, entry)
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
  File "/Users/mac-test/sourcecodes/m1n1/proxyclient/m1n1/proxy.py", line 1072, in hv_start_secondary
    return self.request(self.P_HV_START_SECONDARY, cpu, entry, *args)
           ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mac-test/sourcecodes/m1n1/proxyclient/m1n1/proxy.py", line 694, in request
    return self._request(opcode, *args2, **kwargs)
           ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mac-test/sourcecodes/m1n1/proxyclient/m1n1/proxy.py", line 658, in _request
    reply = self.iface.proxyreq(req, reboot=reboot, no_reply=no_reply, pre_reply=None)
  File "/Users/mac-test/sourcecodes/m1n1/proxyclient/m1n1/proxy.py", line 378, in proxyreq
    return self.reply(self.REQ_PROXY)
           ~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/Users/mac-test/sourcecodes/m1n1/proxyclient/m1n1/proxy.py", line 249, in reply
    reply += self.readfull(1)
             ~~~~~~~~~~~~~^^^
  File "/Users/mac-test/sourcecodes/m1n1/proxyclient/m1n1/proxy.py", line 186, in readfull
    block = self.dev.read(size - len(d))
  File "/opt/homebrew/lib/python3.13/site-packages/serial/serialposix.py", line 581, in read
    raise SerialException('read failed: {}'.format(e))
serial.serialutil.SerialException: read failed: [Errno 6] Device not configured
[cpu0] Guest exception: HV/HOOK_VM
Traceback (most recent call last):
  File "/opt/homebrew/lib/python3.13/site-packages/serial/serialposix.py", line 621, in write
    n = os.write(self.fd, d)
OSError: [Errno 6] Device not configured

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/mac-test/sourcecodes/m1n1/proxyclient/tools/run_guest.py", line 115, in <module>
    hv.start()
    ~~~~~~~~^^
  File "/Users/mac-test/sourcecodes/m1n1/proxyclient/m1n1/hv/__init__.py", line 1936, in start
    self.p.hv_start(self.entry, self.guest_base + self.bootargs_off)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mac-test/sourcecodes/m1n1/proxyclient/m1n1/proxy.py", line 1058, in hv_start
    return self.request(self.P_HV_START, entry, *args)
           ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mac-test/sourcecodes/m1n1/proxyclient/m1n1/proxy.py", line 694, in request
    return self._request(opcode, *args2, **kwargs)
           ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mac-test/sourcecodes/m1n1/proxyclient/m1n1/proxy.py", line 658, in _request
    reply = self.iface.proxyreq(req, reboot=reboot, no_reply=no_reply, pre_reply=None)
  File "/Users/mac-test/sourcecodes/m1n1/proxyclient/m1n1/proxy.py", line 378, in proxyreq
    return self.reply(self.REQ_PROXY)
           ~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/Users/mac-test/sourcecodes/m1n1/proxyclient/m1n1/proxy.py", line 291, in reply
    self.handle_boot(data)
    ~~~~~~~~~~~~~~~~^^^^^^
  File "/Users/mac-test/sourcecodes/m1n1/proxyclient/m1n1/proxy.py", line 314, in handle_boot
    self.handlers[(reason, code)](reason, code, info)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File "/Users/mac-test/sourcecodes/m1n1/proxyclient/m1n1/hv/__init__.py", line 986, in handle_exception
    self.update_pac_mask()
    ~~~~~~~~~~~~~~~~~~~~^^
  File "/Users/mac-test/sourcecodes/m1n1/proxyclient/m1n1/hv/__init__.py", line 1837, in update_pac_mask
    tcr = TCR(self.u.mrs(TCR_EL12))
              ~~~~~~~~~~^^^^^^^^^^
  File "/Users/mac-test/sourcecodes/m1n1/proxyclient/m1n1/proxyutils.py", line 172, in mrs
    return self.exec(op, call=call, silent=silent)
           ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mac-test/sourcecodes/m1n1/proxyclient/m1n1/proxyutils.py", line 217, in exec
    self.iface.writemem(self.code_buffer, func)
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mac-test/sourcecodes/m1n1/proxyclient/m1n1/proxy.py", line 384, in writemem
    self.cmd(self.REQ_MEMWRITE, req)
    ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mac-test/sourcecodes/m1n1/proxyclient/m1n1/proxy.py", line 201, in cmd
    self.dev.write(command)
    ~~~~~~~~~~~~~~^^^^^^^^^
  File "/opt/homebrew/lib/python3.13/site-packages/serial/serialposix.py", line 655, in write
    raise SerialException('write failed: {}'.format(e))
serial.serialutil.SerialException: write failed: [Errno 6] Device not configured

[dgh05t]

on the linux host, the crash log is:

TTY> HV: Initializing secondary 1
TTY> Exception: SYNC
TTY> Exception taken from EL2h
TTY> Running in EL2
TTY> MPIDR: 0x80000001
TTY> Registers: (@0x8069abdd0)
TTY>   x0-x3: 0000000000000001 0000000000000000 0000000000000000 0000000000000000
TTY>   x4-x7: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
TTY>  x8-x11: 0000000030901085 0000000000000001 a005a500f005f00f f005a500f00ff00f
TTY> x12-x15: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
TTY> x16-x19: 0000000000000000 0000000000000000 00000000fffffff5 000000080475d220
TTY> x20-x23: 000000080474d650 000000080475d000 0000000000000001 0000000000000000
TTY> x24-x27: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
TTY> x28-x30: 0000000000000000 000000080475d220 000000080475d220
TTY> PC:       0x80475d220 (rel: 0xc1220)
TTY> SP:       0x8069abed0
TTY> SPSR:     0x10c9
TTY> FAR:      0x0
TTY> ESR:      0x2000000 (unknown)
TTY> L2C_ERR_STS: 0x11000ffc00000000
TTY> L2C_ERR_ADR: 0x0
TTY> L2C_ERR_INF: 0x0
TTY> E_LSU_ERR_STS: 0x0
TTY> E_FED_ERR_STS: 0x0
TTY> E_MMU_ERR_STS: 0x0
TTY> Unhandled exception, rebooting...
[cpu0] Python exception while handling guest exception:
Traceback (most recent call last):
  File "/home/linux-host/sourcecodes/m1n1/proxyclient/m1n1/hv/__init__.py", line 971, in handle_exception
    handled = self.handle_vm_hook(ctx)
  File "/home/linux-host/sourcecodes/m1n1/proxyclient/m1n1/hv/__init__.py", line 591, in handle_vm_hook
    wfunc(base, data.addr - base, d, 8 << data.flags.WIDTH, **kwargs)
    ~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/linux-host/sourcecodes/m1n1/proxyclient/m1n1/hv/__init__.py", line 1554, in cpustart_wh
    self.start_secondary(die, cluster, i)
    ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/linux-host/sourcecodes/m1n1/proxyclient/m1n1/hv/__init__.py", line 1595, in start_secondary
    self.p.hv_start_secondary(index, entry)
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
  File "/home/linux-host/sourcecodes/m1n1/proxyclient/m1n1/proxy.py", line 1072, in hv_start_secondary
    return self.request(self.P_HV_START_SECONDARY, cpu, entry, *args)
           ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/linux-host/sourcecodes/m1n1/proxyclient/m1n1/proxy.py", line 694, in request
    return self._request(opcode, *args2, **kwargs)
           ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/linux-host/sourcecodes/m1n1/proxyclient/m1n1/proxy.py", line 658, in _request
    reply = self.iface.proxyreq(req, reboot=reboot, no_reply=no_reply, pre_reply=None)
  File "/home/linux-host/sourcecodes/m1n1/proxyclient/m1n1/proxy.py", line 378, in proxyreq
    return self.reply(self.REQ_PROXY)
           ~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/home/linux-host/sourcecodes/m1n1/proxyclient/m1n1/proxy.py", line 249, in reply
    reply += self.readfull(1)
             ~~~~~~~~~~~~~^^^
  File "/home/linux-host/sourcecodes/m1n1/proxyclient/m1n1/proxy.py", line 186, in readfull
    block = self.dev.read(size - len(d))
  File "/home/linux-host/.local/lib/python3.13/site-packages/serial/serialposix.py", line 595, in read
    raise SerialException(
        'device reports readiness to read but returned no data '
        '(device disconnected or multiple access on port?)')
serial.serialutil.SerialException: device reports readiness to read but returned no data (device disconnected or multiple access on port?)
[cpu0] Guest exception: HV/HOOK_VM
Traceback (most recent call last):
  File "/home/linux-host/.local/lib/python3.13/site-packages/serial/serialposix.py", line 621, in write
    n = os.write(self.fd, d)
OSError: [Errno 5] Input/output error

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/linux-host/sourcecodes/m1n1/proxyclient/tools/run_guest.py", line 115, in <module>
    hv.start()
    ~~~~~~~~^^
  File "/home/linux-host/sourcecodes/m1n1/proxyclient/m1n1/hv/__init__.py", line 1936, in start
    self.p.hv_start(self.entry, self.guest_base + self.bootargs_off)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/linux-host/sourcecodes/m1n1/proxyclient/m1n1/proxy.py", line 1058, in hv_start
    return self.request(self.P_HV_START, entry, *args)
           ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/linux-host/sourcecodes/m1n1/proxyclient/m1n1/proxy.py", line 694, in request
    return self._request(opcode, *args2, **kwargs)
           ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/linux-host/sourcecodes/m1n1/proxyclient/m1n1/proxy.py", line 658, in _request
    reply = self.iface.proxyreq(req, reboot=reboot, no_reply=no_reply, pre_reply=None)
  File "/home/linux-host/sourcecodes/m1n1/proxyclient/m1n1/proxy.py", line 378, in proxyreq
    return self.reply(self.REQ_PROXY)
           ~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/home/linux-host/sourcecodes/m1n1/proxyclient/m1n1/proxy.py", line 291, in reply
    self.handle_boot(data)
    ~~~~~~~~~~~~~~~~^^^^^^
  File "/home/linux-host/sourcecodes/m1n1/proxyclient/m1n1/proxy.py", line 314, in handle_boot
    self.handlers[(reason, code)](reason, code, info)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File "/home/linux-host/sourcecodes/m1n1/proxyclient/m1n1/hv/__init__.py", line 986, in handle_exception
    self.update_pac_mask()
    ~~~~~~~~~~~~~~~~~~~~^^
  File "/home/linux-host/sourcecodes/m1n1/proxyclient/m1n1/hv/__init__.py", line 1837, in update_pac_mask
    tcr = TCR(self.u.mrs(TCR_EL12))
              ~~~~~~~~~~^^^^^^^^^^
  File "/home/linux-host/sourcecodes/m1n1/proxyclient/m1n1/proxyutils.py", line 172, in mrs
    return self.exec(op, call=call, silent=silent)
           ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/linux-host/sourcecodes/m1n1/proxyclient/m1n1/proxyutils.py", line 217, in exec
    self.iface.writemem(self.code_buffer, func)
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/linux-host/sourcecodes/m1n1/proxyclient/m1n1/proxy.py", line 384, in writemem
    self.cmd(self.REQ_MEMWRITE, req)
    ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/linux-host/sourcecodes/m1n1/proxyclient/m1n1/proxy.py", line 201, in cmd
    self.dev.write(command)
    ~~~~~~~~~~~~~~^^^^^^^^^
  File "/home/linux-host/.local/lib/python3.13/site-packages/serial/serialposix.py", line 655, in write
    raise SerialException('write failed: {}'.format(e))
serial.serialutil.SerialException: write failed: [Errno 5] Input/output error

[dgh05t]

Sometimes, host side doesn't crash but hangs, and guest OS hangs at apple logo (with rainbow version), proxyclient/tools/picocom-sec.sh has no output

[svenpeter42]

might be a duplicate of or related to #463

[tzwenn]

I am having the same issue with guest macOS 13.7.7 and 14.8.3 (stock and development kernel) (both on chainloaded m1n1 main and v1.5.2), just with different n in HV: Initializing secondary <n>.

I never see an apple logo, sometimes just an empty loading bar for brief moment.

If it instead just hangs, it is typically on something like:

[cpu7] Pass: mrs x13, HID1_EL1 = 1440000002000000 (HID1_EL1)
[cpu7] Pass: msr HID1_EL1, x13 = 1440000002000000 (OK) (HID1_EL1)
[cpu7] Pass: mrs x13, HID18_EL1 = 2000040004000 (HID18_EL1)
[cpu7] Pass: msr HID18_EL1, x13 = 2000040004000 (OK) (HID18_EL1)

I could try to provide more info on where that is, if you have a primer on m1n1 development/debugging. This one seems still short atm.

Device Info	when booting 13.7.7
Model	M1 MacMini 2020
OS FW version	13.6.2 (iBoot-8422.141.2.700.1)
System FW version	unknown (iBoot-13822.61.10)

[tzwenn]

Update, maybe for bisection: randomly picked m1n1 v1.14.4 and the 13.7.7. guest (KDK) started.

[svenpeter42]

I have tried bisecting this but always ended up at a different commit. My best guess is that some early commit introduced this error and then later commits made it much more likely to happen. Would be very nice if you manage to bisect it.

What is your host machine and which compiler did you use for m1n1?

[tzwenn]

Host: 2020 M1 MacBook Pro, currently booted into macOS 14.8.3.
Does it make a difference in your experience?

• Homebrew clang version 21.1.8
• cargo/rust 1.92.0 (rustup, with target aarch64-unknown-none-softfloat of course).
• Homebrew LLD 21.1.8

Happy to help bisecting later with other host devices/OS and compiler/m1n1 versions, once I finish today's experiments. The guest turned out slower than expected 🙈

[svenpeter42]

If it's the same as #463 which I strongly suspect compiling with gcc will make it go away. For bisecting you ideally need a setup where it almost always crashes and then probably try each step at least 10 times before you mark the commit as good. I tried this before but only ran 2 attempts before I marked something as good and ended at different original commits that all turned out to be unrelated.

[tzwenn]

Can confirm, building the main branch with aarch64-linux-gnu-gcc 15.1.0 and rustc 1.94.0-nightly on a Linux host works with nothing else changed.

Chainloading the macOS/llvm-binary of v1.14.4 and starting the macOS guest from my Linux host resulted in one freeze out of three runs (again as soon as the menu bar shows up).

[svenpeter42]

I guess that hang you see might just be another symptom of the same bug then. I've only seen the crash when bringup up the secondary cores here