AEM doesn't work on Dell Optiplex 7010 in UEFI mode

[krystian-hebel]

Affected component(s) or functionality
The component or functionality of TrenchBoot that is not working as expected.

AEM

Brief summary
A clear and concise summary of the bug.

AEM does not boot on this platform in UEFI mode.

Version
The version of the component or feature if available, otherwise NA.

v0.5.0-rc3

To Reproduce
Steps to reproduce the behavior:

1. Install AEM v0.5.0 under Qubes OS for legacy and UEFI
2. Try to boot with AEM enabled

Expected behavior
A clear and concise description of what you expected to happen.

System boots normally.

Actual behavior
What actually happened (instead of what you expected to happen).

Platform reboots around the time when Xen is supposed to jump to slaunch stub in GRUB, no TXT error is produced. With additional debug enabled in GRUB (set debug=linux,relocator,multiboot,multiboot_loader,slaunch,chainloader), the platform hangs instead.

Screenshots
If applicable, add screenshots to help explain your problem.

Test 3501 on our internal openQA.

Additional context
Add any other context about the problem here.

Boot log with debug in GRUB: aem.log. No additional lines appear after adding debug options to xen-4.17.5.cfg.

Solutions you've tried
If applicable, any solutions or workarounds you've already tried.

• Booting AEM entry with slaunch and slaunch_module removed: system boots, complaining about PCRs being all Fs, which is expected
• Booting non-AEM entry with slaunch added: platform reboots, TXT reports error 0xc0140431 - I wasn't able to decode this particular minor number, but it has major type of ERR_BASIC_RANGE_CHECK
• The same AEM version in legacy mode also reboots without the fix for AEM not booting on Intel legacy boot xen#22.

Relevant documentation you've consulted
A list of links to any relevant documentation you have consulted.

Related, non-duplicate issues
A list of links to other bug reports, feature requests, or tasks in the
trenchboot-issues tracker (or "none" if you didn't find any). Do not describe
any other unreported bugs, features, or tasks here.

[SergiiDmytruk]

With additional debug enabled in GRUB (set debug=linux,relocator,multiboot,multiboot_loader,slaunch,chainloader), the platform hangs instead.

There seems to be something about grub_dprintf() that doesn't work when GRUB continues execution like that. Replacing such calls with grub_printf() helps to avoid the hang.

[SergiiDmytruk]

This seems to be the same issue as TrenchBoot/xen#22 and fix for that issue makes AEM work here in legacy mode. The platform has some really weird problems with Qubes ISO on PiKVM, so I had to make network boot work for legacy and UEFI cases to be able to test. Need to get firmware with TPM1.2 fix as I ran openQA on the version without the fix and it couldn't clear TPM.

[krystian-hebel]

The platform has some really weird problems with Qubes ISO on PiKVM, so I had to make network boot work for legacy and UEFI cases to be able to test.

SeaBIOS doesn't support composite USB devices, it breaks out of loop on first supported interface. Luckily, the first one is keyboard, otherwise none of our tests would work. UEFI works with the PiKVM ISO just fine for me.

I worked around this by plugging in physical USB stick with Qubes OS installer (platform with asset ID 105), but it fails to boot the installer most of the times in legacy mode:

dmesg shows some disturbing errors, not sure if they are related:

I tried with different USB sticks in different ports, although always on the front panel, it didn't change the outcome. I was able to boot to the first GUI screen of the installer once by choosing kernel-latest, but I wasn't able to repeat it later.

[SergiiDmytruk]

UEFI works with the PiKVM ISO just fine for me.

I haven't actually tried it because I incorrectly assumed a hardware-related issue. Network boot for UEFI doesn't work as expected though, openQA sends F12 too late for it to have an effect as if it used to boot slower or maybe something adds an extra delay now. I ended up making network boot the first boot option.

[SergiiDmytruk]

This seems to be the same issue as TrenchBoot/xen#22 and fix for that issue makes AEM work here in legacy mode.

And only in legacy mode. UEFI fails before reaching Xen's code which restores MTRRs.

[SergiiDmytruk]

Tried adjusting some alignment/range-related things in GRUB, but Slaunch boot still fails with 0xc0140431. It's the same error I saw on an OptiPlex earlier and suspected it to be firmware related, tried to flash original firmware and got a brick, so I don't want to repeat the test.

I did, however, try with tboot and got the same error code, so assuming tboot does everything right for the platform either DRTM doesn't work on OptiPlex with Dasharo+UEFI or it doesn't work on OptiPlex with UEFI (need a firmware test).

ERR_BASIC_RANGE_CHECK is described as "One of ranges in array failed the check: alignment is incorrect top address is below base Range placement is incorrect as required by its attributes" (original punctuation preserved). Possible minor errors (array elements?):

• ERR_DMAR_TAB
• ERR_OSSINIT_TAB
• ERR_LAPIC
• ERR_PMR
• ERR_PRIMARY
• ERR_SECONDARY
• ERR_VTD_IA_EXC
• ERR_VTD_GT_EXC
• ERR_ACPI_TAB
• ERR_MLE_PD_PAGE1
• ERR_MLE_PD_PAGE2
• ERR_MLE_PT_PAGE
• ERR_MLE_PAGE

This looks like SINIT can potentially dislike placement of ACPI tables, although since TXT works for UEFI on VP4670 maybe that's not the issue (EDK2 should behave similarly on both platforms unless coreboot causes variance). Also, the list above is incomplete and meaning of some of the fields isn't clear, so it's hard to say what SINIT could be checking.

[krystian-hebel]

I did, however, try with tboot and got the same error code, so assuming tboot does everything right for the platform either DRTM doesn't work on OptiPlex with Dasharo+UEFI or it doesn't work on OptiPlex with UEFI (need a firmware test).

I was convinced we had a positive test results for UEFI, but there are no such results in openQA, I must be imagining things. It is interesting that tboot has the same problem.

This looks like SINIT can potentially dislike placement of ACPI tables, although since TXT works for UEFI on VP4670 maybe that's not the issue (EDK2 should behave similarly on both platforms unless coreboot causes variance).

They have different ACMs, with different bugs and quirks, they may behave differently. On top of that, VP4670 has TPM2.0, and while it probably isn't the direct cause (there is a dedicated class for TPM errors), it may move something around in memory or in time. AMD was requiring some delay between INIT and SKINIT, maybe something similar happens here.

I'm currently testing the same packages with legacy Optiplex, if it doesn't work there as well, we will know that it isn't caused by UEFI, but may still be caused by a different coreboot revision.

[SergiiDmytruk]

I'm currently testing the same packages with legacy Optiplex, if it doesn't work there as well, we will know that it isn't caused by UEFI, but may still be caused by a different coreboot revision.

Legacy should work fine, it's only UEFI that had issues, but TXT has at least one UEFI-specific field, so that can be a factor. I would have checked against the stock UEFI firmware, but it requires flashing both flash chips and since I bricked OptiPlex twice already (once to do this test), I think it will happen again and won't be able to run the check anyway. If you have had success with stock <-> Dasharo transitions in the past, maybe you could try.

[krystian-hebel]

There are 2 TCPA entries in ACPI, one made by edk2, the other by coreboot (http://192.168.4.32/tests/3815/logfile?filename=serial0.txt#line-191 and few lines below). In case of VP4670, there is only one TPM2 entry: http://192.168.4.32/tests/3446/logfile?filename=serial0.txt#line-364. FACS is also listed twice (although only in dom0 kernel output), but at least it points to the same address, and it is duplicated in UEFI and legacy alike, so probably irrelevant. TCPAs produced by each component is different:

TPM memory also isn't reserved according to dom0 kernel in case of UEFI ([ 0.000000] Xen: [mem 0x00000000fed90000-0x00000000fed91fff] reserved, while it is for legacy ([ 0.000000] Xen: [mem 0x00000000fed20000-0x00000000fed91fff] reserved).

I've found some old TXT error document (not sure where it comes from, it was laying on my disk as sinit_errors.txt) that have slightly different assignment of error codes. According to it, this error would mean DEF_TYPE MTRR not UC or fixed MTRRs not disabled, but those requirements are met.

If you have had success with stock <-> Dasharo transitions in the past, maybe you could try.

I hadn't tried it yet. According to https://docs.dasharo.com/variants/dell_optiplex/recovery/ it requires external flashing, and there is not enough space in the rack to try to attach second clip without accidentally disconnecting something. It also doesn't say how to obtain the backup, which is no longer part of initial deployment since it was obsoleted by DTS. Internal flashing is bound to fail, ME doesn't like having its flash region written at runtime.