Guest kernel uses ring-0 interrupts

[chc4]

In https://info.varnish-software.com/blog/tinykvm-the-fastest-sandbox it says:

We also enter in usermode and try to avoid kernel mode except where impossible. Did you know that you can handle CPU exceptions in usermode on x86?

However, tinykvm doesn't seem to actually do this.

tinykvm/lib/tinykvm/amd64/idt.cpp

Line 63 in b517667

set_entry(idt.entry[vec], handler, 0x8, IDT_PRESENT | IDT_CPL0 | IDT_GATE_INTR);

only sets up a single IDT gate which runs at ring-0 (CPL0), and with ring-0 segment selectors.

I am using tinykvm for a side project which wants specifically ring-3 -> ring-3 interrupts for speed reasons, and so this behavior was surprising to me. I was able to add a second CPL3 gate, along with another usermode accessible mapping of the kernel interrupt code and usermode accessible interrupt stack, fairly easily so this isn't a blocker, but I wasn't sure if this was unintentional behavior or a design decision which has changed since the block post was written.

[fwsGonzo]

Yep, I don't remember why I did that. Perhaps because I didn't have the usermode assembly at the time, or perhaps because the PIT timer I was programming required kernel mode and I wanted to have an array of (sequential) interrupts? I just don't remember, but... I found my old question, and apparently I was doing it initially: https://stackoverflow.com/questions/66233771/handle-cpu-exceptions-outside-of-kvm/66656234#66656234

It's nice to hear that it still works! Perhaps this should be revisited again. There might be interrupts (except page fault) that really don't need to execute in kernel mode. Page fault has to be CPL0 because of invlpg and invpcid. For the case where you are running a kernel without copy-on-write, you can of course run everything in usermode using an alternate interrupts assembly.

[chc4]

Ah, that makes sense. I'm only using a ring-3 -> ring-3 interrupt for a specific interrupts, so the CoW fault would still run at the correct privilege level. I don't know how useful it will be to you, but chc4/tinycov@64371cd is the hacky commit I made for adding a usermode interrupt for reference.

[fwsGonzo]

That's really cool shit. Btw. I have never measured if it's faster to just set exception handler to unmapped memory, which might trigger a KVM_EXIT_MMIO (or something like that) directly. It could be faster?

In any case, the exception handling API is (as you saw) very rudimentary, and it deserves a little polish. If nothing else, to add support for usermode exceptions. It's interesting that it's faster than privilige switching. Also would be interesting to see if SYSCALL+SYSRET combo is faster than a privilege switch, since it's supposed to be only MSR-dependent, and not use memory. The latest design in TinyKVM avoids most CPU exceptions, though, as the reset mechanism uses non-temporal streaming to roll back to a previous good state after a request. But, for debugging, and for your use-case, there are probably many interesting ideas still to be tried.

[chc4]

I did a small benchmark, and using a ring 3 -> ring 0 interrupt in an int3-heavy workload is 57% (!) slower than the ring 3 -> ring 3 one. In practice I expect this to be much less impactful for most of tinykvm's other usecases, which aren't spending a large percentage of their time servicing interrupts, but still potentially a useful touchpoint.