Make it work on 16k page size Arm kernel

[illwieckz]

Attempt to to make it work on 16k page size Arm kernel.

This is work in progress, this doesn't run.

This is based over:

• Write a CMakeLists.txt to build the NaCl loader #2

The commit is:

• cmake: make possible to build sel_ldr with 16K page size

What this does:

• make sure the nacl_loader binary itself is 16K pageSize compatible.

What this doesn't do:

• patch the internal behavior of the loader itself.

It is possible that the loader itself does nasty things that aren't compatible with a 16K page size kernel, especially when reserving the memory or things like that.

At least it is confirmed that this patch doesn't break the loader when running on a 4K page size kernel.

It is not enough to fix the 16K page size issue, but at least it doesn't break the 4K page size issue.

[illwieckz]

I noticed the nexe (including the irt one) already use a 64k page size, so I brought 16k page size 32-bit arm libraries from a recent Debian, and I recompiled the loader for 16k page size (0x1000: 4k, 0x4000: 16k, 0x10000, 64k).

nacl_helper_bootstrap:
  LOAD           0x000000 0x00010000 0x00010000 0x01071 0x01071 R E 0x4000
  LOAD           0x000000 0x00014000 0x00014000 0x00000 0x01005 RW  0x4000
  LOAD           0x002000 0x00016000 0x00016000 0x3ffec000 0x3ffec000     0x4000
  LOAD           0x002000 0x40002000 0x40002000 0x00014 0x00014 RW  0x4000
nacl_loader:
  LOAD           0x000000 0x00000000 0x00000000 0xbd154 0xbd154 R E 0x4000
  LOAD           0x0bf940 0x000c3940 0x000c3940 0x08f14 0x1b668 RW  0x4000
irt_core-armhf.nexe:
  LOAD           0x010000 0x0ffc0000 0x0ffc0000 0x30000 0x30000 R E 0x10000
  LOAD           0x000000 0x3efe0000 0x3efe0000 0x041b8 0x041b8 R   0x10000
  LOAD           0x0041b8 0x3eff41b8 0x3eff41b8 0x00b90 0x011b0 RW  0x10000
helloworld.nexe:
  LOAD           0x010000 0x00020000 0x00020000 0x20000 0x20000 R E 0x10000
  LOAD           0x000000 0x10020000 0x10020000 0x008ec 0x008ec R   0x10000
  LOAD           0x0008ec 0x100308ec 0x100308ec 0x0168c 0x01c60 RW  0x10000
lib-armhf/ld-linux-armhf:
  LOAD           0x000000 0x00000000 0x00000000 0x1b4ac 0x1b4ac R E 0x10000
  LOAD           0x01ee70 0x0002ee70 0x0002ee70 0x017cc 0x01bb4 RW  0x10000
lib-armhf/ld-linux-armhf.so.3:
  LOAD           0x000000 0x00000000 0x00000000 0x1b4ac 0x1b4ac R E 0x10000
  LOAD           0x01ee70 0x0002ee70 0x0002ee70 0x017cc 0x01bb4 RW  0x10000
lib-armhf/libc.so.6:
  LOAD           0x000000 0x00000000 0x00000000 0x1106b4 0x1106b4 R E 0x10000
  LOAD           0x11e80c 0x0012e80c 0x0012e80c 0x02454 0x0ba58 RW  0x10000
lib-armhf/libgcc_s.so.1:
  LOAD           0x000000 0x00000000 0x00000000 0x17500 0x17500 R E 0x10000
  LOAD           0x01ff00 0x0002ff00 0x0002ff00 0x001f8 0x00230 RW  0x10000
lib-armhf/libm.so.6:
  LOAD           0x000000 0x00000000 0x00000000 0x47cfc 0x47cfc R E 0x10000
  LOAD           0x04fe88 0x0005fe88 0x0005fe88 0x00180 0x00188 RW  0x10000
lib-armhf/libpthread.so.0:
  LOAD           0x000000 0x00000000 0x00000000 0x007d4 0x007d4 R E 0x10000
  LOAD           0x00fecc 0x0001fecc 0x0001fecc 0x00138 0x0013c RW  0x10000
lib-armhf/librt.so.1:
  LOAD           0x000000 0x00000000 0x00000000 0x0089c 0x0089c R E 0x10000
  LOAD           0x00feb4 0x0001feb4 0x0001feb4 0x00150 0x00154 RW  0x10000
lib-armhf/libstdc++.so.6:
  LOAD           0x000000 0x00000000 0x00000000 0x18b3d4 0x18b3d4 R E 0x10000
  LOAD           0x19b134 0x0019b134 0x0019b134 0x06a94 0x0944c RW  0x10000

On a 4k page size kernel:

./nacl_helper_bootstrap nacl_loader --r_debug=0xXXXXXXXXXXXXXXXX --reserved_at_zero=0xXXXXXXXXXXXXXXXX -B irt_core-armhf.nexe -e -i 100:57 -- helloworld.nexe 
[2340,4154064384:08:48:06.594309] Native Client module will be loaded at base address 0x0000000000000000
hello world

On a 16k page size kernel:

./nacl_helper_bootstrap nacl_loader --r_debug=0xXXXXXXXXXXXXXXXX --reserved_at_zero=0xXXXXXXXXXXXXXXXX -B irt_core-armhf.nexe -e -i 100:57 -- helloworld.nexe 
Segmentation fault

At least the loader executable itself launchzs:

./nacl_loader
No nacl file specified

Without that rebuild trick we would get a segfault just at executable launch:

./nacl_loader 
Segmentation fault

In the same way the bootstrap itself launchzs:

./nacl_helper_bootstrap
bootstrap_helper: Usage: PROGRAM ARGS...

But nothing more is done.

[illwieckz]

Using nacl_bootstrap_raw as a loader (not the one processed with the custom linker stuff as discussed there), it works:

$ getconf PAGESIZE
16384

$ ./nacl_bootstrap_raw nacl_loader --r_debug=0xXXXXXXXXXXXXXXXX --reserved_at_zero=0xXXXXXXXXXXXXXXXX -B irt_core-armhf.nexe -e -i 100:57 -- helloworld.nexe
[3095,4134395008:19:57:00.667157] Native Client module will be loaded at base address 0x0000000000000000
hello world

So we can get it working!

[illwieckz]

Those two screenshots were taken on a Raspberry Pi 5 running the stock RaspiOS with the default 16k page size kernel. This is the released 0.55.0 nexes running on the recompiled loader from this branch, using nacl_bootstrap_raw as a bootstrap:

[illwieckz]

[illwieckz]

Since it works with the nacl_bootstrap_raw binary but not with the nacl_helper_bootstrap one, I guess some change has to be done to the nacl_bootstrap_munge_phdr.py file to adjust something.

[illwieckz]

It now works:

./nacl_helper_bootstrap sel_ldr --r_debug=0xXXXXXXXXXXXXXXXX --reserved_at_zero=0xXXXXXXXXXXXXXXXX \
 -B irt_core.nexe -e -i 100:57 -l /dev/null -- hello.nexe
Hello World!

Some explicit alignment was needed in the nacl_boostrap.x file.

[illwieckz]

Here we see a simple hello world nexe running on a 16k Arm kernel, now using the nacl_helper_bootstrap tool:

I also tested running a complete game with both cgame and sgame nexe, it worked.

[illwieckz]

I added patches for SCons and GN but I haven't tested them.

[illwieckz]

I noticed that arm libraries on Debian are compiled with a 64k page size, we may do the same to be consistent and future proof. Arm can run 4k, 16k or 64k page size binaries, but the minimum requirement is selected by the way the Linux kernel is built. Debian went from 4k to 16k as a minimum default when going from Bookworm to Trixie. A kernel with a small page size can run binaries from higher page sizes since they're multiple and are then always aligned, while the opposite is not true. So we can do 64k page size right now.

[illwieckz]

Anyway this has an impact on the binary size:

While nacl_bootstrap_raw always weights 33464 bytes, when using 4k, 16k, or 64k page sizes, nacl_helper_bootstrap weights 41656, 90808 or 287416 bytes.

I confirm that the 64k boostrap and loader are working though.

[illwieckz]

Well, it looks like not everything has to be aligned…

This can reduces the sizes to 37560, 66232 or 164536.

[illwieckz]

While nacl_bootstrap_raw always weights 33464 bytes

That part of the comment was wrong, I looked at the wrong file. The rest of the comment is right.

[illwieckz]

I aligned less things and it still runs with 16k and 64k page size, while the size is now 33464.

Which, I now do confirm, is also the size of nacl_bootstrap_raw.

So we not only build a large page binary, but we do enough alignment to keep it working, while not increasing its size from a single byte.

I also get 33464 bytes with a 64k page size binary, so we may just do that.

[illwieckz]

Something we should never do is to strip the helper bootstrap, it breaks it:

$ strip nacl_helper_bootstrap
strip: stNOtSHY: section .reserve lma 0x30000 adjusted to 0x40034
$ ./nacl_helper_bootstrap sel_ldr --r_debug=0xXXXXXXXXXXXXXXXX --reserved_at_zero=0xXXXXXXXXXXXXXXXX -B irt_core.nexe -e -i 100:57 -l /dev/null -f hello.nexe
Segmentation fault

[illwieckz]

Functional 64k page size loader on a 16k page size kernel:

The libc, the bootstrap, the loader and the nexe are all 64k page size.

[illwieckz]

We can strip nacl_bootstrap_raw before being processed to become nacl_helper_bootstrap and then it still works, and the size is divided by 4:

$ du -sb nacl_bootstrap_raw nacl_helper_bootstrap
8788	nacl_bootstrap_raw
8788	nacl_helper_bootstrap

$ ./nacl_helper_bootstrap sel_ldr --r_debug=0xXXXXXXXXXXXXXXXX --reserved_at_zero=0xXXXXXXXXXXXXXXXX \
 -B irt_core.nexe -e -i 100:57 -l /dev/null -- hello.nexe
Hello World!

[illwieckz]

We can strip nacl_bootstrap_raw before being processed to become nacl_helper_bootstrap and then it still works

Also when we do that, we can then run strip on nacl_helper_bootstrap and it doesn't break because there is nothing to strip anymore so it does nothing, meaning rogue strip calls would not break it.

[slipher]

We can strip nacl_bootstrap_raw before being processed to become nacl_helper_bootstrap and then it still works, and the size is divided by 4:

That's roughly the same as what happens when using the orphan handling discard flag.

[illwieckz]

Maybe the strip fixes the build on Ubuntu? 😅️

[illwieckz]

Maybe the strip fixes the build on Ubuntu? 😅️

That cannot. Actually the strip breaks the linux-amd64 binary:

nacl_bootstrap_munge_phdr: Program header 2 has nonzero p_filesz

[illwieckz]

The orphan handling discard flag only reduces to 10676 bytes, but at least it doesn't break it…

[illwieckz]

@slipher actually that --orphan-handling=discard flag looks very good to me, please do a PR!