More architectures like arm32, riscv etc

[wader]

Current issues are:

• Problem finding hosting providing native ARM32, RISCV etc hosts
• Cross compile is messy
• Emulated docker builds a bit less messy but have issues:
  • Emulation is user-land only so /proc/cpuinfo etc from the host leaks into the container confusing some build systems (ex: aom ARM32 build gets confused about CPU features, I think it looks as cpuinfo)
  • How to verify that successful build actually was built with CPU features we want?
• Emulated build is slow and uses lots of resources
  • Current static-ffmpeg with all features can't be build on github action host. Runs out of memory and would probably also run out of CPU time (6h max jobs time). Biggest memory/CPU user is rust it seems, without that it might work.

Some details and attempts here #204

[wader]

@mathieu-aubin let's move arch discussion here

did you read this -> https://www.docker.com/blog/faster-multi-platform-builds-dockerfile-cross-compilation-guide/

and of so, is it relevant? not much time npw but i figured id input

Yeap docker buildx is one thing i'vet tried and it works similar to how the multi-arch images work, does emulation in user-land. It has the issues i described in above.

this one too -> https://hub.docker.com/r/linuxserver/ffmpeg

Had a quick look, would guess they build on native hosts or on fully emulated hosts for each arch https://github.com/linuxserver/docker-ffmpeg/blob/226ddf020af76b99b71dbe3804cfca1919f83c11/Jenkinsfile#L466

It would be interesting to do a github action that boots up a full emulated host per arch. I did some experiments with this once and building worked fine for arm32 etc. The messy part is probably to make it nice to work from a workflow, maybe ideally it could be an action that kind of is a docker action that transfers the job env and runs docker inside the guest.

But sadly if run on a github action host it will still run into the same resources limits that i mentioned above.

[gtsteffaniak]

I maintain https://github.com/gtsteffaniak/filebrowser which I recently started using this image as a source.

I appreciate the statically linked binaries and recent ffmpeg version support this repo offers, which is why I chose it. However, it is problematic that it doesn't have armv7 support.

I have lots of multi arch docker experience and it's not hard at all with buildx. There's a lot that could actually be simplified by using it.

I would love to contribute in order to add support for arm32. Since that's missing I can't fully use this image for all my docker images.

Let me know if you would be open to a contribution in that way.

[wader]

I maintain https://github.com/gtsteffaniak/filebrowser which I recently started using this image as a source.

🥳

I appreciate the statically linked binaries and recent ffmpeg version support this repo offers, which is why I chose it. However, it is problematic that it doesn't have armv7 support.

Yeap i would like to support it too if we could figure out a good way of doing it.

I have lots of multi arch docker experience and it's not hard at all with buildx. There's a lot that could actually be simplified by using it.

As i mentioned above i the issue description i have done some experiments using emulated docker buildx but ran into som issues, slowness, hitting resource limits and confusing about target arch, that made me not very confident in the approach. But maybe things have change? at least the slowness i guess could be fixed now by using a arm64 host to run/emulate arm32?

I would love to contribute in order to add support for arm32. Since that's missing I can't fully use this image for all my docker images.

Let me know if you would be open to a contribution in that way.

That would be great!

I think the solution with the least amount changes, maintenance and future headaches would be to somehow build on a action runner that is native arm32. At the moment i use ubicloud for arm64 but i don't see arm32 in their documentation but maybe there is ways to do it?

[gtsteffaniak]

Thanks for the reply - It is a very reliable and consistent process, but you might be on to something about some timeouts for the emulated versions, because they do take 3-4 times longer and I see your builds are 1h+ right now.

But, the way you are doing multiarch could be improved in both performance and simplicity. I will experiment and make a pull request later.

[wader]

Thanks for the reply - It is a very reliable and consistent process, but you might be on to something about some timeouts for the emulated versions, because they do take 3-4 times longer and I see your builds are 1h+ right now.

Yeap if i remember correctly emulated rust builds are very slow and i think also sometimes runs out of memory on github action runners. But what i'm mostly concern about is that docker buildx is a user-land only emulation so some details of the actual host will leak thru via /proc etc which in turn can confuse compilers and build systems unless you explicitly tell them what to target. If done that way i think it would be good to somehow verify the binaries that they really are only the target ISA so that later on when updating some library incompatible instructions sneaks in, this has even happen when doing totally native builds (ex fraunhoferhhi/vvenc#493)

But, the way you are doing multiarch could be improved in both performance and simplicity. I will experiment and make a pull request later.

Ok! one think that i do value with the current Dockerfile is that it is straight forward and quite easy to follow so it would be great to not introduce too exotic stuff.

[gtsteffaniak]

Yeah I really like how simple it is, I'm not looking to rock the boat with any major changes, just get it building with arm32 as well.

You might be right that the user space emulation could cause some issues. I'll see, qemu has improved a lot, but mostly for arm64 due to apple m series.

One question I had, why do you download and build a lot of the build dependencies from source statically? My thinking was ffmpeg was the only thing that needs to be built statically. And besides version controlling, downloading and building everything before ffmpeg is 90% of the build time.

[wader]

Yeah I really like how simple it is, I'm not looking to rock the boat with any major changes, just get it building with arm32 as well.

ideally for me would be more or less no changes and somehow get a native arm32 action runner env somehow.

You might be right that the user space emulation could cause some issues. I'll see, qemu has improved a lot, but mostly for arm64 due to apple m series.

Yeap i would imagine that arm32 emulated on arm64 could be very fast? but i haven't tried.

One question I had, why do you download and build a lot of the build dependencies from source statically? My thinking was ffmpeg was the only thing that needs to be built statically. And besides version controlling, downloading and building everything before ffmpeg is 90% of the build time.

You mean why not use alpine static packages? or you mean download binary versions of the library somehow? there is a mix of reasons. Why no alpine: because sometimes the alpine package is not updated as fast, sometimes there is no static package (some i've filed bugs about), sometimes i want to build the library a bit differently. Why from source: Some libraries i want to build with hardened flags (especially decoders), sometimes there is no alpine package. So it's a bit of a mix bag, no general rule really.

[gtsteffaniak]

Yeah I am understanding it a bit more as I attempt to build .

Looks like there's a few libraries that have issues , I still need to see if I can figure out support:

• cairo build always fails only on armv7, needed for librsvg, so for now I have that disabled (only for armv7)
• aom requires disabling CPU optimizations for arm32, so I added a condition for that:

# build with generic CPU for armv7l. see cross compiling https://aomedia.googlesource.com/aom
RUN cd aom && \
  if [[ $(uname -m) == "armv7l" ]]; then GENERIC_CPU="-DAOM_TARGET_CPU=generic " else GENERIC_CPU="" fi && \
  mkdir build_tmp && cd build_tmp && \
  cmake \
    -G"Unix Makefiles" \
    -DCMAKE_VERBOSE_MAKEFILE=ON \
    -DCMAKE_BUILD_TYPE=Release \
    -DBUILD_SHARED_LIBS=OFF \
    -DENABLE_EXAMPLES=NO \
    -DENABLE_DOCS=NO \
    -DENABLE_TESTS=NO \
    -DENABLE_TOOLS=NO \
    -DCONFIG_TUNE_VMAF=1 \
    -DENABLE_NASM=ON \
    -DCMAKE_INSTALL_LIBDIR=lib \
    $GENERIC_CPU \
  .. && \
  make -j$(nproc) install

• There's also an issue with xavs2 on arm32, still looking at whether that can be fixed

[gtsteffaniak]

I am testing on an m4 MacBook , while the compile times are faster, arm32 emulation is quite slow in general -- but it does work with qemu. I will test the ci builds to see how long they take, locally armv7 builds take ~20 minutes.

Also, since buildx all happens on the same node, I am using a prefetch sources script to download all the versions once beforehand and unzip them. This should also save some time since it happens on the native node.

Once the branch I have is ready with the optimizations , ill let you decide how you want to include it.

But I'll raise a PR to do a simple update to the dockerfile to support arm32 and update the GitHub action to use buildx.

[wader]

Yeah I am understanding it a bit more as I attempt to build .

Looks like there's a few libraries that have issues , I still need to see if I can figure out support:

• cairo build always fails only on armv7, needed for librsvg, so for now I have that disabled (only for armv7)

Any idea what it fails on? the librsvg build system is quite complex unfortunately, meson with python scripts that wrap cargo 😬 maybe arm64 build host feature leaking into target?

• aom requires disabling CPU optimizations for arm32, so I added a condition for that:

Maybe same as above about arm64 build host features leaks into target?

# build with generic CPU for armv7l. see cross compiling https://aomedia.googlesource.com/aom
RUN cd aom && \
  if [[ $(uname -m) == "armv7l" ]]; then GENERIC_CPU="-DAOM_TARGET_CPU=generic " else GENERIC_CPU="" fi && \
  mkdir build_tmp && cd build_tmp && \
  cmake \
    -G"Unix Makefiles" \
    -DCMAKE_VERBOSE_MAKEFILE=ON \
    -DCMAKE_BUILD_TYPE=Release \
    -DBUILD_SHARED_LIBS=OFF \
    -DENABLE_EXAMPLES=NO \
    -DENABLE_DOCS=NO \
    -DENABLE_TESTS=NO \
    -DENABLE_TOOLS=NO \
    -DCONFIG_TUNE_VMAF=1 \
    -DENABLE_NASM=ON \
    -DCMAKE_INSTALL_LIBDIR=lib \
    $GENERIC_CPU \
  .. && \
  make -j$(nproc) install

• There's also an issue with xavs2 on arm32, still looking at whether that can be fixed

Okok, looks like i already disable some asm stuff for xavs2, is probably assume amd64 etc so probably ok with more hacks.

As i mention one thing that would be great would be to somehow test or verify that binaries that they actually are using only the ISA they are suppose to use, but not sure how that could be done. I might take a look into that, should maybe be done separate from this?