RCD 🤝 Rust (my experience)

[ianks]

I maintain a project called rb-sys that builds on top of the rake-compiler ecosystem, so building native extensions with Rust would have the same workflow as a C extension. I wanted make it easy to precompile Rust gems for multiple platforms in a repeatable and maintaintainable way. Naturally, I turned to RCD.

After chatting with @flavorjones a bit, he nudged me to document my experience using rake-compiler-dock in the rb-sys project (thanks Mike!).

Overall, I'm really happy with how everything has turned out. RCD is such an important gem for Ruby, and I'm grateful for all of the great work that has been put into it.

(ps: I apologize in advance for the massive brain dump, but I knew it was the only way I could document this stuff!)

How I used RCD to compile Rust extensions

Here's a brief overview of things I did to make Rust extensions work with RCD. It's kind of a brain dump, but I hope it's useful.

• I created a bunch of platform-specific dockerfiles which all
  inherit FROM larskanis/rake-compiler-dock-mri-$PLAT:$VERSION.
• Each dockerfile installs the Rust toolchain and any other dependencies.
• There's a CI job that builds the docker images and pushes them to dockerhub after each release.
• Since the rb_sys gem knows how to generate a compatible Makefile from an extconf.rb, users can just use RCD normally by specifying the RCD_IMAGE=rbsys/$RUBY_PLATFORM:$RB_SYS_VERSION environment variable.
• I also made a GitHub Action to make this process a bit easier.

Stumbling blocks

I ran into a few issues while trying to get this to work. I'll try to document them here. Some of them are Rust specific, so take those as you will.

Bundler and rake-compiler-dock (user experience)

rake-compiler-dock doesn't ensure that bundle install is run for each RUBY_CC_VERSION. This means that any Rakefile that uses bundler/setup will fail with Bundler::GemNotFound (example here). For users of RCD, this is not an intuitive fix, as it's not obvious what the problem is. I've seen a few people run into this feel "actually a bit stuck" (their words).

Essentially, the manual fix is to do something like:

1. Do the bundle yourself: rvm 3.1 && bundle install && rvm 3.0 && bundle install ...
2. Be very cautious and make sure to not include bundler/setup or extra gems in your Rakefile.

IMO, neither options is ideal. I think it would be nice if RCD could run bundle install for each RUBY_CC_VERSION before invoking the commands. This would be an easy win and make RCD much more user friendly!

OS differences

Now this is a more general issue with the cross-platform ecosystem, and not RCD specific. However, I think it's worth mentioning.

Depending on which platform you are compiling for, you may have an entirely different OS / package-manager. Some are redhat, some are debian, and the versions seem to be inconsistent. This poses a challenge when installing dependencies. Typically, it's best practice to compile any gem native libs yourself (e.g. with miniportile), but for certain things (like LLVM, CMake, etc) this is not feasible. This puts us in a situation where were have differing versions of these deps depending on the platform. Adds some complexity to the build process.

CC, CXX, AR, LD and the like

This one is not a huge deal, but it has tripped me up a few times. When compiling a third party library, you often need to make sure that the CC, CXX, AR, LD and other environment variables are set correctly.

In the rb-sys dockerfile, I've hardcoded sane defaults for the Rust ecosystem like so:

ENV CC_arm_unknown_linux_gnueabihf="arm-linux-gnueabihf-gcc" \
    CXX_arm_unknown_linux_gnueabihf="arm-linux-gnueabihf-g++" \
    AR_arm_unknown_linux_gnueabihf="arm-linux-gnueabihf-ar"

IIRC, rake-compiler-dock does not do this automatically set CC/CXX for you. I wonder if it should?

Mounting cache directories

By default, rake-compiler-dock mounts the ./tmp directory so things are cached nicely. For Rust, I would also like to be able to cache the ./target directory. I finagled this once, but I don't remember how. I think it's possible, but it's not obvious how to do it. I wonder if RCD should support some type of configuration file to make this type of thing easier?

# .rake-compiler-dock.yml ????
global:
  extra_mounts: ["./target:/wherever/the/gem/is/target"]
  env:
    BAR: baz

x86_64-linux:
  image: larskanis/my-custom-mri-x86_64-linux:latest
  env:
    FOO: bar

Different compilers

This is probably biased for the Rust world, but it would be amazing Ruby were built using clang + lld for every platform (ideally, the same version of clang for each). This would fix so many headaches and edge cases.

Docker Woes

Again, not an RCD problem, but a more general grief... Docker for Mac is almost un-useably slow for me on M1 (and bug ridden). The slow feedback cycle is extra-painful when debugging build issues.

Using a remote DOCKER_HOST doesn't work either since directories cannot be mounted from the host. The only solution I've really found is ssh'ing into another machine. :/

I'm curious if anyone else has run into this, and if they know a better way?

Testing

Testing precompiled gems is a bit of a pain. To solve this, I've created a useful monstrosity to make this a bit easier. It's not ideal, but I'm able to run an entire test suite against a precompiled gem, which is nice.

[Here's the Gist][gist] if you're curious. Maybe we could extract some of this into something proper? It would be nice to have a golden path for testing precompiled gems, because as of now it's a bit of a lone-wolf situation.

Summary

Thanks to RCD, we have a way to reliably build cross-platform gems with Rust. Although there are a couple stumbling blocks for new users, hopefully we can collaborate to make it easier.

PS: Would love to integrate the rb-sys docker stuff as well, if interested.

❤️ Ian

[flavorjones]

@ianks Thank you for the thoughtful writeup! Lots to dig into here, I'll allocate some time this week to digging in.

[larskanis]

@ianks Thank you for this detailed summary of your work! Lots of code actually, so that I feel a bit overwhelmed.

Bundler and rake-compiler-dock (user experience)

rake-compiler-dock doesn't ensure that bundle install is run for each RUBY_CC_VERSION. This means that any Rakefile that uses bundler/setup will fail with Bundler::GemNotFound

It's not the intention to install the gems for each native rvm ruby version. Only the default ruby version should be used for cross compiling and it should build for ruby-2.4 to 3.1. But I didn't use bundler locked environment for cross compilation so far. I guess that the fake.rb generated by rake-compiler might be extended for compatibility with bundler.

OS differences

Depending on which platform you are compiling for, you may have an entirely different OS / package-manager. Some are redhat, some are debian

Sure, I equally don't like the current situation.

CC, CXX, AR, LD and the like

I didn't set them, since the options for cross compilation are often varying, and I preferred a clean environment and leaving these variables to the user. I don't have a strong opinion here.

Mounting cache directories

By default, rake-compiler-dock mounts the ./tmp directory so things are cached nicely.

RCD mounts the current working directory, which is usually the one where the Rakefile resides. So both tmp and target should be mounted usually.

Different compilers

This is probably biased for the Rust world, but it would be amazing Ruby were built using clang + lld for every platform (ideally, the same version of clang for each). This would fix so many headaches and edge cases.

This is interesting. I used gcc a lot for historical reasons and didn't use llvm much. In theory llvm and gcc code should be compatible. Do you have experience with running clang cross compiled code on gcc compiled ruby? On which platforms?

Docker Woes

Again, not an RCD problem, but a more general grief... Docker for Mac is almost un-useably slow for me on M1 (and bug ridden).

I often make use of the multiarch docker feature like here and for instance arm images run quite well on x86_64. So my hope was to keep the RCD images on platform x86_64 only, given that emulation of platforms works so well. Obviously this isn't the case on Mac M1. Maybe the new multiplatform build features of docker can help here, but might be another maintenance burden.

Testing

Testing precompiled gems is a bit of a pain.

OK, I didn't had this feeling. I found the Github Action workflow like here in this repo quite suitable: with a cross build job on Linux, job for testing the resulting gems on native Windows, Linux and MacOS and optionally another job with qemu emulated linux platforms and distributions.

In general I'm fine with integrating a Rust cross build environment. Maybe we should check how we can reduce the docker image size. The Rust enabled images seem to be almost twice the size.

[ianks]

This is interesting. I used gcc a lot for historical reasons and didn't use llvm much. In theory llvm and gcc code should be compatible. Do you have experience with running clang cross compiled code on gcc compiled ruby? On which platforms?

All Rust code is compiled with LLVM, and have been compatible for for at least:

CROSS_PLATFORMS = [
  "arm-linux",
  "aarch64-linux",
  "arm64-darwin",
  "x64-mingw-ucrt",
  "x64-mingw32",
  "x86_64-darwin",
  "x86_64-linux"
]

In this "ideal" scenario though, Ruby would also be compiled with Clang. That would make it much easier to ensure compatible linker flags and such.

It's not the intention to install the gems for each native rvm ruby version.

This one turned out to be an issue with bundler 2 switching versions transparently, causing gems not to be available despite when gems were cached in GitHub actions. I don't fully grok the bug, but suffice to say it is not an RCD issue.

In general I'm fine with integrating a Rust cross build environment. Maybe we should check how we can reduce the docker image size. The Rust enabled images seem to be almost twice the size.

Yes that would be good. I'll see if I can slim the images down.