Commands ros2, rosdep very slow in arm64v8/ros:jazzy

[johannesn]

Hi,

when running the docker image on two of my amd64 machines using qemu, several ros commands are very slow to start.

xxx@xxx:~$ docker run --rm -it --platform linux/arm64 arm64v8/ros:jazzy
xxx@xxx:/# time ros2
...
real	0m35.459s
user	0m35.237s
sys	0m0.213s
xxx@xxx:/# time rosdep --version
0.25.1

real	0m14.426s
user	0m14.308s
sys	0m0.184s

compared to arm64v8/ros:humble:

xxx@xxx:~$ docker run --rm -it --platform linux/arm64 arm64v8/ros:humble
xxx@xxx:/# time ros2 
...

real	0m2.175s
user	0m2.075s
sys	0m0.102s
xxx@xxx:/# time rosdep --version
0.25.1

real	0m1.665s
user	0m1.570s
sys	0m0.108s

Image	ros2 time	rosdep time
arm64v8/ros:humble	2.175s	1.665s
arm64v8/ros:jazzy	35.459s	14.426s

[ruffsl]

Interesting! What is the trend when extrapolating further using ROS Kilted and Rolling?

I've always found python to be particularly slow using qemu, but the difference between distros is quite peculiar. Does this difference persist using a native arm64v8 docker host?

[johannesn]

This is emulated using an AMD64 machine.

Kilted:

xxx@xxx ~ % docker run --rm -it --platform linux/arm64 arm64v8/ros:kilted
xxx@xxx:/# time ros2
...

real	0m30.295s
user	0m30.046s
sys	0m0.094s
xxx@xxx:/# time rosdep --version
0.25.1

real	0m11.465s
user	0m11.440s
sys	0m0.069s

Rolling:

xxx@xxx ~ % docker run --rm -it --platform linux/arm64 arm64v8/ros:rolling
xxx@xxx:/# time ros2
...
real	0m30.339s
user	0m30.104s
sys	0m0.081s
xxx@xxx:/# time rosdep --version
0.25.1

real	0m11.445s
user	0m11.413s
sys	0m0.061s

Quick check on my MacBook M2 yields this result:

Humble:

xxx@xxx ~ % docker run --rm -it --platform linux/arm64 arm64v8/ros:humble
xxx@xxx:/# time ros2 
...

real	0m0.223s
user	0m0.167s
sys	0m0.044s
xxx@xxx:/# time rosdep --version
0.25.1

real	0m0.119s
user	0m0.094s
sys	0m0.025s

Jazzy:

xxx@xxx ~ % docker run --rm -it --platform linux/arm64 arm64v8/ros:jazzy 
xxx@xxx:/# time ros2 
...

real	0m0.287s
user	0m0.223s
sys	0m0.054s
xxx@xxx:/# time rosdep --version
0.25.1

real	0m0.142s
user	0m0.120s
sys	0m0.024s

Kilted:

xxx@xxx ~ % docker run --rm -it --platform linux/arm64 arm64v8/ros:kilted
xxx@xxx:/# time ros2
...
real	0m0.328s
user	0m0.236s
sys	0m0.078s
xxx@xxx:/# time rosdep --version
0.25.1

real	0m0.138s
user	0m0.116s
sys	0m0.024s

Rolling:

xxx@xxx ~ % docker run --rm -it --platform linux/arm64 arm64v8/ros:rolling
xxx@xxx:/# time ros2
...

real	0m0.323s
user	0m0.246s
sys	0m0.064s
xxx@xxx:/# time rosdep --version
0.25.1

real	0m0.145s
user	0m0.122s
sys	0m0.024s

Image	ros2 time (AMD64)	rosdep time (AMD64)	ros2 time (M2)	rosdep time (M2)
arm64v8/ros:humble	2.175s	1.665s	0.119s	0.223s
arm64v8/ros:jazzy	35.459s	14.426s	0.287s	0.142s
arm64v8/ros:kilted	30.295s	11.465s	0.328s	0.138s
arm64v8/ros:rolling	30.339s	11.445s	0.323s	0.145s

As you can see on my M2 it is much faster than on my AMD64, so the problem ist most probably related to emulation.

[ruffsl]

Wow, looks to be reproducible on my x86_64 CPU using Intel as well.

Re-running the command twice after shared libraries have been cached doesn't seem to improve the timings either:

Humble

Details

$ docker run --rm -it --platform linux/arm64 arm64v8/ros:humble
Unable to find image 'arm64v8/ros:humble' locally
humble: Pulling from arm64v8/ros
0e25612b6db2: Pull complete 
3ee5308cdd15: Pull complete 
73381c2c31e3: Pull complete 
1d5d02a65bd7: Pull complete 
fe7420133192: Pull complete 
ced1014ed7cb: Pull complete 
f371b330eabe: Pull complete 
42eaf8eb4130: Pull complete 
014dcb0d353b: Pull complete 
3fa21615c56e: Pull complete 
Digest: sha256:d2dff59f2376de78a86bc539927aae86f64e3c008914297b27c4100c9281a947
Status: Downloaded newer image for arm64v8/ros:humble
root@f24321c8e132:/# time ros2
usage: ros2 [-h] [--use-python-default-buffering] Call `ros2 <command> -h` for more detailed usage. ...

ros2 is an extensible command-line tool for ROS 2.

options:
  -h, --help            show this help message and exit
  --use-python-default-buffering
                        Do not force line buffering in stdout and instead use the python default buffering, which might be affected by PYTHONUNBUFFERED/-u
                        and depends on whatever stdout is interactive or not

Commands:
  action     Various action related sub-commands
  bag        Various rosbag related sub-commands
  component  Various component related sub-commands
  daemon     Various daemon related sub-commands
  doctor     Check ROS setup and other potential issues
  interface  Show information about ROS interfaces
  launch     Run a launch file
  lifecycle  Various lifecycle related sub-commands
  multicast  Various multicast related sub-commands
  node       Various node related sub-commands
  param      Various param related sub-commands
  pkg        Various package related sub-commands
  run        Run a package specific executable
  security   Various security related sub-commands
  service    Various service related sub-commands
  topic      Various topic related sub-commands
  wtf        Use `wtf` as alias to `doctor`

  Call `ros2 <command> -h` for more detailed usage.

real    0m1.555s
user    0m1.356s
sys     0m0.050s
root@f24321c8e132:/# time ros2
usage: ros2 [-h] [--use-python-default-buffering] Call `ros2 <command> -h` for more detailed usage. ...

ros2 is an extensible command-line tool for ROS 2.

options:
  -h, --help            show this help message and exit
  --use-python-default-buffering
                        Do not force line buffering in stdout and instead use the python default buffering, which might be affected by PYTHONUNBUFFERED/-u
                        and depends on whatever stdout is interactive or not

Commands:
  action     Various action related sub-commands
  bag        Various rosbag related sub-commands
  component  Various component related sub-commands
  daemon     Various daemon related sub-commands
  doctor     Check ROS setup and other potential issues
  interface  Show information about ROS interfaces
  launch     Run a launch file
  lifecycle  Various lifecycle related sub-commands
  multicast  Various multicast related sub-commands
  node       Various node related sub-commands
  param      Various param related sub-commands
  pkg        Various package related sub-commands
  run        Run a package specific executable
  security   Various security related sub-commands
  service    Various service related sub-commands
  topic      Various topic related sub-commands
  wtf        Use `wtf` as alias to `doctor`

  Call `ros2 <command> -h` for more detailed usage.

real    0m1.203s
user    0m1.161s
sys     0m0.044s

Kilted

Details

$ docker run --rm -it --platform linux/arm64 arm64v8/ros:kilted
root@d4e58cf38cc2:/# time ros2
usage: ros2 [-h] [--use-python-default-buffering] Call `ros2 <command> -h` for more detailed usage. ...

ros2 is an extensible command-line tool for ROS 2.

options:
  -h, --help            show this help message and exit
  --use-python-default-buffering
                        Do not force line buffering in stdout and instead use the python default buffering, which might be affected by PYTHONUNBUFFERED/-u
                        and depends on whatever stdout is interactive or not

Commands:
  action     Various action related sub-commands
  bag        Various rosbag related sub-commands
  component  Various component related sub-commands
  daemon     Various daemon related sub-commands
  doctor     Check ROS setup and other potential issues
  interface  Show information about ROS interfaces
  launch     Run a launch file
  lifecycle  Various lifecycle related sub-commands
  multicast  Various multicast related sub-commands
  node       Various node related sub-commands
  param      Various param related sub-commands
  pkg        Various package related sub-commands
  run        Run a package specific executable
  security   Various security related sub-commands
  service    Various service related sub-commands
  topic      Various topic related sub-commands
  wtf        Use `wtf` as alias to `doctor`

  Call `ros2 <command> -h` for more detailed usage.

real    0m26.626s
user    0m26.345s
sys     0m0.075s
root@d4e58cf38cc2:/# time ros2
usage: ros2 [-h] [--use-python-default-buffering] Call `ros2 <command> -h` for more detailed usage. ...

ros2 is an extensible command-line tool for ROS 2.

options:
  -h, --help            show this help message and exit
  --use-python-default-buffering
                        Do not force line buffering in stdout and instead use the python default buffering, which might be affected by PYTHONUNBUFFERED/-u
                        and depends on whatever stdout is interactive or not

Commands:
  action     Various action related sub-commands
  bag        Various rosbag related sub-commands
  component  Various component related sub-commands
  daemon     Various daemon related sub-commands
  doctor     Check ROS setup and other potential issues
  interface  Show information about ROS interfaces
  launch     Run a launch file
  lifecycle  Various lifecycle related sub-commands
  multicast  Various multicast related sub-commands
  node       Various node related sub-commands
  param      Various param related sub-commands
  pkg        Various package related sub-commands
  run        Run a package specific executable
  security   Various security related sub-commands
  service    Various service related sub-commands
  topic      Various topic related sub-commands
  wtf        Use `wtf` as alias to `doctor`

  Call `ros2 <command> -h` for more detailed usage.

real    0m16.088s
user    0m16.011s
sys     0m0.066s

---

I wonder if the underlying correlation here is the base image though, I.e: Ubuntu Jammy 22.04 vs Noble 24.04:

$ docker run --rm -it --platform linux/arm64 arm64v8/ros:humble
root@499e24e7e9e7:/# time python3 -c 'print("Hello, world!")'
Hello, world!

real    0m0.164s
user    0m0.153s
sys     0m0.013s
root@499e24e7e9e7:/# time python3 -c 'print("Hello, world!")'
Hello, world!

real    0m0.171s
user    0m0.159s
sys     0m0.013s
root@499e24e7e9e7:/# time python3 -c 'print("Hello, world!")'
Hello, world!

real    0m0.174s
user    0m0.166s
sys     0m0.009s


root@499e24e7e9e7:/# time python3 -c 'import rclpy'

real    0m0.497s
user    0m0.397s
sys     0m0.021s
root@499e24e7e9e7:/# time python3 -c 'import rclpy'

real    0m0.315s
user    0m0.301s
sys     0m0.015s
root@499e24e7e9e7:/# time python3 -c 'import rclpy'

real    0m0.319s
user    0m0.305s
sys     0m0.015s


root@499e24e7e9e7:/# time python3 -c 'import numpy'

real    0m0.549s
user    0m0.528s
sys     0m0.022s
root@499e24e7e9e7:/# time python3 -c 'import numpy'

real    0m0.562s
user    0m0.535s
sys     0m0.028s
root@499e24e7e9e7:/# time python3 -c 'import numpy'

real    0m0.557s
user    0m0.537s
sys     0m0.018s

$ docker run --rm -it --platform linux/arm64 arm64v8/ros:kilted
root@a3c3cbe94fa7:/# time python3 -c 'print("Hello, world!")'
Hello, world!

real    0m0.601s
user    0m0.588s
sys     0m0.012s
root@a3c3cbe94fa7:/# time python3 -c 'print("Hello, world!")'
Hello, world!

real    0m0.607s
user    0m0.591s
sys     0m0.017s
root@a3c3cbe94fa7:/# time python3 -c 'print("Hello, world!")'
Hello, world!

real    0m0.616s
user    0m0.605s
sys     0m0.012s


root@a3c3cbe94fa7:/# time python3 -c 'import rclpy'

real    0m7.360s
user    0m7.212s
sys     0m0.035s
root@a3c3cbe94fa7:/# time python3 -c 'import rclpy'

real    0m4.068s
user    0m4.016s
sys     0m0.029s
root@a3c3cbe94fa7:/# time python3 -c 'import rclpy'

real    0m4.073s
user    0m4.036s
sys     0m0.018s


root@a3c3cbe94fa7:/# time python3 -c 'import numpy'

real    0m4.760s
user    0m4.716s
sys     0m0.042s
root@a3c3cbe94fa7:/# time python3 -c 'import numpy'

real    0m4.752s
user    0m4.730s
sys     0m0.023s
root@a3c3cbe94fa7:/# time python3 -c 'import numpy'

real    0m4.798s
user    0m4.776s
sys     0m0.023s

It looks like a trivial invocation of python is about 3.5x slower, while importing rclpy is about 15x slower. Perhaps cross triaging the ros2 CLI using Jazzy on Jammy 22.04 or Humble on Noble 24.04 could disambiguate the regression. Although running any ROS2 distro on a non Tier 1 supported platform is a little more involved.

However given numpy, this seems more generic to python and ubuntu base images, not necessarily specific to ROS.

@tfoote , @cottsay or @nuclearsandwich , any ideas here?

My relevant system info for reference:

Details

ruffsl@box:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Neon
Description:    KDE neon 6.3
Release:        24.04
Codename:       noble
ruffsl@box:~$ uname -r
6.11.0-26-generic
ruffsl@box:~$ docker --version
Docker version 28.2.2, build e6534b4
ruffsl@box:~$ docker version
Client: Docker Engine - Community
 Version:           28.2.2
 API version:       1.50
 Go version:        go1.24.3
 Git commit:        e6534b4
 Built:             Fri May 30 12:07:27 2025
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          28.2.2
  API version:      1.50 (minimum version 1.24)
  Go version:       go1.24.3
  Git commit:       45873be
  Built:            Fri May 30 12:07:27 2025
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.27
  GitCommit:        05044ec0a9a75232cad458027ca83437aae3f4da
 runc:
  Version:          1.2.5
  GitCommit:        v1.2.5-0-g59923ef
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
ruffsl@box:~$ sudo apt info qemu-user-static
Package: qemu-user-static
Version: 1:8.2.2+ds-0ubuntu1.7
Built-Using: glib2.0 (= 2.80.0-6ubuntu3.2), glibc (= 2.39-0ubuntu8.4), zlib (= 1:1.3.dfsg-3.1ubuntu2.1)
Priority: optional
Section: universe/otherosfs
Source: qemu
Origin: Ubuntu
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Original-Maintainer: Debian QEMU Team <pkg-qemu-devel@lists.alioth.debian.org>
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Installed-Size: 136 MB
Provides: qemu-user-binfmt
Recommends: systemd
Conflicts: qemu-user-binfmt
Homepage: http://www.qemu.org/
Download-Size: 14.7 MB
APT-Manual-Installed: yes
APT-Sources: http://archive.ubuntu.com/ubuntu noble-updates/universe amd64 Packages
Description: QEMU user mode emulation binaries (static version)
ruffsl@box:~$ lscpu
Architecture:             x86_64
  CPU op-mode(s):         32-bit, 64-bit
  Address sizes:          46 bits physical, 48 bits virtual
  Byte Order:             Little Endian
CPU(s):                   32
  On-line CPU(s) list:    0-31
Vendor ID:                GenuineIntel
  Model name:             Intel(R) Core(TM) i9-14900K
...
Virtualization features:  
  Virtualization:         VT-x
...

[tfoote]

I can reproduce as well. And with the numpy import slowed down significantly too, I suspect that this is because importing takes a lot of system calls, and thus a slowdown/performance regression in qemu would cause it.

For example I used strace for the import rclpy and import numpy and they both have ~2000 lines of strace to complete.

https://gist.github.com/tfoote/8b28c0f86dc872681f0c7c397edcebc5

Note that this was in an amd64 container as strace isn't implemented in the qemu containers.

root@99bffefb4286:/# time strace python3 -c 'import numpy'
strace: test_ptrace_get_syscall_info: PTRACE_TRACEME: Function not implemented
strace: ptrace(PTRACE_TRACEME, ...): Function not implemented
strace: PTRACE_SETOPTIONS: Function not implemented
strace: detach: waitpid(889): No child processes
strace: Process 889 detached

There are a lot of qemu changes between jammy and noble: https://discourse.ubuntu.com/t/ubuntu-24-04-lts-noble-numbat-release-notes/39890

[ruffsl]

There are a lot of qemu changes between jammy and noble:

Although, not sure that's a factor here, as modern Docker uses qemu from the host OS.

I suspect that this is because importing takes a lot of system calls

I wonder it's more something to do with changes to the newer cpython interpreter?