Fail to install on Ubuntu 24.04

[hutzelmann]

Atheris apparently cannot be installed on a Ubuntu 24.04.

Steps to Reproduce

1. Get a fresh copy of Ubuntu 24.04
2. Install the main dependencies

sudo apt install -y python3-pip python3-venv llvm clang

3. (optional) get a virtual environment

python3 -m venv .venv && source .venv/bin/activate

4. Try to install atheris

pip3 install atheris

results in the following error

Collecting atheris
  Downloading atheris-2.3.0.tar.gz (304 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 304.1/304.1 kB 6.6 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: atheris
  Building wheel for atheris (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for atheris (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [41 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-cpython-312
      copying atheris_no_libfuzzer.py -> build/lib.linux-x86_64-cpython-312
      creating build/lib.linux-x86_64-cpython-312/atheris
      copying src/coverage_test.py -> build/lib.linux-x86_64-cpython-312/atheris
      copying src/version_dependent.py -> build/lib.linux-x86_64-cpython-312/atheris
      copying src/custom_crossover_fuzz_test.py -> build/lib.linux-x86_64-cpython-312/atheris
      copying src/utils.py -> build/lib.linux-x86_64-cpython-312/atheris
      copying src/function_hooks.py -> build/lib.linux-x86_64-cpython-312/atheris
      copying src/pyinstaller_coverage_test.py -> build/lib.linux-x86_64-cpython-312/atheris
      copying src/instrument_bytecode.py -> build/lib.linux-x86_64-cpython-312/atheris
      copying src/__init__.py -> build/lib.linux-x86_64-cpython-312/atheris
      copying src/custom_mutator_fuzz_test.py -> build/lib.linux-x86_64-cpython-312/atheris
      copying src/fuzz_test.py -> build/lib.linux-x86_64-cpython-312/atheris
      copying src/coverage_test_helper.py -> build/lib.linux-x86_64-cpython-312/atheris
      copying src/fuzzed_data_provider_test.py -> build/lib.linux-x86_64-cpython-312/atheris
      copying src/regex_match_generation_test.py -> build/lib.linux-x86_64-cpython-312/atheris
      copying src/import_hook.py -> build/lib.linux-x86_64-cpython-312/atheris
      copying src/fuzz_test_lib.py -> build/lib.linux-x86_64-cpython-312/atheris
      copying src/custom_mutator_and_crossover_fuzz_test.py -> build/lib.linux-x86_64-cpython-312/atheris
      copying src/hook-atheris.py -> build/lib.linux-x86_64-cpython-312/atheris
      copying src/coverage_g3test.py -> build/lib.linux-x86_64-cpython-312/atheris
      running build_ext
      Your libFuzzer is up-to-date.
      creating tmp
      x86_64-linux-gnu-gcc -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O2 -Wall -fPIC -I/home/user/.venv/include -I/usr/include/python3.12 -c /tmp/tmpj0ufddpk.cpp -o tmp/tmpj0ufddpk.o -std=c++14
      building 'atheris.native' extension
      creating build/temp.linux-x86_64-cpython-312
      creating build/temp.linux-x86_64-cpython-312/src
      creating build/temp.linux-x86_64-cpython-312/src/native
      x86_64-linux-gnu-gcc -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O2 -Wall -fPIC -DVERSION_INFO='2.3.0' -DATHERIS_MODULE_NAME=native -I/tmp/pip-build-env-_fvgqyh7/normal/lib/python3.12/site-packages/pybind11/include -I/home/user/.venv/include -I/usr/include/python3.12 -c src/native/atheris.cc -o build/temp.linux-x86_64-cpython-312/src/native/atheris.o -DPYBIND11_DETAILED_ERROR_MESSAGES=1 -Wno-attributes -Wno-address -Wno-deprecated-declarations -std=c++14
      x86_64-linux-gnu-gcc -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O2 -Wall -fPIC -DVERSION_INFO='2.3.0' -DATHERIS_MODULE_NAME=native -I/tmp/pip-build-env-_fvgqyh7/normal/lib/python3.12/site-packages/pybind11/include -I/home/user/.venv/include -I/usr/include/python3.12 -c src/native/codetable_gen.cc -o build/temp.linux-x86_64-cpython-312/src/native/codetable_gen.o -DPYBIND11_DETAILED_ERROR_MESSAGES=1 -Wno-attributes -Wno-address -Wno-deprecated-declarations -std=c++14
      src/native/codetable_gen.cc: In lambda function:
      src/native/codetable_gen.cc:46:17: error: ‘PRECALL’ was not declared in this scope; did you mean ‘CALL’?
         46 |   opcode_caches[PRECALL] = 1;
            |                 ^~~~~~~
            |                 CALL
      error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for atheris
Failed to build atheris
ERROR: Could not build wheels for atheris, which is required to install pyproject.toml-based projects

pip3 install --no-binary atheris atheris and trying to build with pip from the sources results in the same error. On Ubuntu 22.04 the above works without any problems.

Additional Information

python --version
Python 3.12.3

clang --version
Ubuntu clang version 18.1.3 (1)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

x86_64-linux-gnu-gcc --version
x86_64-linux-gnu-gcc (Ubuntu 13.2.0-23ubuntu4) 13.2.0

It would be great if you can have a closer look into this problem. Please let me know if you need further help in reproducing this bug.

[tecknicaltom]

It looks like this is a bigger issue affecting building Atheris against python 3.12, not Ubuntu specific. The PRECALL instruction was removed before 3.12 which the Atheris code references.

See python/cpython#92925 and https://docs.python.org/3.12/whatsnew/3.12.html#cpython-bytecode-changes

[mrquokka]

Some news? Cannot install anymore

[Hnasar]

Just wanted to note that there's a draft PR to fix this, but it seems to have stalled? #60

[SweetVishnya]

@jvoisin, any news on supporting Atheris compilation on Ubuntu 24.04?

[AidenRHall]

Julien is no longer actively maintaining Atheris, its just me now. Unfortunately it's been hard to find time to work on it given the current environment, working on Atheris is one small part of my overall job and we've been focusing only on core priorities in the immediate past. So I would like to acknowledge that Atheris isn't in the best state that it could be in, and that's on me.

I think this issue is really about adding support for python 3.12, which I am actively working on internally - I was hoping to finish before the holidays but that looks unlikely at this point. Long term I would like to focus more on the OSS use case and move all development to Github instead of using Google's internal tools, hopefully that will happen in 2025. The VM keeps changing and we need to explore some options to reduce the complexity of adding support for new Python versions.

The other part of why there hasn't been more activity is that there hasn't been a clear vision for Atheris to move towards, I am also hoping to change that in 2025. Stay tuned, and thanks for your patience and understanding in the meantime.

[tecknicaltom]

Any news on this front @AidenRHall? I've had to stop using Atheris after my codebase migrated to Python 3.12 only

[AidenRHall]

Yes I actually have this patch mostly working internally. However there is still at least one severe bug where we parse the bytecode wrong if a for loop contains too many instructions. I'm hoping to have all of this wrapped up by EOM, but we'll see how many more bugs I find during the process haha. After that I'm gonna focus on adding support fro 3.13 immediately to avoid getting ourselves into this situation again, and migrating the backend to LibAFL.

[ligurio]

@AidenRHall Hi, Aiden, any news regarding 3.12 support?

[AidenRHall]

I have not been able to finish this :/, although I did fix the for loop bug that just exposed the next one - I am not sure how many more I will find. Unfortunately there is no support that I'm aware of for debugging Python's bytecode because pdb goes line by line but one line corresponds to many instructions. This is what is making the process so slow, along with this being only a small part of my overall job (hoping to change that in the future).

I am working hard on this, I appreciate your collective patience. We will get there.