FuzzdFlag-v1.0-alpha.2

Full Changelog: v1.0.0-alpha.1...v1.0.0-alpha.2

FuzzdFlag

FuzzdFlags: AFL++ Extension for Flag Mutations of C Compilers

FuzzdFlags is a fuzzing and compiler-testing tool designed to systematically explore compiler behaviours and uncover hidden bugs through efficient mutation of compiler flags and source file combinations.

Docker Image For Tool Reproducability

To quickly get started without manually following the Tool Setup & Installation Steps, you can use the pre-built Docker image that contains the full FuzzdFlags environment:

## Pulling the Docker image
docker pull ayseirmak/fuzzdflags-dev:latest
docker run -it ayseirmak/fuzzdflags-dev:latest /bin/sh

This image includes all required dependencies, compilers, AFL++, and the FuzzdFlags tool pre-installed, ensuring a consistent and reproducible environment.

Abstract

FuzzdFlags extends AFL++ with dynamic mutation of compiler flags, enabling deeper exploration of a compiler’s configuration space. Traditionally, fuzzing focuses on program inputs, but FuzzdFlags treats flag combinations as part of the fuzz input, thus broadening the search to reach untested paths in the compiler. This approach can reveal corner-case bugs triggered only by specific combinations of compiler flags (e.g., optimization levels, target architectures, warnings).

General Architecture

Primary Components

• ClangOptions Wrapper Tool: Converts AFL++ inputs into valid compiler invocations.
• Instrumented Clang Compiler: Generates precise runtime coverage feedback.
• AFL++ Fuzzing Engine: Executes grey-box fuzzing based on runtime coverage data.

Key Features:

• Fuzzing Mode: Fuzzes compiler flags, expanding the search to discover unique compiler paths activated by specific flag combinations. FuzzdFlags reads binary input files that indicate the combination of selected C program and associated compiler flag-set. The tool decodes these binary inputs using a custom function called decodeByteToFlags(). Then, it dynamically generates compilation tasks managed by Clang’s Driver API. By quickly switching among various C programs and compiler flag-set combinations, it effectively explores compiler behaviours.
• Differential Testing Mode: Compares behaviours between different compiler versions by using selected test cases captured during fuzzing.
• Flag Debugging Mode: Isolates the minimal flag combination responsible for compiler crashes or miscompilation.

Tool Architecture:

System Requirements

• Operating System: A 64-bit Linux environment is recommended (the framework has been tested on Ubuntu LTS releases).
  FuzzdFlags is likely to work on other Unix-like systems, but Linux is preferred for AFL++ and compiler toolchain support.
• Hardware: At least 8 cores and 64 GB of RAM are suggested to reproduce the experiments in a reasonable time.
  Fuzzing can be CPU-intensive, and more RAM may be useful to handle the compiler and multiple processes.
• Storage: Ensure you have at least 45 gigabytes of free disk space for the fuzzing output directories. AFL++ will store generated test cases and logs; if running for long durations, this can accumulate. Using an SSD or a RAM disk for the fuzzing output (AFL’s temp dir) can improve performance, but it’s optional.

Tool Setup & Installation Steps

1. Corpus Integration
2. System Installations & Environment Configuration
3. Build & Install AFL++ Fuzzing Engine
4. Build Instrumented Clang & Custom ClangOptions Wrapper Tool
5. Build & Install Default Compilers for Differential Testing Mode of the Tool (GCC, Clang latest)
6. FuzzdFlags Runtime Components Setup

1. Corpus Integration

We constructed an input C program corpus from the LLVM test suite’s single-file . The initial corpus contains 2383 C programs. To retain the coverage of the original set while reducing redundancy, we minimised the corpus with afl-cmin. We ran afl-cmin with AFL++’s default timeout limits and 12 parallel threads, disabling its memory limits due to large SUT. The minimised corpus contains 1811 programs. We used this minimised corpus as or default C program corpus in our tool and experiments.

You can analyze how we generate default C corpus from, See corpus-setup script

Note on Cmin: The corpus minimization step (afl-cmin) uses heuristics that may produce slightly different outputs each run. If you want to reproduce our exact minimized corpus, you can use the already minimized and reindexed corpus. Otherwise, re-running afl-cmin yourself might yield small differences.

• Original LLVM test single SingleSource C program Corpus (2383 programs):

wget https://github.com/ayseirmak/FuzzdFlags-ASE/releases/download/v1.0.0-alpha.1/llvmSS-corpus-org.tar.gz
tar -zxvf llvmSS-corpus-org.tar.gz

• Corpus Minimization with afl-cmin:

AFL_DEBUG=1 AFL_USE_ASAN=0 AFL_PRINT_FILENAMES=1 AFL_DEBUG_CHILD_OUTPUT=1 \ 
afl-cmin -i /users/user42/llvmSS-c-corpus -o /users/user42/llvmSS-c-corpus-after-Cmin \
-m none -t 500 -T 12 -- /users/user42/build-clang17/bin/clang -x c -c -O3 -fpermissive \
-w -Wno-implicit-function-declaration -Wno-implicit-int -Wno-return-type -Wno-builtin-redeclared -Wno-int-conversion  \
-march=native -I/usr/include -I/users/user42/llvmSS-include @@  -o /dev/null > /users/user42/afl-cmin-errors.log 2>&1

• Minimized Corpus (1811 programs):

wget https://github.com/ayseirmak/FuzzdFlags-ASE/releases/download/v1.0.0-alpha.1/llvmSS-minimised-corpus.tar.gz
tar -zxvf llvmSS-minimised-corpus.tar.gz

2. System Installation & Environment Configurations

Below are the core commands we used to set up the environment for FuzzdFlags Tool.
We do not provide a single script; instead, you can copy & paste the relevant commands on your machine.

Important: Some commands (like adding a user user42) are optional or can be adapted if you prefer using your own username. These instructions reflect what we did on a fresh Ubuntu 22.04 CloudLab machine

Initial User Setup and Permissions

sudo useradd -m -d /users/user42 -s /bin/bash user42
sudo passwd user42
sudo usermod -aG sudo user42
sudo usermod -aG kclsystemfuzz-PG user42
sudo chown -R user42:kclsystemfuzz-PG /users/user42
sudo chmod 777 /users/user42
sudo chown -R user42:user42 /users/user42/

System Preparation
Install required dependencies and core tools:

sudo apt-get update && sudo apt-get upgrade -y
sudo apt-get install -y software-properties-common build-essential wget curl git cmake flex bison python3-dev libssl-dev libgtk-3-dev ninja-build gdb gcc-11-plugin-dev valgrind ocaml-nox autoconf libtool python3-pip

System Default Compiler and LLVM Setup

• Add Toolchain PPA & install GCC-11

sudo add-apt-repository ppa:ubuntu-toolchain-r/test -y
sudo apt-get update
sudo apt-get -y install gcc-11 g++-11 cpp-11
sudo rm /usr/bin/cpp /usr/bin/gcc /usr/bin/g++ /usr/bin/gcov /usr/bin/c++ /usr/bin/cc 2>/dev/null
sudo ln -s /usr/bin/cpp-11  /usr/bin/cpp
sudo ln -s /usr/bin/gcc-11  /usr/bin/gcc
sudo ln -s /usr/bin/gcc-11  /usr/bin/cc
sudo ln -s /usr/bin/g++-11  /usr/bin/g++
sudo ln -s /usr/bin/g++-11  /usr/bin/c++
sudo ln -s /usr/bin/gcov-11 /usr/bin/gcov

• Download & install LLVM 14 (for system clang)

sudo bash -c "$(wget -O - https://apt.llvm.org/llvm.sh)"
sudo apt-get install -y clang-14 lldb-14 lld-14
sudo ln -s /usr/bin/llvm-config-14 /usr/bin/llvm-config
echo 'export LLVM_CONFIG=/usr/bin/llvm-config' >> ~/.bashrc
cd /users/user42
su user42

3. Build & Install AFL++ Fuzzing Engine

• Setup AFL++ from source

git clone https://github.com/AFLplusplus/AFLplusplus
cd AFLplusplus
sed -i 's/#define MAP_SIZE_POW2.*/#define MAP_SIZE_POW2 22/' include/config.h # MAP_SIZE_POW2=16 ~> MAP_SIZE_POW2=22 (4 MiB)
make distrib
sudo make install

4. Build Instrumented Clang & Custom ClangOptions Wrapper Tool

• Build Instrumented LLVM-Clang-17

cd ~
git clone https://github.com/llvm/llvm-project.git
cd llvm-project
git checkout release/17.x
cd /users/user42/llvm-project/clang/tools
mkdir -p clang-options && cd clang-options
wget https://raw.githubusercontent.com/ayseirmak/FuzzdFlags-ASE/refs/heads/main/clang-options/ClangOptions.cpp
wget https://raw.githubusercontent.com/ayseirmak/FuzzdFlags-ASE/refs/heads/main/clang-options/CMakeLists.txt
sed -i '/add_clang_subdirectory(clang-scan-deps)/a add_clang_subdirectory(clang-options)' ../CMakeLists.txt
export AFL_MAP_SIZE=4194304
mkdir ~/build
cd ~/build

# Configure LLVM build to use AFL's clang-fast
LD=/usr/local/bin/afl-clang-fast++ cmake -G Ninja -Wall ../llvm-project/llvm/ \
  -DLLVM_ENABLE_PROJECTS="clang" \
  -DLLVM_USE_SANITIZER=OFF \
  -DCMAKE_BUILD_TYPE="Release" \
  -DCMAKE_C_COMPILER=/usr/local/bin/afl-clang-fast \
  -DCMAKE_CXX_COMPILER=/usr/local/bin/afl-clang-fast++ \
  -DBUILD_SHARED_LIBS=OFF \
  -DLLVM_TARGETS_TO_BUILD="X86" \
  -DCMAKE_C_FLAGS="-pthread -L/usr/lib/x86_64-linux-gnu" \
  -DCMAKE_CXX_FLAGS="-pthread -L/usr/lib/x86_64-linux-gnu" \
  -DCMAKE_EXE_LINKER_FLAGS="-L/usr/lib/x86_64-linux-gnu" \
  -DLLVM_BUILD_DOCS="OFF"

# Build only clang 
ninja clang

• Build custom ClangOptions wrapper tool

mkdir ~/build-clang-options
cd ~/build-clang-options
cmake -G Ninja ../llvm-project/llvm \
  -DLLVM_ENABLE_PROJECTS="clang" \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_C_COMPILER=/usr/bin/clang-14 \
  -DCMAKE_CXX_COMPILER=/usr/bin/clang++-14 \
  -DLLVM_USE_SANITIZER=OFF \
  -DBUILD_SHARED_LIBS=OFF \
  -DLLVM_TARGETS_TO_BUILD="X86" \
 ...