Skip to content

Latest commit

 

History

History
231 lines (174 loc) · 8.27 KB

getting_started.md

File metadata and controls

231 lines (174 loc) · 8.27 KB

Getting started with fuzzing in Chromium

This document walks through how to get started adding fuzz tests to Chromium.

It guides you how to use our latest fuzzing technology, called FuzzTest. This replaces earlier technology called libfuzzer. See the section at the end for reasons why you might sometimes still want to use libfuzzer.

[TOC]

What to fuzz

You should fuzz any function which takes input from any untrusted source, such as the internet. If the code parses, decodes, or otherwise manipulates that input, it definitely should be fuzzed!

How to fuzz

  1. Find your existing unit test target. Create a new similar target alongside. (In the future, you'll be able to add them right into your unit test code directly.)
  2. Add a gn target definition a lot like a normal unit test, but with enable_fuzztest = true. See below for details. Create a .cc file.
  3. In the unit tests code, #include "third_party/fuzztest/src/fuzztest/fuzztest.h"
  4. Add a FUZZ_TEST macro, which might be as simple as FUZZ_TEST(MyApiTest, ExistingFunctionWhichTakesUntrustedInput) (though you may wish to structure things differently, see below)
  5. Run the unit tests and ensure they pass.
  6. Land the CL.

That's it!

This fuzzer will be built automatically, using various [sanitizers], and run on our distributed fuzzing infrastructure ClusterFuzz. If it finds bugs, they'll be reported back to you.

More detail in all the following sections.

Creating a new FUZZ_TEST target

*** note Note: Fuzztests don't yet build on Android or Windows component builds . We recommendwrapping these new targets in if (fuzztest_supported) { } blocks in your gn file for now. We'll remove these in future when it works on all platforms.

import("//build/config/sanitizers/sanitizers.gni")
import("//testing/test.gni")

if (fuzztest_supported) {
  test("hypothetical_fuzztests") {
    sources = [ "hypothetical_fuzztests.cc" ]

    enable_fuzztest = true

    deps = [
      ":hypothetical_component",
      "//third_party/fuzztest:fuzztest_gtest_main",
    ]
  }
}

Adding FUZZ_TEST support to a target

*** note Note: Currently, you must create a new unit test target. While the FuzzTest framework supports mixed unit and fuzz tests, we don't yet support this option in Chromium.

In the near future we'll support adding FUZZ_TESTs alongside existing unit tests, even in the same .cc file. You will add an extra enable_fuzztest = true line:

if (is_linux) {
  test("existing_unit_tests") {
    sources = [ "existing_unit_tests.cc" ] # add FUZZ_TESTs here

    enable_fuzztest = true   # add this!

    deps = [
      ":existing_component",
      # Other stuff
    ]
  }
}

This will:

  • add a dependency on the appropriate fuzztest libraries;
  • cause the target to be built on all our fuzzer builders
  • construct metadata so that ClusterFuzz knows how to run the resulting binary.

This relies on something, somewhere, calling base::LaunchUnitTests within your executable to initialize FuzzTest. This should be the case already.

(If you have other code targets, such as source_sets, contributing to your unit test target they may need to explicitly depend upon //third_party/fuzztest too.)

*** note Note: Again, this is not yet supported!

Adding FUZZ_TESTs in the code

First, #include "third_party/fuzztest/src/fuzztest/fuzztest.h".

Then, it's normal to create a function named after the thing you're trying to prove, with assertions to prove it.

For instance,

void MyApiCanSuccessfullyParseAnyString(std::string input) {
    bool success = MyApi(input);
    EXPECT_TRUE(success);
}

Then, declare the FUZZ_TEST macro:

FUZZ_TEST(MyApiTest, MyApiCanSuccessfullyParseAnyString);

Our fuzzing infrastructure will generate all possible strings and prove it works. Obviously, that takes infinite time, so instead our fuzzing infrastructure will carefully craft strings to explore more and more branches within MyApi, mutating the input according to code coverage, so there's a good chance bugs will be found quickly.

Fuzzing should always be alongside traditional unit testing - never rely on it to find all the bugs! It should be a backstop to prevent unexpected security flaws sneaking past your regular testing.

In more complex cases, you'll need to tell FuzzTest about the expected domains of valid input. For example:

void MyApiAlwaysSucceedsOnPositiveIntegers(int i) {
  bool success = MyApi(i);
  EXPECT_TRUE(success);
}
FUZZ_TEST(MyApiTest, MyApiAlwaysSucceedsOnPositiveIntegers)
    .WithDomains(/*i:*/fuzztest::Positive<int>());

See the FuzzTest reference for all your options here.

Running this locally

Simply build and run your unit tests as normal. FUZZ_TESTs are supported only on some platforms. If you're on such a platform, you'll see your fuzz test run for one second:

[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from ScaleFuzz
[ RUN      ] ApiTest.MyApiCanSuccessfullyParseAnyString
[       OK ] ApiTest.MyApiCanSuccessfullyParseAnyString (1000 ms)
[----------] 1 test from ScaleFuzz (1000 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (1000 ms total)
[  PASSED  ] 1 test.

On other platforms, the test will be ignored.

If you want to try actually fuzzing with FuzzTest, add the gn argument enable_fuzztest_fuzz = true. You can then run your unit test with the extra command line argument --fuzz=, optionally specifying a test name. You'll see lots of output as it explores your code:

[*] Corpus size:     1 | Edges covered:     73 | Fuzzing time:        1.60482ms | Total runs:  1.00e+00 | Runs/secs:   623 | Max stack usage:        0
[*] Corpus size:     2 | Edges covered:    103 | Fuzzing time:          1.844ms | Total runs:  2.00e+00 | Runs/secs:  1084 | Max stack usage:        0
[*] Corpus size:     3 | Edges covered:    111 | Fuzzing time:       2.747931ms | Total runs:  3.00e+00 | Runs/secs:  1091 | Max stack usage:        0
[*] Corpus size:     4 | Edges covered:    135 | Fuzzing time:        2.92305ms | Total runs:  4.00e+00 | Runs/secs:  1368 | Max stack usage:        0
[*] Corpus size:     5 | Edges covered:    173 | Fuzzing time:        3.35237ms | Total runs:  5.00e+00 | Runs/secs:  1491 | Max stack usage:        0
[*] Corpus size:     6 | Edges covered:    178 | Fuzzing time:        4.15666ms | Total runs:  6.00e+00 | Runs/secs:  1443 | Max stack usage:        0

("Edges covered") is how many different code blocks have been explored (that is, sections between branches). Over time, you'll see it explore more and more until it runs out of new edges to explore.

Landing the CL

Nothing special is required here!

After a day or two, we should see ClusterFuzz starting to run your new fuzzer, and it should be visible on ClusterFuzz Fuzzer Stats. Look for fuzzers starting with centipede_ and your test target's name.

*** note Note: This is all very new, and ClusterFuzz isn't reliably spotting these new fuzztests yet. We're working on it!

Thanks very much for doing your part in making Chromium more secure!

Unusual cases

There are some situations where FuzzTests may not work. For example:

  • You need to run on platforms not currently supported by FuzzTest
  • You need more structured input
  • You need to mutate the input in a more precise way

In these cases, you may be best off creating a standalone fuzzer using our older fuzzing technology, libfuzzer. There are further options beyond that, e.g. uploading "black box" fuzzers to ClusterFuzz, or even running fuzzers outside of ClusterFuzz which then upload results to ClusterFuzz for triage and diagnosis. To explore any of those options, please discuss with the fuzzing team (email security@chromium.org if you're outside Google).