Skip to content
This repository has been archived by the owner on May 9, 2019. It is now read-only.

Differential testing with NEZHA

Jump to bottom
Theofilos Petsios edited this page May 23, 2017 · 2 revisions

Driver functionality and templates

The cycle for one testing session (one generation) for NEZHA is as follows:

  1. LLVMFuzzerTestOneInput is called. This function replaces the main function and is controlled by the main engine. Its arguments is a buffer (@data) with data to be passed to the different tested applications, as well as the size (@size) of that buffer.

  2. Depending on your specific testing scenario, for each of the tested libraries you will want to process the above data in different ways. For instance, two SSL libraries might have different routines for certificate verification. The easiest way to invoke these libraries would be to construct the appropriate driver (see Writing your own drivers for each application you would like to test, then include diff.h in your main driver similarly to the one of the examples, and invoke the EXCERCISE() macro on each of your applications inside LLVMFuzzerTestOneInput. That should be it!

  3. After each execution of LLVMTestOneInput, NEZHA will check if there is a discrepancy, and log it in the directory specified via the -artifact_prefix flag (remember to include a '/' in the end of your output if you would like the output units to be dumped inside an existing directory). NEZHA logs discrepancies based on the return values of your driver programs. That is, a discrepancy will be logged only if one driver program returns a 0 whereas another driver program returns a value != 0. You can, however, choose to model different types of information on your return driver (e.g., exceptions, error messages) using different return values.

During differential testing, different applications of the same functionality serve as cross-referencing oracles. As such, if two applications are supposed to perform the same task, however exhibit a discrepancy in their behavior (e.g., one returns an error whereas the other doesn't), there is likely that a semantic bug is present.

In NEZHA, the first driver to be called is treated as an oracle of truth, and discrepancies are always examined with respect to deviations from that first driver. For instance, suppose you test OpenSSL, LibreSSL an BoringSSL, and you provide the same input to all three driver functions. Now suppose, the return values for the above libraries are (-2, -1, 0), meaning that OpenSSL and LibreSSL reject the input, whereas BoringSSL accepts it. The order in which the different drivers are called inside LLVMFuzzerTestOneInput is the same order in which the respective return values will appear on the discrepancy logs. For instance, if the input causing the discrepancy which will be logged by NEZHA is will be

diff_-2_-1_0_<hash_of_input> before_mutation_<hash_of_input>

Clone this wiki locally