Fuzz testing aims to automatically generate program inputs to evaluate the robustness of a program to arbitrary inputs. The goal here is to trigger an undesirable behavior, such as a program crash. If a vulnerability is found in this way, developers can find the underlying cause of this behavior and solve the problem at an early stage.

Due to the high effectiveness of the method, many research projects have recently developed approaches to improve the fuzzing process. However, comparing different fuzzers is difficult as current research activities use many different methods and do not always fully respect existing recommendations. David Paaßen, Sebastian Surminski, Michael Rodler, and Prof. Lucas Davi have systematically analyzed the influence of different parameters on the evaluation of fuzzers. For this purpose, they conducted experiments with a runtime of over 280,000 CPU hours. The experiments showed that the use of different test programs has a decisive influence on the results. The paluno researchers will present their evaluation setup and the results in October at the 26th European Symposium on Research in Computer Security (ESORICS) 2021.

To support future research, they are publishing their evaluation framework SENF (Statistical EvaluatioN of Fuzzers) and all related datasets on Githu. A preprint of the ESORICS paper is available on arXiv.

Publication

Paaßen, David; Surminski, Sebastian; Rodler, Michael; Davi, Lucas: My Fuzzer Beats Them All! Developing a Framework for Fair Evaluation and Comparison of Fuzzers. In: Proc. of 26th European Symposium on Research in Computer Security. Springer International Publishing, Darmstadt 2021. arXiv