On Benchmarking the Capability of Symbolic Execution Tools with Logic Bombs

Symbolic execution now becomes an indispensable technique for software testing and program analysis. There are several symbolic execution tools available off-the-shelf, and we need a practical benchmark approach to learn their capabilities. Therefore, this paper introduces a novel approach to benchmark symbolic execution tools in a fine-grained and efficient manner. In particular, our approach evaluates the performance of such tools against the known challenges faced by general symbolic execution techniques, such as floating-point numbers and symbolic memories. To this end, we first survey related papers and systematize the challenges of symbolic execution. We extract 12 distinct challenges from the literature and categorize them into two categories: symbolic-reasoning challenges and path-explosion challenges. Then, we develop a dataset of logic bombs and a framework to benchmark symbolic execution tools automatically. For each challenge, our dataset contains several logic bombs, each of which is guarded by a specific challenging problem. If a symbolic execution tool can find test cases to trigger logic bombs, it indicates that the tool can handle the corresponding problems. We have conducted real-world experiments with three popular symbolic execution tools: KLEE, Angr, and Triton. Experimental results show that our approach can reveal their capabilities and limitations in handling particular issues accurately and efficiently. The benchmark process generally takes only dozens of minutes to evaluate a tool. We release our dataset on GitHub as open source, with an aim to better facilitate the community to conduct future work on benchmarking symbolic execution tools.