- CamBench – Cryptographic API Misuse Detection Tool Benchmark Suite
by Michael Schlichtig (Paderborn University), Anna-Katharina Wickert (TU Darmstadt), Stefan Krüger (Independent), Eric Bodden (Paderborn University & Fraunhofer IEM), and Mira Mezini (TU Darmstadt)
has been accepted for publication at the International Conference on Mining Software Repositories 2022 (MSR 2022).
Context: Cryptographic APIs are often misused in real-world applications. To mitigate that, many cryptographic API misuse detection tools have been introduced. However, there exists no established reference benchmark for a fair and comprehensive comparison and evaluation of these tools. While there are benchmarks, they often only address a subset of the domain or were only used to evaluate a subset of existing misuse detection tools.
Objective: To fairly compare cryptographic API misuse detection tools and to drive future development in this domain, we will devise such a benchmark. Openness and transparency in the generation process are key factors to fairly generate and establish the needed benchmark.
Method: We propose an approach where we derive the benchmark generation methodology from the literature which consists of general best practices in benchmarking and domain-specific benchmark generation. A part of this methodology is transparency and openness of the generation process, which is achieved by pre-registering this work. Based on our methodology we design CamBench, a fair “Cryptographic API Misuse Detection Tool Benchmark Suite“. We will implement the first version of Cam-Bench limiting the domain to Java, the JCA, and static analyses. Finally, we will use CamBench to compare current misuse detection tools and compare CamBench to related benchmarks of its domain.
The is the premier conference for data science, machine learning, and artificial intelligence in software engineering. CORE ranking of this conference: A. Mining Software Repositories (MSR) conference
MSR 2022 will be a hybrid event. The virtual MSR 2022 will take place from May 18-20 online. The in-person MSR 2022 will take place from May 23-24 in Pittsburgh, PA, USA.