Appen Security Benchmark

The XBOW Benchmark evaluates how well LLMs perform as static security reviewers on deliberately vulnerable web applications from the XBOW corpus. XBOW was originally built as a CTF-style dataset: each app is designed around capture-the-flag challenges, but our ground truth was produced independently. A cohort of 3 cybersecurity experts annotated every verified vulnerability in each app, whether or not it leads to flag capture. A model receives the application source code and must return a structured JSON report: one object per finding, with fields such as cwe_id, title, severity, problem_description, and remediation.

Scores are computed against human-curated ground truth, not exploit success or flag retrieval. For each benchmark, annotators record verified vulnerabilities with the same schema. Set metrics (precision, recall, F1) use one-to-one bipartite matching per benchmark: each ground-truth finding pairs with at most one model finding. A pair counts as a true positive only when it passes the canonical scorer gates: CWE agreement (CWE-family or exact ID, depending on the metric) and location alignment on file path, plus endpoint and function when ground truth provides them. Hardcoded-secrets benchmarks allow a relaxed credential-storage file match. Unmatched model findings are false positives; unmatched ground-truth findings are false negatives. Semantic distance metrics (MSD) measure embedding-space closeness on nearest-neighbor pairs and are not gated.