A Benchmark for Evaluating Autonomous Vehicles in Safety-critical Scenarios

The goal of SafeBench is to systematically evaluate the safety and security of autonomous driving (AD) algorithms based on diverse testing scenarios and comprehensive evaluation metrics. Safebench is based on Carla, a high-fidelity and open-sourced AD Simulator. There are three main features that distinguish Safebench from other scenario-based benchmarks:

  • Parallel Running. Safebench supports the parallel running of multiple scenarios in one map, which dramatically increases the efficiency of collecting training data and evaluation.
  • Comprehensive Scenario. Safebench integrates lots of scenario-generation algorithms, including data-driven generation, rule-based design, and adversarial examples.
  • Usage Coverage. Safebench supports three running modes, including training ego policy, training scenario policy, and evaluating ego in scenarios.

Overview Video

Safebench provides scenarios for both perception module and control module of AD system. This overview video illustrates the scenarios in Safebench for both modules.


  • We are hosting a challenge in CVPR 2023 using Safebench platform. Please check our website for more details.
  • The new version of Safebench is avaliable now.
  • The paper introducing Safebench is published on NeurIPS2022.

Safebench Team
Wenhao Ding
Chejian Xu
Haohong Lin
Shuai Wang
Jiawei Zhang
Shiqi Liu
Zuxin Liu
Mansur Arief
Ding Zhao
Bo Li