SefeBench-leaderboard
A Benchmarking Platform for Safety Evaluation of Autonomous Vehicles

In SafeBench leaderboard, we present the diagnostic report for each AD algorithm from 3 different levels: safety level, functionality level, and etiquette level. Notably, we offer the evaluation results in different scenarios to enable better understanding of AD algorithm performance in different traffic situations. Please find more details in our paper here.

Overall Leaderboard
State Space Algo. Safety Level Functionality Level Etiquette Level OS ↑
CR ↓ RR ↓ SS ↓ OR ↓ RF ↑ Comp↑ TS ↓ ACC↓ YV ↓ LI ↓
4D DDPG 0.780 0.089 0.087 12.619 0.504 0.466 20.860 2.488 0.405 5.764 0.489
SAC 0.829 0.216 0.146 3.115 0.882 0.648 16.827 1.830 0.704 2.580 0.499
TD3 0.783 0.231 0.141 2.535 0.903 0.670 17.644 2.680 1.493 2.545 0.516
PPO 0.603 0.287 0.150 0.099 0.901 0.751 18.021 2.461 1.506 3.528 0.606
Dir SAC 0.676 0.209 0.152 5.658 0.740 0.705 23.386 1.892 0.640 4.565 0.558
TD3 0.655 0.270 0.144 0.885 0.887 0.718 18.899 2.417 1.187 4.694 0.579
PPO 0.739 0.045 0.077 17.607 0.685 0.534 21.336 2.911 0.893 4.875 0.513
BEV SAC 0.782 0.229 0.141 6.057 0.883 0.674 17.863 2.952 1.566 4.448 0.506
PPO 0.416 0.262 0.151 2.180 0.782 0.756 30.651 2.592 1.290 7.319 0.679
Cam SAC 0.829 0.261 0.149 0.014 0.926 0.637 15.480 4.354 1.885 6.139 0.485
PPO 0.600 0.050 0.127 15.101 0.708 0.599 31.914 2.631 0.827 6.327 0.576
Available Leaderboard
1
1
Leaderboard of Secnario: Straight Obstacle
State Space Algo. Safety Level Functionality Level Etiquette Level OS ↑
CR ↓ RR ↓ SS ↓ OR ↓ RF ↑ Comp↑ TS ↓ ACC↓ YV ↓ LI ↓
4D DDPG 0.649 0.083 0.013 27.908 0.636 0.582 20.387 2.473 0.276 5.263 0.545
SAC 0.820 0.101 0.000 0.225 0.824 0.547 15.423 1.216 0.348 0.899 0.533
TD3 0.930 0.088 0.000 0.042 0.926 0.538 15.519 2.186 0.971 0.610 0.479
PPO 0.285 0.465 0.083 0.003 0.916 0.860 19.615 2.031 1.346 2.136 0.761
Dir SAC 0.589 0.302 0.064 7.204 0.764 0.686 16.771 1.388 0.323 3.728 0.608
TD3 0.360 0.399 0.070 0.027 0.891 0.819 19.732 2.028 0.873 5.781 0.728
PPO 0.715 0.035 0.000 32.834 0.640 0.605 19.355 2.945 0.842 5.627 0.506
BEV SAC 0.873 0.114 0.000 0.006 0.917 0.564 15.028 2.612 0.979 4.399 0.501
PPO 0.114 0.535 0.096 0.518 0.774 0.941 32.754 2.644 1.361 5.083 0.818
Cam SAC 0.556 0.292 0.060 0.007 0.929 0.735 16.454 3.981 1.303 7.069 0.634
PPO 0.640 0.035 0.013 28.394 0.674 0.620 33.832 2.718 0.700 6.127 0.542
1
1
Leaderboard of Secnario: Turning Obstacle
State Space Algo. Safety Level Functionality Level Etiquette Level OS ↑
CR ↓ RR ↓ SS ↓ OR ↓ RF ↑ Comp↑ TS ↓ ACC↓ YV ↓ LI ↓
4D DDPG 0.613 0.313 0.025 18.160 0.000 0.543 18.175 2.229 0.225 6.577 0.526
SAC 0.811 0.555 0.213 0.035 0.766 0.778 15.142 2.450 0.354 4.628 0.474
TD3 0.579 0.598 0.159 0.369 0.877 0.873 15.262 3.373 0.564 4.848 0.596
PPO 0.530 0.671 0.177 0.057 0.876 0.897 17.855 2.815 0.609 5.341 0.611
Dir SAC 0.579 0.427 0.159 5.126 0.598 0.839 16.417 2.059 0.239 6.171 0.591
TD3 0.677 0.628 0.171 0.324 0.868 0.860 16.866 2.723 0.397 5.640 0.543
PPO 0.671 0.152 0.061 26.030 0.553 0.720 18.689 2.950 0.526 7.543 0.526
BEV SAC 0.628 0.585 0.159 0.252 0.851 0.867 14.957 3.523 0.689 8.628 0.567
PPO 0.457 0.591 0.159 3.281 0.756 0.915 30.213 2.727 0.620 8.951 0.632
Cam SAC 0.609 0.623 0.179 0.021 0.904 0.870 15.624 4.545 0.682 9.616 0.570
PPO 0.713 0.122 0.116 22.859 0.621 0.801 31.323 2.646 0.443 10.762 0.503
1
1
Leaderboard of Secnario: Lane Changing
State Space Algo. Safety Level Functionality Level Etiquette Level OS ↑
CR ↓ RR ↓ SS ↓ OR ↓ RF ↑ Comp↑ TS ↓ ACC↓ YV ↓ LI ↓
4D DDPG 0.874 0.028 0.000 19.304 0.627 0.591 29.347 2.339 0.962 11.550 0.440
SAC 0.692 0.015 0.000 10.105 0.866 0.648 20.510 1.750 0.942 7.478 0.577
TD3 0.871 0.018 0.000 7.896 0.861 0.579 25.722 2.636 2.031 5.226 0.477
PPO 0.992 0.021 0.000 0.215 0.874 0.547 29.967 2.338 2.195 6.797 0.426
Dir SAC 0.427 0.063 0.000 16.257 0.601 0.792 34.497 1.428 1.154 10.496 0.670
TD3 0.843 0.021 0.000 3.051 0.877 0.567 27.022 2.381 1.912 6.918 0.499
PPO 0.468 0.008 0.000 43.038 0.569 0.787 27.886 2.567 1.915 7.951 0.601
BEV SAC 0.458 0.057 0.000 24.713 0.789 0.759 22.034 2.631 2.149 11.249 0.647
PPO 0.653 0.051 0.000 8.701 0.723 0.683 49.359 2.496 2.077 9.530 0.555
Cam SAC 1.000 0.015 0.000 0.000 0.930 0.495 0.000 4.333 2.853 9.969 0.436
PPO 0.848 0.021 0.000 33.008 0.614 0.647 48.953 2.661 1.744 10.807 0.407
1
1
Leaderboard of Secnario: Vehicle Passing
State Space Algo. Safety Level Functionality Level Etiquette Level OS ↑
CR ↓ RR ↓ SS ↓ OR ↓ RF ↑ Comp↑ TS ↓ ACC↓ YV ↓ LI ↓
4D DDPG 0.833 0.077 0.000 8.385 0.805 0.132 0.000 2.502 0.211 2.761 0.501
SAC 0.887 0.216 0.000 7.609 0.874 0.679 18.101 2.332 0.947 4.378 0.471
TD3 0.632 0.198 0.000 7.105 0.869 0.746 22.531 2.940 1.652 5.938 0.592
PPO 0.964 0.224 0.000 0.057 0.894 0.612 26.714 2.681 1.707 3.956 0.432
Dir SAC 1.000 0.224 0.000 2.077 0.854 0.533 0.000 2.092 0.753 3.661 0.435
TD3 0.928 0.219 0.000 1.835 0.882 0.624 21.550 2.645 1.390 6.077 0.451
PPO 1.000 0.082 0.000 3.113 0.864 0.164 0.000 3.195 0.507 3.517 0.428
BEV SAC 0.900 0.208 0.000 11.291 0.854 0.673 19.526 3.251 2.084 4.023 0.446
PPO 0.979 0.172 0.000 1.908 0.834 0.317 44.431 2.595 0.996 9.468 0.393
Cam SAC 1.000 0.234 0.000 0.009 0.921 0.583 0.000 4.528 2.213 3.969 0.427
PPO 1.000 0.036 0.000 5.902 0.830 0.127 0.000 2.625 0.458 3.656 0.425
1
1
Leaderboard of Secnario: Red-light Running
State Space Algo. Safety Level Functionality Level Etiquette Level OS ↑
CR ↓ RR ↓ SS ↓ OR ↓ RF ↑ Comp↑ TS ↓ ACC↓ YV ↓ LI ↓
4D DDPG 0.646 0.003 0.010 0.378 0.590 0.692 19.241 2.293 0.512 6.486 0.611
SAC 0.937 0.011 0.109 0.000 0.927 0.575 14.222 1.318 0.479 0.028 0.482
TD3 0.823 0.017 0.108 0.000 0.940 0.634 14.190 2.203 1.486 0.160 0.532
PPO 0.378 0.014 0.108 0.000 0.923 0.819 17.642 2.221 1.474 2.910 0.755
Dir SAC 0.620 0.014 0.108 0.861 0.766 0.716 23.802 1.729 0.486 2.624 0.624
TD3 0.557 0.014 0.108 0.000 0.912 0.738 17.690 2.133 1.128 1.983 0.665
PPO 0.752 0.003 0.000 0.854 0.646 0.640 17.615 2.726 1.054 4.657 0.558
BEV SAC 0.906 0.021 0.108 0.000 0.927 0.592 13.793 2.631 1.200 3.288 0.486
PPO 0.024 0.035 0.108 0.007 0.769 0.989 29.201 2.609 1.412 4.115 0.918
Cam SAC 0.892 0.021 0.108 0.000 0.934 0.590 14.916 4.200 1.870 5.648 0.481
PPO 0.003 0.010 0.080 0.347 0.669 0.998 30.789 2.592 1.046 5.250 0.928
1
1
Leaderboard of Secnario: Unprotected Left-turn
State Space Algo. Safety Level Functionality Level Etiquette Level OS ↑
CR ↓ RR ↓ SS ↓ OR ↓ RF ↑ Comp↑ TS ↓ ACC↓ YV ↓ LI ↓
4D DDPG 0.911 0.038 0.000 4.016 0.347 0.298 16.277 2.945 0.212 3.722 0.444
SAC 0.909 0.041 0.000 0.099 0.927 0.615 12.334 2.268 1.019 0.032 0.501
TD3 0.846 0.044 0.000 0.030 0.921 0.642 12.076 3.067 1.870 0.263 0.525
PPO 0.439 0.110 0.000 0.298 0.917 0.825 16.115 2.792 1.462 0.966 0.728
Dir SAC 0.809 0.034 0.000 0.541 0.890 0.661 13.997 2.491 0.766 2.016 0.548
TD3 0.702 0.069 0.000 0.153 0.890 0.698 16.076 2.827 1.128 2.621 0.595
PPO 0.912 0.009 0.000 0.990 0.824 0.296 16.157 3.155 0.618 2.614 0.474
BEV SAC 0.850 0.044 0.000 0.050 0.926 0.640 11.877 3.450 2.050 0.191 0.521
PPO 0.524 0.094 0.000 0.274 0.830 0.648 26.943 2.520 0.960 6.925 0.664
Cam SAC 0.815 0.072 0.000 0.052 0.934 0.621 13.271 4.615 1.876 3.179 0.529
PPO 0.805 0.003 0.000 1.848 0.811 0.293 27.237 2.580 0.444 2.506 0.519
1
1
Leaderboard of Secnario: Right-turn
State Space Algo. Safety Level Functionality Level Etiquette Level OS ↑
CR ↓ RR ↓ SS ↓ OR ↓ RF ↑ Comp↑ TS ↓ ACC↓ YV ↓ LI ↓
4D DDPG 0.852 0.292 0.025 19.265 0.278 0.388 23.700 2.658 0.090 4.877 0.411
SAC 0.713 1.048 0.000 1.072 0.894 0.772 15.404 1.772 0.505 1.540 0.503
TD3 0.775 1.137 0.000 0.170 0.910 0.755 14.972 2.820 1.057 1.553 0.459
PPO 0.475 1.144 0.000 0.075 0.887 0.848 18.766 2.593 1.080 4.447 0.605
Dir SAC 0.575 0.793 0.000 12.366 0.538 0.812 19.600 1.872 0.388 4.579 0.552
TD3 0.399 1.142 0.000 0.167 0.877 0.860 18.382 2.331 0.833 5.231 0.645
PPO 0.780 0.117 0.000 22.058 0.652 0.478 17.659 3.113 0.312 4.440 0.487
BEV SAC 0.806 1.032 0.000 0.639 0.898 0.719 16.029 2.929 1.023 2.489 0.449
PPO 0.225 0.947 0.000 0.782 0.784 0.856 28.183 2.644 1.051 9.898 0.729
Cam SAC 0.614 1.154 0.000 0.028 0.910 0.803 16.329 4.322 1.390 7.000 0.527
PPO 0.575 0.229 0.000 17.902 0.711 0.595 31.675 2.648 0.330 8.664 0.579
1
1
Leaderboard of Secnario: Crossing Negotiation
State Space Algo. Safety Level Functionality Level Etiquette Level OS ↑
CR ↓ RR ↓ SS ↓ OR ↓ RF ↑ Comp↑ TS ↓ ACC↓ YV ↓ LI ↓
4D DDPG 0.693 0.000 0.675 9.053 0.629 0.658 19.109 2.350 0.538 4.282 0.507
SAC 0.873 0.000 0.942 0.000 0.924 0.607 13.822 1.478 0.604 0.031 0.432
TD3 0.763 0.000 0.942 0.000 0.938 0.662 13.751 2.296 1.510 0.471 0.482
PPO 0.419 0.000 0.942 0.000 0.924 0.800 17.392 2.214 1.446 1.182 0.655
Dir SAC 0.667 0.000 0.952 0.774 0.825 0.703 20.992 1.962 0.567 2.990 0.522
TD3 0.550 0.000 0.904 0.074 0.896 0.750 18.337 2.206 1.105 2.918 0.590
PPO 0.564 0.000 0.584 16.680 0.631 0.743 17.647 2.660 0.999 3.598 0.568
BEV SAC 0.852 0.000 0.942 0.000 0.932 0.616 13.507 2.719 1.410 1.333 0.434
PPO 0.010 0.000 0.945 0.108 0.775 0.996 28.796 2.609 1.429 3.402 0.847
Cam SAC 0.852 0.000 0.942 0.000 0.936 0.612 14.416 4.226 1.708 4.323 0.425
PPO 0.031 0.000 0.869 15.061 0.666 0.978 30.467 2.598 1.090 4.557 0.808