In SafeBench leaderboard, we present the diagnostic report for each AD algorithm from 3 different levels: safety level, functionality level, and etiquette level. Notably, we offer the evaluation results in different scenarios to enable better understanding of AD algorithm performance in different traffic situations. Please find more details in our paper here.
State Space | Algo. | Safety Level | Functionality Level | Etiquette Level | OS ↑ | |||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
CR ↓ | RR ↓ | SS ↓ | OR ↓ | RF ↑ | Comp↑ | TS ↓ | ACC↓ | YV ↓ | LI ↓ | |||
4D | DDPG | 0.780 | 0.089 | 0.087 | 12.619 | 0.504 | 0.466 | 20.860 | 2.488 | 0.405 | 5.764 | 0.489 |
SAC | 0.829 | 0.216 | 0.146 | 3.115 | 0.882 | 0.648 | 16.827 | 1.830 | 0.704 | 2.580 | 0.499 | |
TD3 | 0.783 | 0.231 | 0.141 | 2.535 | 0.903 | 0.670 | 17.644 | 2.680 | 1.493 | 2.545 | 0.516 | |
PPO | 0.603 | 0.287 | 0.150 | 0.099 | 0.901 | 0.751 | 18.021 | 2.461 | 1.506 | 3.528 | 0.606 | |
Dir | SAC | 0.676 | 0.209 | 0.152 | 5.658 | 0.740 | 0.705 | 23.386 | 1.892 | 0.640 | 4.565 | 0.558 |
TD3 | 0.655 | 0.270 | 0.144 | 0.885 | 0.887 | 0.718 | 18.899 | 2.417 | 1.187 | 4.694 | 0.579 | |
PPO | 0.739 | 0.045 | 0.077 | 17.607 | 0.685 | 0.534 | 21.336 | 2.911 | 0.893 | 4.875 | 0.513 | |
BEV | SAC | 0.782 | 0.229 | 0.141 | 6.057 | 0.883 | 0.674 | 17.863 | 2.952 | 1.566 | 4.448 | 0.506 |
PPO | 0.416 | 0.262 | 0.151 | 2.180 | 0.782 | 0.756 | 30.651 | 2.592 | 1.290 | 7.319 | 0.679 | |
Cam | SAC | 0.829 | 0.261 | 0.149 | 0.014 | 0.926 | 0.637 | 15.480 | 4.354 | 1.885 | 6.139 | 0.485 |
PPO | 0.600 | 0.050 | 0.127 | 15.101 | 0.708 | 0.599 | 31.914 | 2.631 | 0.827 | 6.327 | 0.576 |
State Space | Algo. | Safety Level | Functionality Level | Etiquette Level | OS ↑ | |||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
CR ↓ | RR ↓ | SS ↓ | OR ↓ | RF ↑ | Comp↑ | TS ↓ | ACC↓ | YV ↓ | LI ↓ | |||
4D | DDPG | 0.649 | 0.083 | 0.013 | 27.908 | 0.636 | 0.582 | 20.387 | 2.473 | 0.276 | 5.263 | 0.545 |
SAC | 0.820 | 0.101 | 0.000 | 0.225 | 0.824 | 0.547 | 15.423 | 1.216 | 0.348 | 0.899 | 0.533 | |
TD3 | 0.930 | 0.088 | 0.000 | 0.042 | 0.926 | 0.538 | 15.519 | 2.186 | 0.971 | 0.610 | 0.479 | |
PPO | 0.285 | 0.465 | 0.083 | 0.003 | 0.916 | 0.860 | 19.615 | 2.031 | 1.346 | 2.136 | 0.761 | |
Dir | SAC | 0.589 | 0.302 | 0.064 | 7.204 | 0.764 | 0.686 | 16.771 | 1.388 | 0.323 | 3.728 | 0.608 |
TD3 | 0.360 | 0.399 | 0.070 | 0.027 | 0.891 | 0.819 | 19.732 | 2.028 | 0.873 | 5.781 | 0.728 | |
PPO | 0.715 | 0.035 | 0.000 | 32.834 | 0.640 | 0.605 | 19.355 | 2.945 | 0.842 | 5.627 | 0.506 | |
BEV | SAC | 0.873 | 0.114 | 0.000 | 0.006 | 0.917 | 0.564 | 15.028 | 2.612 | 0.979 | 4.399 | 0.501 |
PPO | 0.114 | 0.535 | 0.096 | 0.518 | 0.774 | 0.941 | 32.754 | 2.644 | 1.361 | 5.083 | 0.818 | |
Cam | SAC | 0.556 | 0.292 | 0.060 | 0.007 | 0.929 | 0.735 | 16.454 | 3.981 | 1.303 | 7.069 | 0.634 |
PPO | 0.640 | 0.035 | 0.013 | 28.394 | 0.674 | 0.620 | 33.832 | 2.718 | 0.700 | 6.127 | 0.542 |
State Space | Algo. | Safety Level | Functionality Level | Etiquette Level | OS ↑ | |||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
CR ↓ | RR ↓ | SS ↓ | OR ↓ | RF ↑ | Comp↑ | TS ↓ | ACC↓ | YV ↓ | LI ↓ | |||
4D | DDPG | 0.613 | 0.313 | 0.025 | 18.160 | 0.000 | 0.543 | 18.175 | 2.229 | 0.225 | 6.577 | 0.526 |
SAC | 0.811 | 0.555 | 0.213 | 0.035 | 0.766 | 0.778 | 15.142 | 2.450 | 0.354 | 4.628 | 0.474 | |
TD3 | 0.579 | 0.598 | 0.159 | 0.369 | 0.877 | 0.873 | 15.262 | 3.373 | 0.564 | 4.848 | 0.596 | |
PPO | 0.530 | 0.671 | 0.177 | 0.057 | 0.876 | 0.897 | 17.855 | 2.815 | 0.609 | 5.341 | 0.611 | |
Dir | SAC | 0.579 | 0.427 | 0.159 | 5.126 | 0.598 | 0.839 | 16.417 | 2.059 | 0.239 | 6.171 | 0.591 |
TD3 | 0.677 | 0.628 | 0.171 | 0.324 | 0.868 | 0.860 | 16.866 | 2.723 | 0.397 | 5.640 | 0.543 | |
PPO | 0.671 | 0.152 | 0.061 | 26.030 | 0.553 | 0.720 | 18.689 | 2.950 | 0.526 | 7.543 | 0.526 | |
BEV | SAC | 0.628 | 0.585 | 0.159 | 0.252 | 0.851 | 0.867 | 14.957 | 3.523 | 0.689 | 8.628 | 0.567 |
PPO | 0.457 | 0.591 | 0.159 | 3.281 | 0.756 | 0.915 | 30.213 | 2.727 | 0.620 | 8.951 | 0.632 | |
Cam | SAC | 0.609 | 0.623 | 0.179 | 0.021 | 0.904 | 0.870 | 15.624 | 4.545 | 0.682 | 9.616 | 0.570 |
PPO | 0.713 | 0.122 | 0.116 | 22.859 | 0.621 | 0.801 | 31.323 | 2.646 | 0.443 | 10.762 | 0.503 |
State Space | Algo. | Safety Level | Functionality Level | Etiquette Level | OS ↑ | |||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
CR ↓ | RR ↓ | SS ↓ | OR ↓ | RF ↑ | Comp↑ | TS ↓ | ACC↓ | YV ↓ | LI ↓ | |||
4D | DDPG | 0.874 | 0.028 | 0.000 | 19.304 | 0.627 | 0.591 | 29.347 | 2.339 | 0.962 | 11.550 | 0.440 |
SAC | 0.692 | 0.015 | 0.000 | 10.105 | 0.866 | 0.648 | 20.510 | 1.750 | 0.942 | 7.478 | 0.577 | |
TD3 | 0.871 | 0.018 | 0.000 | 7.896 | 0.861 | 0.579 | 25.722 | 2.636 | 2.031 | 5.226 | 0.477 | |
PPO | 0.992 | 0.021 | 0.000 | 0.215 | 0.874 | 0.547 | 29.967 | 2.338 | 2.195 | 6.797 | 0.426 | |
Dir | SAC | 0.427 | 0.063 | 0.000 | 16.257 | 0.601 | 0.792 | 34.497 | 1.428 | 1.154 | 10.496 | 0.670 |
TD3 | 0.843 | 0.021 | 0.000 | 3.051 | 0.877 | 0.567 | 27.022 | 2.381 | 1.912 | 6.918 | 0.499 | |
PPO | 0.468 | 0.008 | 0.000 | 43.038 | 0.569 | 0.787 | 27.886 | 2.567 | 1.915 | 7.951 | 0.601 | |
BEV | SAC | 0.458 | 0.057 | 0.000 | 24.713 | 0.789 | 0.759 | 22.034 | 2.631 | 2.149 | 11.249 | 0.647 |
PPO | 0.653 | 0.051 | 0.000 | 8.701 | 0.723 | 0.683 | 49.359 | 2.496 | 2.077 | 9.530 | 0.555 | |
Cam | SAC | 1.000 | 0.015 | 0.000 | 0.000 | 0.930 | 0.495 | 0.000 | 4.333 | 2.853 | 9.969 | 0.436 |
PPO | 0.848 | 0.021 | 0.000 | 33.008 | 0.614 | 0.647 | 48.953 | 2.661 | 1.744 | 10.807 | 0.407 |
State Space | Algo. | Safety Level | Functionality Level | Etiquette Level | OS ↑ | |||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
CR ↓ | RR ↓ | SS ↓ | OR ↓ | RF ↑ | Comp↑ | TS ↓ | ACC↓ | YV ↓ | LI ↓ | |||
4D | DDPG | 0.833 | 0.077 | 0.000 | 8.385 | 0.805 | 0.132 | 0.000 | 2.502 | 0.211 | 2.761 | 0.501 |
SAC | 0.887 | 0.216 | 0.000 | 7.609 | 0.874 | 0.679 | 18.101 | 2.332 | 0.947 | 4.378 | 0.471 | |
TD3 | 0.632 | 0.198 | 0.000 | 7.105 | 0.869 | 0.746 | 22.531 | 2.940 | 1.652 | 5.938 | 0.592 | |
PPO | 0.964 | 0.224 | 0.000 | 0.057 | 0.894 | 0.612 | 26.714 | 2.681 | 1.707 | 3.956 | 0.432 | |
Dir | SAC | 1.000 | 0.224 | 0.000 | 2.077 | 0.854 | 0.533 | 0.000 | 2.092 | 0.753 | 3.661 | 0.435 |
TD3 | 0.928 | 0.219 | 0.000 | 1.835 | 0.882 | 0.624 | 21.550 | 2.645 | 1.390 | 6.077 | 0.451 | |
PPO | 1.000 | 0.082 | 0.000 | 3.113 | 0.864 | 0.164 | 0.000 | 3.195 | 0.507 | 3.517 | 0.428 | |
BEV | SAC | 0.900 | 0.208 | 0.000 | 11.291 | 0.854 | 0.673 | 19.526 | 3.251 | 2.084 | 4.023 | 0.446 |
PPO | 0.979 | 0.172 | 0.000 | 1.908 | 0.834 | 0.317 | 44.431 | 2.595 | 0.996 | 9.468 | 0.393 | |
Cam | SAC | 1.000 | 0.234 | 0.000 | 0.009 | 0.921 | 0.583 | 0.000 | 4.528 | 2.213 | 3.969 | 0.427 |
PPO | 1.000 | 0.036 | 0.000 | 5.902 | 0.830 | 0.127 | 0.000 | 2.625 | 0.458 | 3.656 | 0.425 |
State Space | Algo. | Safety Level | Functionality Level | Etiquette Level | OS ↑ | |||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
CR ↓ | RR ↓ | SS ↓ | OR ↓ | RF ↑ | Comp↑ | TS ↓ | ACC↓ | YV ↓ | LI ↓ | |||
4D | DDPG | 0.646 | 0.003 | 0.010 | 0.378 | 0.590 | 0.692 | 19.241 | 2.293 | 0.512 | 6.486 | 0.611 |
SAC | 0.937 | 0.011 | 0.109 | 0.000 | 0.927 | 0.575 | 14.222 | 1.318 | 0.479 | 0.028 | 0.482 | |
TD3 | 0.823 | 0.017 | 0.108 | 0.000 | 0.940 | 0.634 | 14.190 | 2.203 | 1.486 | 0.160 | 0.532 | |
PPO | 0.378 | 0.014 | 0.108 | 0.000 | 0.923 | 0.819 | 17.642 | 2.221 | 1.474 | 2.910 | 0.755 | |
Dir | SAC | 0.620 | 0.014 | 0.108 | 0.861 | 0.766 | 0.716 | 23.802 | 1.729 | 0.486 | 2.624 | 0.624 |
TD3 | 0.557 | 0.014 | 0.108 | 0.000 | 0.912 | 0.738 | 17.690 | 2.133 | 1.128 | 1.983 | 0.665 | |
PPO | 0.752 | 0.003 | 0.000 | 0.854 | 0.646 | 0.640 | 17.615 | 2.726 | 1.054 | 4.657 | 0.558 | |
BEV | SAC | 0.906 | 0.021 | 0.108 | 0.000 | 0.927 | 0.592 | 13.793 | 2.631 | 1.200 | 3.288 | 0.486 |
PPO | 0.024 | 0.035 | 0.108 | 0.007 | 0.769 | 0.989 | 29.201 | 2.609 | 1.412 | 4.115 | 0.918 | |
Cam | SAC | 0.892 | 0.021 | 0.108 | 0.000 | 0.934 | 0.590 | 14.916 | 4.200 | 1.870 | 5.648 | 0.481 |
PPO | 0.003 | 0.010 | 0.080 | 0.347 | 0.669 | 0.998 | 30.789 | 2.592 | 1.046 | 5.250 | 0.928 |
State Space | Algo. | Safety Level | Functionality Level | Etiquette Level | OS ↑ | |||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
CR ↓ | RR ↓ | SS ↓ | OR ↓ | RF ↑ | Comp↑ | TS ↓ | ACC↓ | YV ↓ | LI ↓ | |||
4D | DDPG | 0.911 | 0.038 | 0.000 | 4.016 | 0.347 | 0.298 | 16.277 | 2.945 | 0.212 | 3.722 | 0.444 |
SAC | 0.909 | 0.041 | 0.000 | 0.099 | 0.927 | 0.615 | 12.334 | 2.268 | 1.019 | 0.032 | 0.501 | |
TD3 | 0.846 | 0.044 | 0.000 | 0.030 | 0.921 | 0.642 | 12.076 | 3.067 | 1.870 | 0.263 | 0.525 | |
PPO | 0.439 | 0.110 | 0.000 | 0.298 | 0.917 | 0.825 | 16.115 | 2.792 | 1.462 | 0.966 | 0.728 | |
Dir | SAC | 0.809 | 0.034 | 0.000 | 0.541 | 0.890 | 0.661 | 13.997 | 2.491 | 0.766 | 2.016 | 0.548 |
TD3 | 0.702 | 0.069 | 0.000 | 0.153 | 0.890 | 0.698 | 16.076 | 2.827 | 1.128 | 2.621 | 0.595 | |
PPO | 0.912 | 0.009 | 0.000 | 0.990 | 0.824 | 0.296 | 16.157 | 3.155 | 0.618 | 2.614 | 0.474 | |
BEV | SAC | 0.850 | 0.044 | 0.000 | 0.050 | 0.926 | 0.640 | 11.877 | 3.450 | 2.050 | 0.191 | 0.521 |
PPO | 0.524 | 0.094 | 0.000 | 0.274 | 0.830 | 0.648 | 26.943 | 2.520 | 0.960 | 6.925 | 0.664 | |
Cam | SAC | 0.815 | 0.072 | 0.000 | 0.052 | 0.934 | 0.621 | 13.271 | 4.615 | 1.876 | 3.179 | 0.529 |
PPO | 0.805 | 0.003 | 0.000 | 1.848 | 0.811 | 0.293 | 27.237 | 2.580 | 0.444 | 2.506 | 0.519 |
State Space | Algo. | Safety Level | Functionality Level | Etiquette Level | OS ↑ | |||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
CR ↓ | RR ↓ | SS ↓ | OR ↓ | RF ↑ | Comp↑ | TS ↓ | ACC↓ | YV ↓ | LI ↓ | |||
4D | DDPG | 0.852 | 0.292 | 0.025 | 19.265 | 0.278 | 0.388 | 23.700 | 2.658 | 0.090 | 4.877 | 0.411 |
SAC | 0.713 | 1.048 | 0.000 | 1.072 | 0.894 | 0.772 | 15.404 | 1.772 | 0.505 | 1.540 | 0.503 | |
TD3 | 0.775 | 1.137 | 0.000 | 0.170 | 0.910 | 0.755 | 14.972 | 2.820 | 1.057 | 1.553 | 0.459 | |
PPO | 0.475 | 1.144 | 0.000 | 0.075 | 0.887 | 0.848 | 18.766 | 2.593 | 1.080 | 4.447 | 0.605 | |
Dir | SAC | 0.575 | 0.793 | 0.000 | 12.366 | 0.538 | 0.812 | 19.600 | 1.872 | 0.388 | 4.579 | 0.552 |
TD3 | 0.399 | 1.142 | 0.000 | 0.167 | 0.877 | 0.860 | 18.382 | 2.331 | 0.833 | 5.231 | 0.645 | |
PPO | 0.780 | 0.117 | 0.000 | 22.058 | 0.652 | 0.478 | 17.659 | 3.113 | 0.312 | 4.440 | 0.487 | |
BEV | SAC | 0.806 | 1.032 | 0.000 | 0.639 | 0.898 | 0.719 | 16.029 | 2.929 | 1.023 | 2.489 | 0.449 |
PPO | 0.225 | 0.947 | 0.000 | 0.782 | 0.784 | 0.856 | 28.183 | 2.644 | 1.051 | 9.898 | 0.729 | |
Cam | SAC | 0.614 | 1.154 | 0.000 | 0.028 | 0.910 | 0.803 | 16.329 | 4.322 | 1.390 | 7.000 | 0.527 |
PPO | 0.575 | 0.229 | 0.000 | 17.902 | 0.711 | 0.595 | 31.675 | 2.648 | 0.330 | 8.664 | 0.579 |
State Space | Algo. | Safety Level | Functionality Level | Etiquette Level | OS ↑ | |||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
CR ↓ | RR ↓ | SS ↓ | OR ↓ | RF ↑ | Comp↑ | TS ↓ | ACC↓ | YV ↓ | LI ↓ | |||
4D | DDPG | 0.693 | 0.000 | 0.675 | 9.053 | 0.629 | 0.658 | 19.109 | 2.350 | 0.538 | 4.282 | 0.507 |
SAC | 0.873 | 0.000 | 0.942 | 0.000 | 0.924 | 0.607 | 13.822 | 1.478 | 0.604 | 0.031 | 0.432 | |
TD3 | 0.763 | 0.000 | 0.942 | 0.000 | 0.938 | 0.662 | 13.751 | 2.296 | 1.510 | 0.471 | 0.482 | |
PPO | 0.419 | 0.000 | 0.942 | 0.000 | 0.924 | 0.800 | 17.392 | 2.214 | 1.446 | 1.182 | 0.655 | |
Dir | SAC | 0.667 | 0.000 | 0.952 | 0.774 | 0.825 | 0.703 | 20.992 | 1.962 | 0.567 | 2.990 | 0.522 |
TD3 | 0.550 | 0.000 | 0.904 | 0.074 | 0.896 | 0.750 | 18.337 | 2.206 | 1.105 | 2.918 | 0.590 | |
PPO | 0.564 | 0.000 | 0.584 | 16.680 | 0.631 | 0.743 | 17.647 | 2.660 | 0.999 | 3.598 | 0.568 | |
BEV | SAC | 0.852 | 0.000 | 0.942 | 0.000 | 0.932 | 0.616 | 13.507 | 2.719 | 1.410 | 1.333 | 0.434 |
PPO | 0.010 | 0.000 | 0.945 | 0.108 | 0.775 | 0.996 | 28.796 | 2.609 | 1.429 | 3.402 | 0.847 | |
Cam | SAC | 0.852 | 0.000 | 0.942 | 0.000 | 0.936 | 0.612 | 14.416 | 4.226 | 1.708 | 4.323 | 0.425 |
PPO | 0.031 | 0.000 | 0.869 | 15.061 | 0.666 | 0.978 | 30.467 | 2.598 | 1.090 | 4.557 | 0.808 |