Benchmarks#
Latest ADBench (2022)#
We just released a 36-page, the most comprehensive anomaly detection benchmark paper [AHHH+22]. The fully open-sourced ADBench compares 30 anomaly detection algorithms on 55 benchmark datasets.
The organization of ADBench is provided below:
Old Results (2019)#
A benchmark is supplied for select algorithms to provide an overview of the implemented models. In total, 17 benchmark datasets are used for comparison, which can be downloaded at ODDS.
For each dataset, it is first split into 60% for training and 40% for testing. All experiments are repeated 10 times independently with random splits. The mean of 10 trials is regarded as the final result. Three evaluation metrics are provided:
The area under receiver operating characteristic (ROC) curve
Precision @ rank n (P@N)
Execution time
You could replicate this process by running benchmark.py.
We also provide the hardware specification for reference.
Specification |
Value |
|---|---|
Platform |
PC |
OS |
Microsoft Windows 10 Enterprise |
CPU |
Intel i7-6820HQ @ 2.70GHz |
RAM |
32GB |
Software |
PyCharm 2018.02 |
Python |
Python 3.6.2 |
Core |
Single core (no parallelization) |
ROC Performance#
Data |
#Samples |
# Dimensions |
Outlier Perc |
ABOD |
CBLOF |
FB |
HBOS |
IForest |
KNN |
LOF |
MCD |
OCSVM |
PCA |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
arrhythmia |
452 |
274 |
14.6018 |
0.7688 |
0.7835 |
0.7781 |
0.8219 |
0.8005 |
0.7861 |
0.7787 |
0.7790 |
0.7812 |
0.7815 |
cardio |
1831 |
21 |
9.6122 |
0.5692 |
0.9276 |
0.5867 |
0.8351 |
0.9213 |
0.7236 |
0.5736 |
0.8135 |
0.9348 |
0.9504 |
glass |
214 |
9 |
4.2056 |
0.7951 |
0.8504 |
0.8726 |
0.7389 |
0.7569 |
0.8508 |
0.8644 |
0.7901 |
0.6324 |
0.6747 |
ionosphere |
351 |
33 |
35.8974 |
0.9248 |
0.8134 |
0.8730 |
0.5614 |
0.8499 |
0.9267 |
0.8753 |
0.9557 |
0.8419 |
0.7962 |
letter |
1600 |
32 |
6.2500 |
0.8783 |
0.5070 |
0.8660 |
0.5927 |
0.6420 |
0.8766 |
0.8594 |
0.8074 |
0.6118 |
0.5283 |
lympho |
148 |
18 |
4.0541 |
0.9110 |
0.9728 |
0.9753 |
0.9957 |
0.9941 |
0.9745 |
0.9771 |
0.9000 |
0.9759 |
0.9847 |
mnist |
7603 |
100 |
9.2069 |
0.7815 |
0.8009 |
0.7205 |
0.5742 |
0.8159 |
0.8481 |
0.7161 |
0.8666 |
0.8529 |
0.8527 |
musk |
3062 |
166 |
3.1679 |
0.1844 |
0.9879 |
0.5263 |
1.0000 |
0.9999 |
0.7986 |
0.5287 |
0.9998 |
1.0000 |
1.0000 |
optdigits |
5216 |
64 |
2.8758 |
0.4667 |
0.5089 |
0.4434 |
0.8732 |
0.7253 |
0.3708 |
0.4500 |
0.3979 |
0.4997 |
0.5086 |
pendigits |
6870 |
16 |
2.2707 |
0.6878 |
0.9486 |
0.4595 |
0.9238 |
0.9435 |
0.7486 |
0.4698 |
0.8344 |
0.9303 |
0.9352 |
pima |
768 |
8 |
34.8958 |
0.6794 |
0.7348 |
0.6235 |
0.7000 |
0.6806 |
0.7078 |
0.6271 |
0.6753 |
0.6215 |
0.6481 |
satellite |
6435 |
36 |
31.6395 |
0.5714 |
0.6693 |
0.5572 |
0.7581 |
0.7022 |
0.6836 |
0.5573 |
0.8030 |
0.6622 |
0.5988 |
satimage-2 |
5803 |
36 |
1.2235 |
0.8190 |
0.9917 |
0.4570 |
0.9804 |
0.9947 |
0.9536 |
0.4577 |
0.9959 |
0.9978 |
0.9822 |
shuttle |
49097 |
9 |
7.1511 |
0.6234 |
0.6272 |
0.4724 |
0.9855 |
0.9971 |
0.6537 |
0.5264 |
0.9903 |
0.9917 |
0.9898 |
vertebral |
240 |
6 |
12.5000 |
0.4262 |
0.3486 |
0.4166 |
0.3263 |
0.3905 |
0.3817 |
0.4081 |
0.3906 |
0.4431 |
0.4027 |
vowels |
1456 |
12 |
3.4341 |
0.9606 |
0.5856 |
0.9425 |
0.6727 |
0.7585 |
0.9680 |
0.9410 |
0.8076 |
0.7802 |
0.6027 |
wbc |
378 |
30 |
5.5556 |
0.9047 |
0.9227 |
0.9325 |
0.9516 |
0.9310 |
0.9366 |
0.9349 |
0.9210 |
0.9319 |
0.9159 |
P@N Performance#
Data |
#Samples |
# Dimensions |
Outlier Perc |
ABOD |
CBLOF |
FB |
HBOS |
IForest |
KNN |
LOF |
MCD |
OCSVM |
PCA |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
arrhythmia |
452 |
274 |
14.6018 |
0.3808 |
0.4539 |
0.4230 |
0.5111 |
0.4961 |
0.4464 |
0.4334 |
0.3995 |
0.4614 |
0.4613 |
cardio |
1831 |
21 |
9.6122 |
0.2374 |
0.5876 |
0.1690 |
0.4476 |
0.5041 |
0.3323 |
0.1541 |
0.4317 |
0.5011 |
0.6090 |
glass |
214 |
9 |
4.2056 |
0.1702 |
0.0726 |
0.1476 |
0.0000 |
0.0726 |
0.0726 |
0.1476 |
0.0000 |
0.1726 |
0.0726 |
ionosphere |
351 |
33 |
35.8974 |
0.8442 |
0.6088 |
0.7056 |
0.3295 |
0.6369 |
0.8602 |
0.7063 |
0.8806 |
0.7000 |
0.5729 |
letter |
1600 |
32 |
6.2500 |
0.3801 |
0.0749 |
0.3642 |
0.0715 |
0.1003 |
0.3312 |
0.3641 |
0.1933 |
0.1510 |
0.0875 |
lympho |
148 |
18 |
4.0541 |
0.4483 |
0.7517 |
0.7517 |
0.8467 |
0.9267 |
0.7517 |
0.7517 |
0.5183 |
0.7517 |
0.7517 |
mnist |
7603 |
100 |
9.2069 |
0.3555 |
0.3348 |
0.3299 |
0.1188 |
0.3135 |
0.4204 |
0.3343 |
0.3462 |
0.3962 |
0.3846 |
musk |
3062 |
166 |
3.1679 |
0.0507 |
0.7766 |
0.2230 |
0.9783 |
0.9680 |
0.2733 |
0.1695 |
0.9742 |
1.0000 |
0.9799 |
optdigits |
5216 |
64 |
2.8758 |
0.0060 |
0.0000 |
0.0244 |
0.2194 |
0.0301 |
0.0000 |
0.0234 |
0.0000 |
0.0000 |
0.0000 |
pendigits |
6870 |
16 |
2.2707 |
0.0812 |
0.2768 |
0.0658 |
0.2979 |
0.3422 |
0.0984 |
0.0653 |
0.0893 |
0.3287 |
0.3187 |
pima |
768 |
8 |
34.8958 |
0.5193 |
0.5413 |
0.4480 |
0.5424 |
0.5111 |
0.5413 |
0.4555 |
0.4962 |
0.4704 |
0.4943 |
satellite |
6435 |
36 |
31.6395 |
0.3902 |
0.4152 |
0.3902 |
0.5690 |
0.5676 |
0.4994 |
0.3893 |
0.6845 |
0.5346 |
0.4784 |
satimage-2 |
5803 |
36 |
1.2235 |
0.2130 |
0.8846 |
0.0555 |
0.6939 |
0.8754 |
0.3809 |
0.0555 |
0.6481 |
0.9356 |
0.8041 |
shuttle |
49097 |
9 |
7.1511 |
0.1977 |
0.2943 |
0.0695 |
0.9551 |
0.9546 |
0.2184 |
0.1424 |
0.7506 |
0.9542 |
0.9501 |
vertebral |
240 |
6 |
12.5000 |
0.0601 |
0.0000 |
0.0644 |
0.0071 |
0.0343 |
0.0238 |
0.0506 |
0.0071 |
0.0238 |
0.0226 |
vowels |
1456 |
12 |
3.4341 |
0.5710 |
0.0831 |
0.3224 |
0.1297 |
0.1875 |
0.5093 |
0.3551 |
0.2186 |
0.2791 |
0.1364 |
wbc |
378 |
30 |
5.5556 |
0.3060 |
0.5055 |
0.5188 |
0.5817 |
0.5088 |
0.4952 |
0.5188 |
0.4577 |
0.5125 |
0.4767 |
Execution Time#
Data |
#Samples |
# Dimensions |
Outlier Perc |
ABOD |
CBLOF |
FB |
HBOS |
IForest |
KNN |
LOF |
MCD |
OCSVM |
PCA |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
arrhythmia |
452 |
274 |
14.6018 |
0.3667 |
0.2123 |
0.5651 |
0.1383 |
0.2669 |
0.1075 |
0.0743 |
1.4165 |
0.0473 |
0.0596 |
cardio |
1831 |
21 |
9.6122 |
0.3824 |
0.1255 |
0.7741 |
0.0053 |
0.2672 |
0.2249 |
0.0993 |
0.5418 |
0.0883 |
0.0035 |
glass |
214 |
9 |
4.2056 |
0.0352 |
0.0359 |
0.0317 |
0.0022 |
0.1724 |
0.0173 |
0.0025 |
0.0325 |
0.0010 |
0.0011 |
ionosphere |
351 |
33 |
35.8974 |
0.0645 |
0.0459 |
0.0728 |
0.0082 |
0.1864 |
0.0302 |
0.0070 |
0.0718 |
0.0048 |
0.0018 |
letter |
1600 |
32 |
6.2500 |
0.3435 |
0.1014 |
0.7361 |
0.0080 |
0.2617 |
0.1882 |
0.0935 |
1.1942 |
0.0888 |
0.0041 |
lympho |
148 |
18 |
4.0541 |
0.0277 |
0.0353 |
0.0266 |
0.0037 |
0.1712 |
0.0111 |
0.0021 |
0.0327 |
0.0014 |
0.0012 |
mnist |
7603 |
100 |
9.2069 |
7.4192 |
1.1339 |
48.2750 |
0.0480 |
1.9314 |
7.3431 |
6.7901 |
4.7448 |
5.0203 |
0.1569 |
musk |
3062 |
166 |
3.1679 |
2.3860 |
0.4134 |
13.8610 |
0.0587 |
1.2736 |
2.2057 |
1.9835 |
25.5501 |
1.3774 |
0.1637 |
optdigits |
5216 |
64 |
2.8758 |
2.7279 |
0.4977 |
14.2399 |
0.0303 |
0.7783 |
2.1205 |
1.7799 |
1.8599 |
1.5618 |
0.0519 |
pendigits |
6870 |
16 |
2.2707 |
1.4339 |
0.2847 |
3.8185 |
0.0090 |
0.5879 |
0.8659 |
0.5936 |
2.2209 |
0.9666 |
0.0062 |
pima |
768 |
8 |
34.8958 |
0.1357 |
0.0698 |
0.0908 |
0.0019 |
0.1923 |
0.0590 |
0.0102 |
0.0474 |
0.0087 |
0.0013 |
satellite |
6435 |
36 |
31.6395 |
1.7970 |
0.4269 |
7.5566 |
0.0161 |
0.6449 |
1.2578 |
0.9868 |
2.6916 |
1.3697 |
0.0245 |
satimage-2 |
5803 |
36 |
1.2235 |
1.5209 |
0.3705 |
5.6561 |
0.0148 |
0.5529 |
1.0587 |
0.7525 |
2.3935 |
1.1114 |
0.0151 |
shuttle |
49097 |
9 |
7.1511 |
14.3611 |
1.2524 |
59.2131 |
0.0953 |
3.3906 |
9.4958 |
11.1500 |
12.1449 |
44.6830 |
0.0378 |
vertebral |
240 |
6 |
12.5000 |
0.0529 |
0.0444 |
0.0339 |
0.0014 |
0.1786 |
0.0161 |
0.0025 |
0.0446 |
0.0015 |
0.0010 |
vowels |
1456 |
12 |
3.4341 |
0.3380 |
0.0889 |
0.3125 |
0.0044 |
0.2751 |
0.1125 |
0.0367 |
0.9745 |
0.0469 |
0.0023 |
wbc |
378 |
30 |
5.5556 |
0.1014 |
0.0691 |
0.0771 |
0.0063 |
0.2030 |
0.0287 |
0.0078 |
0.0864 |
0.0062 |
0.0035 |