what's the meaning of ROCAUC when using QuantileFilter

[hesamgh77]

>>> from river import anomaly
>>> from river import compose
>>> from river import datasets
>>> from river import metrics
>>> from river import preprocessing

>>> model = anomaly.QuantileFilter(
...     anomaly.OneClassSVM(nu=0.2),
...     q=0.995
... )

>>> auc = metrics.ROCAUC()

>>> for x, y in datasets.CreditCard().take(2500):
...     score = model.score_one(x)
...     is_anomaly = model.classify(score)
...     model = model.learn_one(x)
...     auc = auc.update(y, is_anomaly)

>>> auc
ROCAUC: 74.68%

what's the meaning of ROCAUC when we are using OCSVM with QuantileFilter?
what are thresholds of ROCAUC while we used QuantileFilter with q=0.995?
pleate pay attention to below Code:

>>> auc.thresholds
[-1e-07, 0.1111111111111111, 0.2222222222222222, 0.3333333333333333, 0.4444444444444444, 0.5555555555555556, 0.6666666666666666, 0.7777777777777778, 0.8888888888888888, 1.0000001]

>>> auc.cms
[        False   True   
False       0   2,498  
 True       0       2  ,         
         False   True  
False   2,482     16  
 True       1      1  ,         
         False   True  
False   2,482     16  
 True       1      1  ,         
         False   True  
False   2,482     16  
 True       1      1  ,         
         False   True  
False   2,482     16  
 True       1      1  ,         
         False   True  
False   2,482     16  
 True       1      1  ,         
         False   True  
False   2,482     16  
 True       1      1  ,         
         False   True  
False   2,482     16  
 True       1      1  ,         
         False   True  
False   2,482     16  
 True       1      1  ,         
         False   True  
False   2,498      0  
 True       2      0  ]

Why all of confusion matrices of different thresholds are the same? what's problem? And what's the meaning of ROCAUC: 74.68% in this case

[MaxHalford]

Hey there @hesamgh77. I think you're mixing things up. metrics.ROCAUC and anomaly.QuantileFilter have nothing to do with each other.

The metrics.ROCAUC class measures the ROC AUC score in an online fashion. It does that by maintaining several confusion matrices, each one representing the confusion matrix when the output score is thresholded by p.

The anomaly.QuantileFilter class is here to filter out anomalies, so that the underlying anomaly detector doesn't learn on them.

I hope that helps.

[hesamgh77]

thanks so much dear @MaxHalford

in that case i said what is p and how p apply on anomaly detector and why all confusion matrices are the same?

[MaxHalford]

So here p is just a threshold in the metrics.ROCAUC class. It's generated uniformly in [0, 1]. All the confusion matrices are the same because the model outputs are close to 0 and 1, therefore the confusion is the same whether p=0.3 or p=0.7.

Note that River's ROC AUC metric is appropriate. I would advise storing all the predicitions and scikit-learn's roc_auc_score if you want the real ROC AUC value.

[hesamgh77]

Thanks so much for answering my question dear @MaxHalford

I think there is a bug in that case i said in last message! i tested it with another dataset and the results of all thresholds were the same.

1- I have a question:
what is the philosophy of Filters like QuantileFilter and ThresholdFilter?
I think metrics.ROCAUC considers 10 thresholds and we select just 1 out of 10 with QuantileFilter. is it true?

2- I have another question:
Can we use QuantileFilter with different q such as 0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9 and plot ROC Curve based on results and calculate AUC of ROC? is it true?

3- OCSVM scores are not limited to [0,1] and how does OCSVM work with ROCAUC?
for example: x1 is a data instance and if we give x1 to OCSVM model, the score is (for example) 2.236. and at the end we can sort scores and determine n % of total numbers abnormal. in this case we can not use ROCAUC directly! because of scores are not limited to [0,1]. in this case we should use Filters like QuantileFilter and using ROCAUC as a metric has no meaning! please see this link: https://stackoverflow.com/questions/28022081/how-to-calculate-auc-for-one-class-svm-in-python

[MaxHalford]

what is the philosophy of Filters like QuantileFilter and ThresholdFilter?

The idea is that you don't always want to show a sample to an anomaly detector. Indeed, you might now want to pollute it with anomalies, and instead only train it on normal data.

Can we use QuantileFilter with different q such as 0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9 and plot ROC Curve based on results and calculate AUC of ROC? is it true?

I think you're getting confused. As I said, QuantileFilter and the ROC AUC metric are different concepts. The former is for filtering anomalies, the second is a metric. I'm not sure what you're trying to do.

OCSVM scores are not limited to [0,1] and how does OCSVM work with ROCAUC?
for example: x1 is a data instance and if we give x1 to OCSVM model, the score is (for example) 2.236. and at the end we can sort scores and determine n % of total numbers abnormal. in this case we can not use ROCAUC directly! because of scores are not limited to [0,1]. in this case we should use Filters like QuantileFilter and using ROCAUC as a metric has no meaning! please see this link: https://stackoverflow.com/questions/28022081/how-to-calculate-auc-for-one-class-svm-in-python

This is clearly a limitation of River's ROC AUC: it assumes values are in [0, 1]. The results are likely to be garbage if not.

[hesamgh77]

please see 2 below cases:

1-

model = compose.Pipeline(
     preprocessing.MinMaxScaler(),
         anomaly.HalfSpaceTrees(seed=42)
)

for x, y in datasets.CreditCard().take(2500):
	score = model.score_one(x)
	model = model.learn_one(x)
	auc = auc.update(y, score)

auc = metrics.ROCAUC()
print(auc.cms)

[        False   True   
False       0   2,498  
 True       0       2  ,         
         False   True   
False     250   2,248  
 True       0       2  ,         
         False   True   
False     250   2,248  
 True       0       2  ,         
         False   True   
False     250   2,248  
 True       0       2  ,         
         False   True   
False     258   2,240  
 True       0       2  ,         
         False   True   
False     476   2,022  
 True       0       2  ,         
         False   True   
False   1,147   1,351  
 True       0       2  ,         
         False   True  
False   1,980    518  
 True       0      2  ,         
         False   True  
False   2,454     44  
 True       1      1  ,         
         False   True  
False   2,498      0  
 True       2      0  ]
ROCAUC: 93.94%

2-

model = compose.Pipeline(
     preprocessing.MinMaxScaler(),
      anomaly.QuantileFilter(
         anomaly.HalfSpaceTrees(seed=42),
	 protect_anomaly_detector = False,
         q=0.95
     )
)
for x, y in datasets.CreditCard().take(2500):
	score = model.score_one(x)
	is_anomaly = model['QuantileFilter'].classify(score)
	model = model.learn_one(x)
	auc = auc.update(y, is_anomaly)

print(auc.cms)
print(auc)

[        False   True   
False       0   2,498  
 True       0       2  ,         
         False   True  
False   2,396    102  
 True       1      1  ,         
         False   True  
False   2,396    102  
 True       1      1  ,         
         False   True  
False   2,396    102  
 True       1      1  ,         
         False   True  
False   2,396    102  
 True       1      1  ,         
         False   True  
False   2,396    102  
 True       1      1  ,         
         False   True  
False   2,396    102  
 True       1      1  ,         
         False   True  
False   2,396    102  
 True       1      1  ,         
         False   True  
False   2,396    102  
 True       1      1  ,         
         False   True  
False   2,498      0  
 True       2      0  ]
ROCAUC: 72.96%

why are the Confusion matrices of case 2 (with filter) all the same and Confusion matrices of case 1(without filter) is correct?

[MaxHalford]

why are the Confusion matrices of case 2 (with filter) all the same and Confusion matrices of case 1(without filter) is correct?

In the second case, you're using a boolean input to auc.update. The way ROC AUC works, it's expecting scores between 0 and 1. Therefore, if all the predictions are close to 0 or 1, the confusion matrices can't be trusted. Your first implementation is correct though.

[hesamgh77]

Thanks @MaxHalford.

as i understood, two concepts of ROCAUC and QuantileFilter can't use simultaneously. is it correct?

[MaxHalford]

Indeed. You could, but you would have to use scikit-learn's ROC AUC implementation, not River's.

[hesamgh77]

Thanks so much dear @MaxHalford

[MaxHalford]

No problem 😊