Chaining framework independent preprocessing defences

[bunni90]

Describe the bug
When combining multiple defences of which at least one is framework independent art raises the NotImplemented error "The current combination of preprocessing types is not supported.".

To Reproduce
Steps to reproduce the behavior:

1. Import and instantiate mutliple preprocessing defences (at least one framework independent)

from art.defences.preprocessor import SpatialSmoothingPyTorch #framework specific
from art.defences.preprocessor import FeatureSqueezing #framework independent
spatial_defense = SpatialSmoothingPyTorch(clip_values=[0, 1], channels_first=True)
feature_squeezing = FeatureSqueezing(clip_values=[0, 1])

2. Instantiate a (framework specific) classifier with the preprocessing defences as preproseccing_defences

classifier_def = PyTorchClassifier(
    model=model,
    loss=criterion,
    optimizer=optimizer,
    input_shape=(3, 224, 224),
    nb_classes=1000,
    clip_values=[0, 1],
    preprocessing_defences=[spatial_defense, feature_squeezing]
)

3. Call predict

prediction = classifier_def.predict(x)

4. See error

Expected behavior
Execute the defences on x one after another.

System information (please complete the following information):

• Linux
• Python 3.8.10
• ART 1.9 - 1.12
• TensorFlow / PyTorch

[beat-buesser]

Hi @bunni90 Thank you for your question, the reason for this behavior is the limitation of gradient calculation with Numpy based defenses to one backward step.

[bunni90]

Hi Beat,
thanks for the explanation. In my situation I just need the preprocessing defences in prediction without gradient calcualation. In this case we could just add

        elif not self.all_framework_preprocessing and not fit:
            for preprocess in self.preprocessing_operations:
                    if preprocess.apply_predict:
                        x, y = preprocess(x, y)

to the Pytorch/Tensorflow estimators.
Greetings

[beat-buesser]

Hi @bunni90 Interesting. It looks like this could be useful to chain pre-processing in the black-box scenario?

I think this could work. I'm wondering if we should also include a warning if somebody would try to run the loss_gradient method in this case?

[bunni90]

Hi @beat-buesser ,
I'm not sure where and when exactly you want to show the warning. If you try do the chaining during prediction with a white box attack art calls _apply_preprocessing_gradient and throws NotImplementedError("The current combination of preprocessing types is not supported."). In a framework specific white box attack the error is handled there e.g. in ProjectedGradientDescentPytorch throws NotImplementedError("The framework-specific implementation only supports framework-specific preprocessing.").
If you try the chaining during the fitting art will still throw the NotImplementedError("The current combination of preprocessing types is not supported.").
So the behaviour is not changed except for the case where you chain at least one framework independent preprocessing defense in prediction with a blackbox attack, but never the less we can still implement a warning.
Greetings

[beat-buesser]

Hi @bunni90 I think the warning should be added to art.estimator._apply_preprocessing_gradient and raise an exception if _apply_preprocessing_gradient is called with more than one (two including the default standardization preprocessor). That way the forward passes will be enabled with your proposed changes, but the user will be informed that backward propaagation is not yet supported for more than one Numpy-based preprocessor.