The attack vector against AI applications is best described using the knowledge and the skills of the attacker

Attacker knowledge

Depending on the attacker’s access to the internal information of the AI application (full, partially, none) it is referred to as “blackbox, greybox or whitebox” scenarios. Attackers could have knowledge and access to any of the following aspects:

1. the AI application’s training data
2. the training algorithm
3. the AI application’s trained model and its parameters

An attacker with access to all three aspects is called an attacker with “perfect knowledge”. Observing such attackers allows making worst case assessments for the security of AI systems.

However, in practice such attackers are not realistic. Attackers with “limited knowledge” are more realistic.

Attacker capabilities

The capabilities of an attacker can be the following, depending on the influence he has on the trained model und it’s training data

Integrity in AI applications can be impaired on three different levels.

1. Data level: targeted corruption of data the model should predict
2. Model level: targeted manipulation of the model’s behaviour or the model’s parameter.
3. Output/Object level: targeted manipulation of reactions from the AI system towards specific model predictions.

Data level

On the data level, the model’s inputs may be altered specifically to produce incorrect predictions.

Model level

On the model level, the model’s parameters may be altered during or after training, producing incorrect predictions.

Object level

On the object level, the model’s outputs may be manipulated in order to make the surrounding system react incorrectly to the input.

Integrity is also linked to confidentiality and controllability. In case of confidentiality, the attacker may achieve invalid system states due to nonsensical inputs. On the other hand the controllability is compromised if the attacker teaches the system unintended behaviour on the model level.

If an attackers is targeting the availability of a model he makes the system decline harmless instances in which case the system can't work properly anymore. If the output of an AI model is integrated into the function of the system this can be seen as a denial-of-service attack.

An attack on the confidentiality may allow an attacker to gather sensitive and confidential information regarding the trained AI model, its properties, structure and parameters. With this information, the attacker could be able to steal the intellectual properties, manipulate or attack the privacy of the training data.

In case of an attack on a model’s privacy the attacker may gain information regarding the - maybe sensitive - training data. This can lead to serious consequences for the privacy of the involved parties. An important aspect of privacy is anonymity and the disctinction to pseudonymity. Anonymity can be defined as protections against general identification and pseudonymity as protection against identification of the name.

Adversarial attacks

For an adversial attack, the attacker specifically manipulates a data point making the trained AI model predict said data point incorrectly. This can be a targeted change of a pixel in an image so that the object in the picture can’t be recognised anymore.

Data Poisoning

For data poisoning, the attacker may manipulate training data while training. Therefore the quality of the model’s predictions can be manipulated. For example, an attacker can use this to bypass a spam filter. When a model was trained to associate certain words with non-spam messages an attacker can use these words to trick the model into not recognising spam.

Model Stealing, Model Extraction

Model stealing is used to copy or steal a trained model from its original owner.

Model extraction is a special form of model stealing in which a model is stolen via blackbox access. This means an attacker only has access to an interface of the model but the received predictions are sufficient to train a substitute which performs just like the original.

Model Inversion

Model inversion is all about gaining information about the training data of a trained AI model. More specifically, this means an average representation of the training data (of a certrain class) may be recovered.

Membership Inference

Membership Inference attacks focus on the question if a given data point was used in training of the model. This may result in a privacy problem. For example if we have a classificator that proposes treatment methods to cancer patient, we can assume it was trained on data of cancer patients. The information of a specific data point being used in training translates to the information that the associated person has cancer.

Attribute Inference

For attribute inference we use public available characteristics of a data point which was used to train a model and access to this model to gather information about all sensitive and private characteristics of this data point.

Property Inference

For property inference we receive properties of the whole training set of an AI model, for example the distribution of the data. This kind of information can lead zu privacy problems especially for minorities in the distribution.