Overview

• 1: AI Classes
• 2: Security Goals
• 3: Software Measures
• 4: Hardware Measures
• 5: Evaluation
• 6: Recommendations
• 7: Best Practices

1 - AI Classes

Classification of AI applications

The stated goal of the SENSIBLE-KI project is to secure embedded and mobile AI applications. In order to ensure a standardized security protection, it is necessary to systematize AI applications. The specific needs for protection can then be determined by means of discrete AI classes.

Based on the evaluation of a wide range of AI applications, the following classes were identified in this project. It is a vertical classification which is based on different properties of the AI applications.

The individual protection needs can be determined by categorizing the application with these different levels.

2 - Security Goals

An important step for evaluating the security of systems using Artificial Intelligence (AI)- applications is to define specific security goals for the AI. With these security goals, applications may be rated and their protection needs can be determined.

Similarly to the type of classical security goals in IT security, the following security goals were derived and transferred to the field of AI.

Integrity

Integrity of AI models means the models make the right predictions for a variety of data.

An example would be a spam filter trained on reasonable data. It should still work correctly detecting spam, even if an attacker were to write spam mails containing many elements of a non-spam mail.

Availability

Availability in AI models ensures that the model fulfills the functionality it is used for.

We take a look at the spam filter again. Availability aims less at declining spam (see integrity) but to ensure that all non-spam messages pass.

Confidentiality

Confidentiality describes a state in which the internal properties and confidential information on the trained model remain hidden from a potential attacker.

The attacker should not be able to work out which words, characters or formatting the spam filter uses to detect spam.

Privacy

Privacy in AI applications means the protecting the privacy of training data used to train the AI model. The models should withhold all private information.

A spam filter which was trained with private mails of Alice and Bob should not allow conclusions on what Alice or Bob wrote. Neither due to properties nor due to behaviour.

Assignment of security goals and concrete attacks against AI models

There are known attacks in existence against AI applications models targeting the following security goals.

Literature recommendations

Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete Problems in AI Safety. 2016. url: https://arxiv.org/pdf/1606.06565.

Marco Barreno, Blaine Nelson, Anthony D. Joseph, and J. D. Tygar. “The security of machine learning”. In: Machine Learning 81.2 (2010), pp. 121–148. issn: 1573-0565. doi: 10.1007/s10994-010-5188-5. url: https://link.springer.com/content/pdf/10.1007/s10994-010-5188-5.pdf.

Mark Bedner and Tobias Ackermann, “Schutzziele der IT-Sicherheit,” Datenschutz und Datensicherheit - DuD 34, no. 5 (May 2010): 323–28, doi: 10.1007/s11623-010-0096-1.

Luis Muñoz-González, Battista Biggio, Ambra Demontis, Andrea Paudice, Vasin Wongrassamee, Emil C. Lupu, and Fabio Roli. “Towards Poisoning of Deep Learning Algorithms with Back-gradient Optimization”. In: Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security - AISec ’17. pp. 27–38. isbn: 9781450352024. doi: 10.1145/3128572.3140451.

Nicolas Papernot. A Marauder’s Map of Security and Privacy in Machine Learning.url: http://arxiv.org/pdf/1811.01134v1. 2016

Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z. Berkay Celik, and Ananthram Swami. “Practical Black-Box Attacks against Machine Learning”. In: Proceedings of the 2017 ACM on Asia Conference on Computerand Communications Security. Ed. by Ramesh Karri. ACM, 2017. isbn: 9781450349444. doi: 10.1145/3052973.3053009.

Nicolas Papernot, Patrick McDaniel, Arunesh Sinha, and Michael P.Wellman. “SoK: Security and Privacy in Machine Learning”. In: 3rd IEEE European Symposium on Security and Privacy. Los Alamitos, California: Conference Publishing Services, IEEE Computer Society, 2018. isbn: 9781538642283. doi: 10.1109/eurosp.2018.00035.

3 - Software Measures

In the following, a selection of protection measures which increase the privacy or integrity of machine learning (ML) models (e.g. neural networks) is presented.

First, the general training of neural networks is described since the protective measures are applied within the training. A neural network is a complex mathematical function that is represented by interlinked layers which consist of many parameters. A neural net learns to classify data points of a target data distribution by means of training data. Here, learning refers to the adjustment of the neural net’s internal parameters in an iterative process, the training. Therein, the neural net classifies training data, s.t. the predicted class is propagated back through the network to calculate the direction in which the parameters have to be adjusted in order to get close to the correct class. The adjustment is called gradient descent. The training is terminated when the model’s prediction quality stops improving.

Adversarial Retraining

Adversarial Retraining is a protection measure against so-called adversarial examples (AE). AEs are manipulated inputs for neural networks which aim at causing a wrong prediction. The manipulation is as subtle such that human observers can hardly recognize it as such. In adversarial retraining, AEs are integrated into the training data set of the model to protect so that its adversarial robustness and hence its integrity is improved.

One impressive example of adversarial attacks is the one pixel attack which only changes a single pixel of an image. This minor perturbation is sufficient in many cases to trick the targeted neural net s.t. it predicts a wrong class.

Differential Privacy

Differential Privacy (DP) is a popular metric that allows to measure the influence of particular data points on the result of a data processing. This is equivalent to the privacy loss of the persons to which the data belongs. The privacy loss can be limited by inducing noise into the data processing. Although DP was originally invented in the context of databases, it is applied to privacy-preserving ML in recent years.

In 2006, the streaming service Netflix announced a public competition called Netflix Prize. The goal was to find the best algorithm to predict the movie ratings of certain users. For this purpose, the streaming service provided user data from more than 500,000 users for the training of competing algorithms. Two researchers at the University of Texas developed a linkage attack that maps the published user data with movie ratings from IMDb, a movie rating platform. As a result, many Netflix users have been de-anonymized.

Homomorphic Encryption

Homomorphic Encryption (HE) enables the execution of mathematical operations like addition or multiplication on encrypted data. Results are still present in an encrypted form and are visible only with knowledge of the corresponding decryption key. Note that the same results are obtained as if neither encryption nor decryption would have been applied.

Researchers work on practical applications of HE like, e.g., the improvement of security and transparency of elections. Encrypted votes could be counted automatically—and therefore correctly—by means of the Paillier crypto system for example. This would enable authenticity of elections while also protecting the privacy of voters.

Anomaly Detection

Anomaly detection comprises methods to detect unusual patterns in data. Critical inputs like AEs can be filtered out of training data by means of anomaly detection. This leads to an improved robustness and thus to the integrity of ML models.

Anomaly detection is applied in many areas like for example text and image processing, fraud detection, or medicine.

Evaluation

In order to compare the different protection measures and to evaluate their suitability in the context of resource constrained embedded and mobile AI, several ML experiments were conducted. Therein, neural networks were trained each with one or without any protection mechanism on the MNIST or CIFAR-10 dataset. The MNIST dataset contains 70.000 gray-scaled images of hand-written digits. In contrast, the CIFAR-10 dataset comprises 60,000 colored images of animals or vehicles. For the MNIST dataset, a multi-layer perceptron (MLP) consisting of about 670,000 adjustable parameters was used, while a convolutional neural network (CNN) was selected for the CIFAR-10 dataset that consists of approximately 5.8 mio. parameters. The models were trained over 50 epochs and afterwards deployed on different embedded devices to infer 10,000 test data points. For the inference, the most accurate version of a model over all training epochs was selected to classify the test data.

Comparison of Prediction Quality

The chart shows the course of the test accuracy (prediction quality) for each protection mechanism on each dataset. On each dataset, the accuracy of models trained with differential privacy is the worst. ML models without protection measure similarly achieve a high accuracy on each dataset. Adversarial Training and retraining influences the accuracy positively on the MNIST dataset while they lead to a decreased accuracy on the CIFAR-10 dataset.

Comparison of Training Time

Embedded and mobile devices have lower computational power compared to desktop PCs, rack servers, or computing clusters. Therefore, the training of neural networks is executed on one of the latter machines. Nevertheless, some embedded devices like the Jetsons of Nvidia enable training. The following chart shows the training time of the different ML experiments on two different Jetson devices and on a desktop PC. The training time is expressed in seconds on a logarithmic scale. Trainings without protection measure and with anomaly detection require approximately the same time, while differential privacy leads to significantly higher times. Adversarial training in turn requires a multiple of that training time. Furthermore, there are large differences between different devices. The Jetson Nano is way slower than the Jetson Xavier NX, which in turn is substantially slower than the desktop PC. Note that the installation of many code libraries is very difficult on the Jetson devices, wherefore the protection mechanisms anomaly detection and adversarial (re)training were not applied on them.

Comparison of Inference Time

The trained ML models were deployed on different devices to infer each of the 10,000 test samples of each dataset. The average inference time in milliseconds per data point is illustrated in the following chart. A clear order between the devices PC, Jetson Xavier NX, Jetson Nano, Coral Dev Board Mini, and Neural Compute Stick 2 (NCS2) can be recognized for the simpler MNIST dataset, where the protection measures do not have any influence on the inference time. On the more complex CIFAR-10 dataset, the NCS2 is faster than the Jetson Nano. Further, the inference of models protected by differential privacy is slightly slower than others. This is due to a replaced model layer in the CNN so that differential privacy can be tracked. Unfortunately, no CIFAR-10 experiments could be executed on the Coral Dev Board Mini since the CNN could not have been compiled for its inference chip named Edge TPU.

❮ ❯

Literaturempfehlungen

Goodfellow, I. J., Shlens, J., and Szegedy, C. “Explaining and Harnessing Adversarial Examples” in ICLR (2015).

Abadi, M., Chu, A., Goodfellow, I. J., McMahan, H. B., Mironov, I., Talwar, K., and Zhang, L. “Deep Learning with Differential Privacy” in ACM CCS (2016).

Acar, A., Aksu, H., Uluagac, A. S., and Conti, M. “A Survey on Homomorphic Encryption Schemes: Theory and Implementation” in ACM CSUR (2018).

Chandola, V., Banerjee, A., and Kumar, V. “Anomaly Detection: A Survey” in ACM CSUR (2009).

4 - Hardware Measures

Evaluation Results - Trusted Computing Solutions

General Overview of Identified Mechanisms

The solutions and mechanisms of Trusted Computing were sorted into two groups, which differ in the way they realize the goals of Trusted Computing.

The first group, Trusted Execution Environments and related Technologies, comprises implementations that use preexisting features of the application processor. Opposed to that, the second group consists of Tamper Resistant Hardware and includes hardware modules specifically designed to provide functionality needed to reach the goals of Trusted Computing.

Trusted Execution Environments (TEEs) and Frameworks

Underlying Technologies

The idea of Trusted Execution Environments (TEEs) is to achieve the separation of critical processes and their data from any sort of non-trustworthy code. In the case of a TEE the domain of “non-trustworthy code” even includes the normal operating system (Rich Execution Environment (REE)) and all processes running in it. The TEE has got its own secure operating system and own processes (Trusted Applications (TAs)).

To ensure the separation of the secure domain of the TEE from the non-secure REE, an underlying hardware-based technology is required. Application processors from different manufacturers provide differing technologies to enforce separation which will be described in the following:

TrustZone (TZ) is supported by (some) ARM Cortex-A processors. It supports the separation of resources into a single “Secure World” and a single “Non-Secure World”. The separation is enforced and realized by processor bits signaling the current context, which can be changed by specific instruction set operations that trigger a secure context switch from one world into the other. The resources governed by TrustZone include regions of the main memory, access to certain peripherals and CPU usage. Due to the high prevalence of ARM Cortex-A processors in mobile and embedded systems, TrustZone is supported by many devices in these sectors.

TrustZone-M (TZM) is the equivalent technology for ARM Cortex-M processors. TZ and TZM work mostly in the same way, except that TZM facilitates the context switches via branch instructions which allows a lower latency. TZM is optimized for low power profiles.

Intel SGX is an Instruction Set Extension provided by Intel on (some) of its x86 CPUs. Intel SGX allows to separate the main memory into “Enclaves”. The separation is enforced by encryption of the respective memory region. This technology allows an arbitrary number of secure “Enclaves” but does not secure any other resources than the memory.

AMD SEV/PSP: AMD provides a virtualization technology using the AMD Platform Security Processor (PSP), which is itself an ARM chip supporting TrustZone. The PSP generates and manages the cryptographic key material used to encrypt the memory regions and CPU registers belonging to the secured “Guests”. An arbitrary amount of “Guest” can be secured this way.

RISC-V Physical Memory Protection (PMP) is an additional module, part of RISC-V architecture, that enforces memory access policies via the application processor. These policies can be used to implement a secure domain for a TEE on a RISC-V processor.

Implementations

In the following table, the necessary requirements of the CPU (Platform) have been listed for each solution. In addition, it has been indicated whether the implementation is compliant with the GlobalPlatform TEE API Specification (GP Compliance).

Tamper Resistant Hardware (TRH)

Types

It is hard to cleanly differentiate between the different types of TRH, since there appears to be no universally accepted definition about when which term is applicable. The only type that is officially specified by the Trusted Computing Group (TCG) is the term Trusted Platform Module (TPM).

To avoid unnecessary confusion, the types used in this document to describe and characterize the identified TRH, will be defined. If a specific module did not fit any of these descriptions or not enough information was available, the term used by the manufacturer was adopted.

Trusted Plattform Module (TPM): Provides secure, tamper resistant on-chip storage, means for key generation and management, a random number generator (RNG) - which does not need to be a True Random Number Generator (TRNG) -, and attestation. Specified by the TCG and in ISO/IEC 11889. The currently “active” Versions of the TPM specification are TPM 1.2 and TPM 2.0 which differ mainly regarding their supported cryptographic primitives.

Secure Element (SE): A module or chip resistant to physical manipulations. Provides key generation and management as well as an unspecified set of cryptographic primitives like hash- or encryption algorithms. On chip-storage is optional.

Implementations

For every TRH the respective Type is given (as explained in Section “Types”), as well as – in case of TRH from the mobile sector - the corresponding Platform. Some products can be used with arbitrary platforms and are marked accordingly.

Androids StrongBox Keymaster API was introduced with Android version 9 and provides functions to check whether key material is stored and managed by a StrongBox compliant Secure Element, providing a separate CPU, secure storage, a TRNG¹ and tamper resistance. In case of TRH belonging to mobile devices, compliance with StrongBox is given as well (StrongBox).

Criteria of Evaluation

To evaluate some of the implementations, which seemed promising regarding the goals of the project SENSIBLE-KI, the following criteria of evaluation were chosen:

• Accessibility and Usability:
  The probability that a proposed security mechanism is actually used, is lowered by the direct (financial) and indirect (additional knowledge and time required) cost of this mechanism.

• Supported Features:
  The actual functionality of a specific solution or module is relevant, especially when matching Trusted Computing mechanisms to AI Use-Cases, later on. The evaluated functionality was focused on the supported crypto-primitives and if or how much data can be securely stored by the solution.

• Performance:
  Using any Trusted Computing mechanism will cause additional overhead e.g., from context switches or data transmission to an external module. To gain an overview of the respective performance effects of the solutions, performance measurements were conducted on selected cryptographic functions. The results are relevant when matching Trusted Computing mechanisms to AI Use-Cases, later on.

• Market share:
  The prevalence of a specific implementation is important to rate its relevance in the current economy and was hence – if possible – evaluated.

Evaluated Solutions - Mobile devices

Choice of Devices

Choosing devices by Apple and Samsung, we are covering two of the largest smartphone manufacturers on the market. Because of the inherent differences in both soft- and hardware between iOS and Android devices, comparing phones with each operating system is essential.

Android devices currently cover about 70% of the market in smartphones. Because of this, different Pixel devices by Google were evaluated, since they use an optimized and unedited Android implementation, which does not contain any “bloatware” - meaning third-party apps by the Original Equipment Manufacturer (OEM).

In each case current high-end devices were picked, because these are sure to contain all of the important functionalities and features provided by the OEM at that time.

Note on the Performance Measurements

8 MB Blocks of random data were used as input for the symmetric crypto algorithms.

The API of the iOS Secure Enclave solely supports the usage of asymmetric cryptography. Hence, the performance measurements of the AES algorithm were conducted via ECIES decryption with a secp256r1 key (for AES-128-GCM) and a secp521r1 key (for AES-256-GCM) respectively. It must be noted that the performance of AES-GCM via ECIES under iOS is not comparable to the AES-GCM under Android, since key stored in hardware is only used for initial key-derivation when using the ECIES method.

Evaluations

Results - Overview

Table 12: Results Mobile Devices - Overview

Evaluated Solutions – Embedded Systems

Choice of Devices

OP-TEE was chosen as a representative for TEEs on embedded systems with ARM Cortex-A CPUs, because it is open-source and hence one of the few implementations that allows the cost-free implementation and integration of third-party TAs, which was necessary for our experiments. In addition, OP-TEE implements the GlobalPlatform (GP) TEE API, so implemented TAs for OP-TEE can thus be easily ported to other GP conforming TEEs.

As an alternative TEE, targeting ARM-Cortex M Processors with TrustZone-M, we also chose the commercial implementation Kinibi-M by Trustonic.

The NXP SE050 and the Zymkey 4i by Zymbit both represent Secure Elements and can be used with the Raspberry Pi, which makes them easier to evaluate and ideal for prototyping. The two SEs differ in the available crypto-primitives and the developer target-group. Additionally, a TPM 2.0 implementation by Let’sTrust was picked to cover the group of TPMs. The chosen module too is compatible with the RaspberryPi and implements the TCG TPM 2.0 specification.

Note on Performance Measurements

For the crypto algorithms either the maximum supported buffer size or, in case of no obvious cap, 1MB was used as size for the input buffer. In every case, the time was measured on the lowest possible API level: either right before and after the context switch (TEE) or at the start and end of data transmission to the module (TRH).

Evaluations

Results - Overview

Table 12: Results Embedded Devices - Overview

5 - Evaluation

Software-based and hardware-based protection measures can be evaluated by means of different criteria. Some important criteria are

• Accuracy (proportion of correct predictions on test data)
• Robustness (accuracy on manipulated data)
• Privacy (effectiveness of membership inference attacks)
• Training Time (duration of the model’s training process)

To interpret an evaluation adequately, it should be compared to the evaluation of the same AI application without any protection measure.

In the following, the evaluation results of some protection measures on exemplary AI applications are depicted.

Software Measures

MNIST Dataset

CIFAR10 Dataset

Hardware Measures

6 - Recommendations

Protective Measures for Prototypes and Application Scenarios

1. Introduction

In the following, software-based protective measures and procedures from the field of Trusted Computing are proposed to enhance protection against attacks on AI applications on mobile and embedded devices.

Chapter 2. Considered Measures provides an overview of all the measures considered in this document, along with their respective protective effects. Subsequently, in chapter 3. Protective Measures Based on Application Characteristics, these protective measures are initially suggested generically for applications with specific characteristics. Then, based on these generic suggestions, protective concepts for two concrete applications developed as part of the SENSIBLE-KI project, “SeamlessMe” and “Self-ID,” are presented in Chapter 4. Reference Architectures.

The proposed measures and protective concepts should be considered in addition to general system and application security procedures (e.g., TLS encryption for network communication), which are not within the scope of this collaborative project.

2. Considered Measures

The measures considered here fall into either the realm of software-based protective measures or Trusted Computing procedures. In this section, all considered measures, along with their protective effects and other notable effects, are listed for reference.

2.1 Software-Based Measures

2.2 Trusted Computing

3. Protective Measures Based on Application Characteristics

In the following, application scenarios are differentiated into two abstract groups based on the architectural features of AI systems: network-based applications (such as P2P networks or the classical server-client architecture) and end-node-only applications – architectures limited to an end device. Network-based architectures require regular data exchange or, in the case of OTA updates, model updates over a network. In contrast, end-node-only architectures are restricted to an end device, and communication with another device is limited to information not related to the AI component of the application.

As these two architectural types mainly differ in the need for network-based exchange, general measures for AI applications with specific characteristics are recommended. For a specific application, the intersection of all proposed protective measures for that application can then be determined based on its characteristics.

It should be noted that software-based protective measures for privacy and robustness tend to negatively influence each other. This means that software-based privacy protection adversely affects robustness and vice versa. Therefore, it is necessary to assess on a case-by-case basis which protection goal is more important or whether adequate protection can be achieved solely through hardware-based measures. Furthermore, all protective measures that require the execution of custom code within a Trusted Execution Environment (TEE) are currently not feasible for mobile applications. This limitation arises from the fact that any applications to be executed in a TEE must be compiled together with its code. Therefore, as of today, only a few, mostly manufacturer-specific system apps can execute code within a user’s mobile device TEE.

3.1 Measures Based on the Source of Input Data:

When using trusted input data to improve robustness and overall accuracy, Data Sanitization is recommended to clean or remove erroneous or incomplete data and normalize all data, if necessary.

For untrusted input data sources, the additional use of Adversarial Training is recommended to strengthen system robustness. Additionally, if the necessary resources are available, Anomaly Detection should be applied to detect manipulated training and inference data to protect against Data Poisoning or Adversarial Attacks. This is particularly relevant for systems with significant potential for harm but is desirable in all cases.

When using external sensor data on a system with significant potential for harm, which is transmitted via an untrusted medium (e.g., but not limited to, the network), a Secure Element (SE) can be used to perform data attestation directly when reading from the sensor. In addition, attacks on input data, as described earlier, should be intercepted through Adversarial Training and Anomaly Detection.

3.2 Measures Based on Personalization and Confidentiality

When processing personal data or sensitive data in general, the privacy of the model should be enhanced. This can be achieved, for example, by using Differential Privacy during training and Data Sanitization of sensitive output data.

In cases of significant potential for harm and/or direct personal data involvement and/or high confidentiality of the model, execution of inference within a Trusted Execution Environment (TEE) should be considered. However, the specific implementation of this protective measure may involve some adverse effects (such as increased latency, power consumption, and technical hurdles). Furthermore, it must be ensured that the protected data never leaves the TEE unencrypted. If this cannot be guaranteed, the measure is no longer suitable for privacy protection and is only suitable for enhancing model robustness and confidentiality when executed on the end device.

3.3 Specific Measures for Network-Based Architectures

In a server-client architecture or other network-based architectures, data and/or model updates are transmitted over the Internet. In this architecture, depending on the level of protection needed, it is advisable to take additional measures against manipulations during transmission, in addition to general application security. All measures mentioned here can generally be implemented without using Trusted Computing but may be more vulnerable, although they also result in lower latencies. Therefore, after selecting the measures, it is advisable to reconsider when Trusted Computing is beneficial and appropriate. Additionally, in the case of P2P network architectures, distributing keys, such as for signatures, may be more complex than in a client-server architecture. In P2P architectures, trust must be extended to multiple devices, not just to a single device (the server). Therefore, the corresponding concepts must be taken into account when distributing keys for data integrity and authenticity verification.

In the case of a system with significant damage potential, a device attestation by means of a TEE can enable the receiver (usually the server) to verify the integrity of the target device’s software. Analogously, the data can be attested, e.g., to prove that the data were produced by the latest version of the AI model. This could further improve the application’s robustness.

In the case of transmitting model updates over the network, it should be ensured that the recipient obtained the correct model by applying a model signature. The signature is created by the sender and is verified at the end-node before the model’s execution. This measure should be taken in the course of the general application security, but in the case of significant damage potential, the signature and attestation should be done by means of trusted computing.

In the case of transmitting data with personal reference, an encryption is mandatory. Depending on the sensitivity of the data, the encryption can be enhanced by incorporating a secure element (SE).

4. Reference Architectures

4.1 SeamlessMe

Description
The Machine Learning algorithm used in the “SeamlessMe” demonstrator is a One Class Classification algorithm extended by a Novelty Detection methodology. A generic classifier is loaded onto the mobile device. Subsequently, the model learns locally based on the user data collected on the device (User Specific Classifier). The trained model is then used to generate a Trust Level (confidence score). The calculation of the user’s trust level takes place locally on the smartphone without communication with an external server. Thus, both training and inference occur directly on the end device.

“SeamlessMe” is primarily implemented on smartphones. During the research and development phase, various wearables such as smartwatches were also tested. However, these devices currently have limited functionality and are not yet suitable for the application.

The biometric data processed by “SeamlessMe” is highly sensitive since it can be used for direct person identification. In a medical context, gait profiles, for example, can serve as indicators for certain diseases, further increasing the sensitivity of this data. However, the model architecture does not need protection as it is already publicly known.

Proposed Measures
Based on the associations of protective measures with characteristic properties of the application outlined in Chapter 3. Protective Measures According to Application Characteristics, the following protective measures are to be implemented:

The personally identifiable nature of the processed biometric data leads to an increased privacy protection requirement for the application. However, since it is a mobile application, the use of custom Trusted Applications is not possible, and therefore, execution in a Trusted Execution Environment (TEE) is not an option. In general, on mobile devices, access to Trusted Computing solutions (TEE/SE) present in the device is limited to the functions provided by the Android API. Therefore, to enhance the privacy of the application, only the use of the hardware-backed keystore or StrongBox, or the key management of the Secure Enclave in iOS can be implemented to secure the collected data. The use of Differential Privacy is not suitable in this case because it would result in a significant loss of prediction quality. This is because the training data all come from the same person, and therefore, much stronger noise would be needed for practical privacy protection than usual.

Since training is unsupervised on the mobile device and thus performed with untrusted data, Anomaly Detection on the input data is used to increase the model’s robustness. SeamlessMe already employs the approach of Anomaly Detection as part of the authentication algorithm. The application is designed to recognize gait sequences that do not match the previously trained samples. If this is the case, the person is not considered authenticated. Also, the data collected at that time is not used for model training and therefore no longer poses a threat to the model.

In addition, Data Sanitization of input data is performed to remove unusable data. The application already implicitly implements this measure since only data that corresponds to the walking pattern of a human is used for gait authentication - and thus with the model.

Adversarial Training is not suitable for this application because it is assumed that attackers cannot manipulate sensor data within the end device. If they could, it would imply that they could compromise the entire AI application, making the manipulation of sensor data unnecessary from the attackers’ perspective. Instead, it is assumed that attackers can only manipulate analog input data by impersonating a authorized person for that individual, i.e., mimicking their gait profile using the smartphone. To defend against such an analog attack, a high overall prediction quality of the application must be ensured.

4.2 Self-ID

Description
The “Self-ID” demonstrator application employs a binary classifier (Self-image vs. Non-self-image) trained on a population. This classifier learns to distinguish which class of face images the user is currently viewing based on eye-tracking data. The training process takes place offline and is based on previously collected training data. If personal enrollment is necessary, it will occur on the end device, and the trained model will subsequently be transmitted to the server. The inference takes place online on the server, with the eye-tracking data being transmitted from the client to the server. The server evaluates the result and takes appropriate actions if necessary.

In the initial stage, for accessibility reasons, a desktop client is used via a plugin or extension of an existing video conferencing system. This desktop prototype serves to gather initial experience with the technology. In a later stage, a mobile application will also be implemented on smartphones (Android and iOS). However, as of the current state of technology, eye-tracking technology is not yet available in sufficient quality on these platforms.

The eye tracker data contains biometric information and serves to validate the user’s identity. Therefore, this data is sensitive and falls under the General Data Protection Regulation (GDPR) as a form of personally identifiable data. Additionally, the data may have medical implications, such as indications of disease symptoms. The model ensures the security of the video conferencing system and, accordingly, has increased robustness requirements depending on the confidentiality of the conversation content. In the target system, an attack on the availability of video conferences (generating false positives) or successful identity theft (generating false negatives) can be carried out by manipulating the model or inference data. Furthermore, the model can be misused to identify individuals in systems where this is not permitted according to the intended use. The model may potentially reveal insights into the biometric data with which it was trained. In the finished product, the model, which will be trained with internal data, is the intellectual property of the Bundesdruckerei and should therefore be kept confidential.

Proposed Measures
Based on the associations of protective measures with characteristic properties of the application outlined in Chapter 3. Protective Measures According to Application Characteristics, the following protective measures are to be implemented:

The confidentiality to be achieved for the model at the time of enrollment on the end device can only be strengthened through general application security procedures (e.g., obfuscation) because access to a Trusted Execution Environment (TEE) is not possible on either desktop or mobile devices. The confidentiality of the model on the server is ensured through general system security measures or available server-side Trusted Computing methods. The consideration of these measures is beyond the scope of this document.

For the intended implementation on mobile devices to protect the privacy of personal data, the hardware-backed keystore or StrongBox, or the Secure Enclave, should be used to secure any data stored on the end device and, if applicable, for data encryption before transmission to the server. In the case of the desktop prototype, the use of the available encryption functions of the device/operating system is recommended instead.

The more complex the output of an ML model, the more effective privacy attacks on the model’s training data can be (see Shokri et al.: “Membership Inference Attacks Against Machine Learning Models,” IEEE Symposium on Security and Privacy, 2017). Therefore, the personal data of the application should be additionally protected by data sanitization of the output data if the classifier used not only outputs the class but also the confidence. Furthermore, Differential Privacy can be applied during the training of the classifier to obfuscate information about individual training data.

The increased potential for harm necessitates the application’s robustness against manipulated input data, which can be improved through general system security by implementing client authentication. In the mobile client, this can be reinforced by using the hardware-backed keystore or StrongBox. Additionally, the use of anomaly detection and adversarial training is required since it cannot be ensured that no manipulated input data will be processed. Anomaly detection is used to identify and filter out manipulated input data before the model processes it, while adversarial training enhances the model’s robustness against manipulated input data.

Current methods for privacy protection often degrade robustness, and vice versa. Moreover, methods for privacy and robustness protection typically reduce the prediction quality of the model. Therefore, careful consideration is needed regarding the acceptable trade-offs between privacy, robustness, and overall prediction quality. For this application, it appears best to prioritize robustness over privacy protection within the model. Since the model itself is not executed on an end device, only black-box attacks on privacy are possible, which are significantly weaker than white-box attacks. Therefore, no further measures for privacy protection are proposed here in favor of software-based robustness protection. It is assumed that general system security measures make it difficult for attackers to gain access to the model on the server.

5. Conclusion

In this document, selected measures that were evaluated in the collaborative project SENSIBLE-KI are proposed for general use cases and for two specific applications - the prototypes to be developed in the project. The more specific the use case, the more the selection of measures to be sensibly employed was narrowed down, as both the practicality in the specific application and the interaction between different measures with varying protective effects need to be considered. In particular, hardware-based Trusted Computing methods can only be used to a limited extent in the protection concepts of the prototypes, as access to them is highly restricted in mobile applications. Software-based measures do not face such limitations but are more susceptible to negative interactions, so the use of two negatively influencing methods must be carefully weighed against each other on a case-by-case basis.

7 - Best Practices

Description of Security Goals

This chapter briefly describes the security goals mentioned in this document.

Robustness

The robustness of an AI-Model consists of two factors: The integrity and the availability of the model. Integrity of AI models means the models make the right predictions for a variety of data. This can be impaired by influencing the model-inputs, -outputs and/or parameters. The availability in AI models ensures that the model fulfills the functionality it is used for. If an attackers is targeting the availability of a model he makes the system decline harmless instances in which case the system can’t work properly anymore. If the output of an AI model is integrated into the function of the system this can be seen as a denial-of-service attack.

Privacy

Privacy in AI applications means the protecting the privacy of training data used to train the AI model. The models should withhold all private information. In case of an attack on a model’s privacy the attacker may gain information regarding the - maybe sensitive - training data. This can lead to serious consequences for the privacy of the involved parties. An important aspect of privacy is anonymity and its disctinction to pseudonymity. Anonymity can be defined as protections against general identification and pseudonymity as protection against identification of the name. This implies, that in order to achieve anonymity two picutures should not be correlated with each other. Especially during Training of AI-based identification with biometrics, anonymity is unachiveable and pseudonymity is the main goal. However pseudonymity could be circumvented by correlating multiple databases in so called “Linkage Attacks”.

Confidentiality

Confidentiality describes a state in which the internal properties and confidential information on the trained model remain hidden from a potential attacker. An attack on the confidentiality may allow an attacker to gather sensitive and confidential information regarding the trained AI model, its properties, structure and parameters. With this information, the attacker could be able to steal the intellectual properties, manipulate or attack the privacy of the training data.

Prototypes

Two prototypes were developed during the projects runtime. The application SeamlessMe authenticates a person based on their gait pattern. Self-ID verifies the identity of a user during a video call by analysing eyetracker-data in response to their displayed image. The security requirements of these prototypes were identified and based on the requirements project related security measures were recommended. Recommendations were implemented and evaluated.

Gait-Authentication (SeamlessMe)

The machine learning algorithm used in the “SeamlessMe” demonstrator is a one-class classification, which is enhanced by an outlier detection method (novelty detection). A generic classifier is loaded onto the mobile device. The model then learns locally based on the user data collected on the device (User Specific Classifier). The trained model is then used to generate a trust level (confidence value). The user’s trust level is calculated locally on the smartphone and without communication with an external server. Both the training and the inference therefore take place directly on the end device.

Self identification (Self-ID)

In the application of the demonstrator “Self-ID”, a binary classifier (self-image vs. external image) is trained on a population. Using eye-tracking data, this classifier learns to distinguish which class of facial images the user is currently seeing. The training process takes place in advance - offline - and on the basis of previously collected training data. If personal enrollment is necessary, it takes place on the end device and the trained model is then transferred to the server. The inference runs online on the server. The eye-tracking data is transferred from the client to the server which evaluates the result and initiates appropriate measures if necessary.

Security Measures

Adversarial Training

Adversarial retraining is a protective measure to strengthen the robustness of a machine learning model and a defense technique against so-called Adversarial Examples (AE). AEs are manipulated inputs for neural networks that are intended to lead to an incorrect prediction. The manipulation is so subtle that it is barely recognizable to the human observer. In adversarial retraining, such AEs are included in the training data of the model in order to increase its integrity.

Scenario

The proposed security measure was tested using the Self-ID demonstrator.

In the selected attack scenario, subtle changes to the input data of the model led to incorrect outputs. This reduced the reliability and confidence of the machine learning system. Neural networks were not used.

In the chosen scenario the targeted manipulation of input data was carried out by an “internal attacker” . More complex evasion attacks, such as the generation of AEs by Generative Adversarial Networks (GANs), were not used. For the example attack, a sufficiently strong bias had to be found first so that a false prediction would result.

The designed attack scenario had various prerequisites: Access to the input data

In the chosen example, the type and exact structure of the input data were known, thus relatively small changes of the data were sufficient to manipulate the outcome.

Insight into the predictions

For the example, the degree of change in the input data at which a different label was predicted was recorded. Within the test environment, it was thus possible to trace when the values exceeded a threshold value.

Arbitrary number of predictions

The number of predictions made in the test environment was unlimited.

Experiences

In the selected example, an “internal” attacker who had access to the entire project and the results of the classification was simulated . Without knowledge of the effects of the AEs, these would not have changed the prediction result in a foreseeable way. Furthermore, knowledge of the used features is required, which is not necessarily available to the attacker.

Experience has shown that a change without sufficient knowledge cannot lead to the desired result. Depending on the system architecture, the attacker may not be able to understand whether his AEs have actually triggered a change in the prediction.

Implementation help

Eine Vielzahl an Beispielen und Implementierungsmöglichkeit finden sich in folgender Bibliothek: Adversarial Robustness Toolbox¹

Code-Snippets “Tips & Tricks” A large number of examples and implementation options can be found in the following library: Adversarial Robustness Toolbox¹

Code snippets Manipulate input data

    perturbation_sample = sample_array + perturbation_strength

View effects and compare predictions

    original_prediction = classifier.predict(original_sample_df)
    perturbed_prediction = classifier.predict(perturbation_sample)

Anomaly Detection

Anomaly detection is used to strengthen the robustness of a machine learning system. By using anomaly detection, critical inputs (such as AEs) can be filtered to remove unusually deviant data inputs that could affect the integrity of the model.

Scenario

The proposed security measure was tested using the SeamlessMe demonstrator. The aim is to find the smallest possible boundary that surrounds the normal data points, while at the same time outliers lie outside this boundary. SeamlessMe already uses the anomaly detection procedure indirectly as an authentication algorithm:

SeamlessMe recognizes gait sequences that do not match the previously trained samples. If this is the case, the person is not considered authenticated. Likewise, the data recorded at the time (non-authenticated) is not used to train the model. Only if the algorithm is certain that the data matches the trained person the data is used for training.

A One-Class Support Vector Machine was used to detect outliers. A one-class SVM is a machine learning algorithm that is able to recognize outliers in a data set. In the selected example, only one class of data is available. In the demonstrator, this class is the person carrying the cell phone. The deviating data or outliers are data that point to a different person and therefore belong to a different class.

The One-Class SVM constructs a decision boundary around the “normal” data points so that an area is delimited. All data points outside this area are considered outliers. An illustration of this distinction is shown below:

Figure 1: One-Class SVM ²

Anomaly detection was proposed for Self-ID, but was not implemented because the effort required was too high for a research prototype. Data cleansing was sufficient for data acquisition and anomaly detection was not necessary. The measure should be reconsidered for the final product.

Implementation help

“Tips & Tricks” Used SVM parameters:

• Kernel: Gaussian Radial Basis Function (RBF)

• Kernel functions: “Polynomial” and “Sigmoid” not used

The gamma describes the kernel coefficient. The gamma parameter refers to $ \frac{1}{2*\sigma^2} $ in the following formula: $ K(x_i,x_j) = exp(-\frac{| x_i - x_j |^2}{2*\sigma^2}) $

Code-Snippet (Python: Initialization One-Class-SVM)

    from sklearn.svm import OneClassSVM

    OneClassSVM.__init__(self, kernel='rbf', degree=3, gamma='scale')

Data-Sanitization

Data sanitization of input data is a measure to improve robustness by e.g, cleansing or removing incomplete data in order to bring all data into the appropriate form.

Data sanitization of output data, on the other hand, mitigates privacy attacks. Sensitive output data is filtered before it is emitted. Data is considered sensitive if it allows conclusions about the model’s training data. An example of this in the case of classification would be the confidence values per class. These values can be used by attackers to carry out stronger privacy attacks. Instead of giving the attacker access to every value, only the class with the highest confidence value is emitted.

Scenario

Unusable or incorrect data is filtered. For the Self-ID demonstrator, incorrect images or images containing a blink are therefore removed. The filtering also includes the removal of unnecessary data from the eye tracker data. The data is normalized and scaled. Part of the output data are the confidence values. These values are only shown in the prototype phase in the video client. In the finished product, the strength of confidence in the video client should be represented in a more generic way for the video call participant in question.

With SeamlessMe, activity recognition of the authentication model is performed upstream. Only data that corresponds to the walking pattern of a person is used for gait authentication. This means that data sanitization of the input data is already carried out.

Experiences

Data Sanitization of Input-Data

One experience that has been made involves approximating the correct threshold value for filtering. It should be noted that no training data is removed that still holds enough significance, eventhough a smaller part is invalid. To enable comparability within the Self-ID demonstrator, a minimum number of data points is calculated before an image is presented and a minimum number of data points is calculated after the image is presented. Data points that fall outside this window are removed so that all trials are the same length. The minimum number of data points describes the shortest period of time that does not distort the image. The differences between the recordings result from the latency of the eye tracker.

Data Sanitization of Output-Data

As described in the section “Scenario”, a log of the model results is required to interpret the results of the model creation. Each run with different parameters was logged and stored. The log files describe the changes in the performance metrics caused by the change in input parameters. This logging included the confidence values of all classes. An overview of these values was necessary for the optimization of the models. These values are no longer supplied for the final application, so that no conclusions can be drawn about the model’s training data. Feedback between the user and the application is useful if a confidence level is to be returned. An example of this is that the recognition of classes should be marked within the demonstrators.

Implementation help

“Tips & Tricks”

The research project worked with time series. One type of data cleansing is the normalization of data. In this subject area, the source Normalization and Bias in Time Series Data³ by Aishwarya Asesh has proven to be helpful

Code-Snippet

    def normalize(x):
        minimum = np.min(x)
        maximum = np.max(x)
        y = (x - minimum) / (maximum - minimum)

        return np.array(y)

Differential Privacy

Differential privacy (DP) is a popular metric used to measure the impact of individual data on the outcome of data processing. This also corresponds to the privacy loss of the individuals to whom the data belongs. The loss of privacy is limited by the addition of noise in the data processing. DP originally comes from the field of databases, but has also been used for privacy-preserving ML for several years.

Scenario

Differential Privacy was tested within the Self-ID demonstrator to determine the impact of noise on prediction quality.

A publicly available library was used to examine the effects of differential privacy (DP): Diffprivlib⁴ is a general library for experimenting, investigating and developing applications in the field of differential privacy.

Experiences

The influence of differential privacy led to a reduction in the values of the metrics for all classes considered. Precision, Recall and F1-Score decreased for all classes. The use of differential privacy can be compared to the addition of noise, so a trade-off must be made between prediction quality and protection. Too much noise can affect the accuracy of predictions, while too little noise jeopardizes privacy.

Figure 2: Differences in metrics

The noise of input data is viewed critically. Reference is made to the paper “Jekyll and Hyde: On The Double-Faced Nature of Smart-Phone Sensor Noise Injection “. It observes the introduction of noise into smartphone sensor data used for gait authentication.

The influence of “noise injection on the average F-scores obtained for gait-based authentication” can be seen in the following graph from the paper:

Figure 3: Influence of Differential Privacy

It can be shown how the F-score decreases significantly when more noise is added. High accuracy is important for the functionality of SeamlessMe so that a person can be recognized with a high probability.

Implementation help

“Tips & Tricks” The use of matplot is recommended to illustrate the results, as shown in the diagrams above.

Code-Snippet Example use of the Differential Privacy Library

    from diffprivlib.models import GaussianNB

    clf = GaussianNB()
    clf.fit(X_train, y_train)

    clf.predict(X_test)

Device attestation

In device attestation, the integrity of a device (e.g. a smartphone) is confirmed by a cryptographic signature of a trusted component - e.g. Trusted Execution Environment (TEE), Secure Element (SE). The technical mechanisms used for this differ depending on the implementation, but the general aim is to confirm that the device is in a certain state that is not considered compromised. Device attestatioinferencee of a trusted component is also used for data attestation. This time, however, the signature is calculated using a specific date and attests that it has certain properties (e.g. that the data originates from a specific hardware). This form of attestation is also used to strengthen robustness.

This protective measure was not implemented in the demonstrators of the SENSIBLE-KI joint project, as no data is collected from external sensor nodes or mobile devices that could be attested, and the measure can also only be implemented by specific hardware that could not be used sensibly for the development of the demonstrators. However, the evaluations of the measure as part of AP2 showed an increased latency in data collection, which could make the measure impractical depending on the type of data collection of the ML application.

Model-Signatur

When updating models via the network, the model should be signed to ensure that the receiver has received the correct model. The signature is created by the sender and checked on the end device before execution. For critical applications in particular, it makes sense to perform the attestation and signature using a trustworthy component. This measure strengthens the robustness of the ML system.

This measure was not implemented in the demonstrators of the SENSIBLE-KI joint project, as there are no plans to transfer model updates via the network for either of the two applications. The evaluations of the measure carried out earlier in the project revealed increased latency when starting the application, the length of which depended heavily on the hardware used for signature and verification. It should also be noted that implementing this measure in a TEE would require the implementation and execution of a separate trusted application, which is currently not easily possible on standard mobile devices, for example.

Encryption of Output Data

One measure to improve privacy is to encrypt personalized output data before transmission or storage. It is advisable to use a trusted component for encryption, especially if the data is highly personal, as this offers better protection for the key material used.

Scenario

The encryption was tested within the SeamlessMe demonstrator. Save key material

iOs:

The key material is stored in the Secure Enclave. The Secure Enclave is a secure hardware component. This isolated area in the processor chip serves as security and protection for sensitive data.

Android:

The key material is stored in the hardware-backed keystore. The hardware-backed keystore is a physical hardware component that is responsible for the secure storage of cryptographic keys and sensitive data. Not all devices support the cryptographic algorithms required to secure communication. This will result in additional work during implementation.

Experiences

Correct choice of encryption algorithms:

The choice of a robust encryption algorithm such as AES or RSA depends on the desired level of security.

Key length and key management:

It is recommended to choose an appropriate key length, as a longer key length provides a higher level of security. The keys used should also be managed securely and only be accessible to authorized persons. Regular rotation of the keys is also recommended.

No loss of performance

The relatively small amount of data in the project did not lead to any loss of performance.

Implementation help

The documentation on the Android Keystore System⁵ and the iOS Secure Enclave⁶ is recommended for further research and for current examples.

Inference in a TEE

One way to achieve confidentiality, robustness and privacy at the same time is to execute an ML application within a Trusted Execution Environment as a so-called Trusted Application. In this way, the trained model would not come into contact with potentially malicious third-party applications and would be protected even if the operating system were compromised. Unfortunately, this measure cannot currently be implemented in practice without high technical hurdles and significant performance losses.

Accordingly, the protective measure was not implemented in the demonstrators of the SENSIBLE-KI joint project.