This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Evaluation

Evaluation of protection measures

    Software-based and hardware-based protection measures can be evaluated by means of different criteria. Some important criteria are

    • Accuracy (proportion of correct predictions on test data)
    • Robustness (accuracy on manipulated data)
    • Privacy (effectiveness of membership inference attacks)
    • Training Time (duration of the model’s training process)

    To interpret an evaluation adequately, it should be compared to the evaluation of the same AI application without any protection measure.

    In the following, the evaluation results of some protection measures on exemplary AI applications are depicted.

    Software Measures

    MNIST Dataset

    Protection MeasureAccuracyRobustnessPrivacyTraining Time
    no98,3 %11,9 %0,4572 s
    DP-SGD94,2 %4,8 %< 0,198 s
    Anomaly Detection98,3 %4,4 %< 0,165 s
    Adversarial Training98,9 %76,8 %0,12286 s

    CIFAR10 Dataset

    Protection MeasureAccuracyRobustnessPrivacyTraining Time
    no81,6 %18,4 %0,31373 s
    DP-SGD63,9 %54,5 %< 0,11061 s
    Anomaly Detection79,4 %17,6 %0,4313 s
    Adversarial Training71,9 %23,1 %22,02930 s

    Hardware Measures

    Protection MeasureDelayPower ConsumptionDevice / Setup
    Modell-Signierung282 ms< 0,01 WJetson Nano / Raspberry Pi 3, Zymkey 4i
    Modell-Signierung12 msmittelHuawei P20 Pro (Android)
    Sensordaten-Attestierung77 ms0,15 WRaspberry Pi 3, NXP SE050 Edge Lock, 3-Axis Accelerometer, Burst-Read (6 Byte, I2C API)
    Sensordaten-Attestierung0,221 msgeringHuawei P20 Pro (Android)
    Verschlüsselung (AES128)2,68 kB/s0,07 - 0,15 WRaspberry Pi 3, NXP SE050 Edge Lock (CBC Mode)
    Verschlüsselung (AES128)2,617 kB/s< 0,01 WRaspberry Pi 3, Zymkey 4i (ECDSA Signature, Mode unknown)
    Verschlüsselung (AES128)4566 kB/s0,07 - 0,14 WOP-TEE, STM32MP1 (CTR Mode)
    Verschlüsselung (AES128)0,095 msgeringHuawei P20 Pro (Android)