A machine and human reader study on AI diagnosis model safety under attacks of adversarial images

Qianwei Zhou,Shandong Wu,Dooman Arefan,Bronwyn Nair,Yuan Guo,Suzanne Ghannam,Margarita Zuley,Lu Yang,Adrienne Vargo

doi:10.1038/s41467-021-27577-x

Abstract

While active efforts are advancing medical artificial intelligence (AI) model development and clinical translation, safety issues of the AI models emerge, but little research has been done. We perform a study to investigate the behaviors of an AI diagnosis model under adversarial images generated by Generative Adversarial Network (GAN) models and to evaluate the effects on human experts when visually identifying potential adversarial images. Our GAN model makes intentional modifications to the diagnosis-sensitive contents of mammogram images in deep learning-based computer-aided diagnosis (CAD) of breast cancer. In our experiments the adversarial samples fool the AI-CAD model to output a wrong diagnosis on 69.1% of the cases that are initially correctly classified by the AI-CAD model. Five breast imaging radiologists visually identify 29%-71% of the adversarial samples. Our study suggests an imperative need for continuing research on medical AI model’s safety issues and for developing potential defensive solutions against adversarial attacks.

Highlights

While active efforts are advancing medical artificial intelligence (AI) model development and clinical translation, safety issues of the AI models emerge, but little research has been done
The evaluation was based on adversarial attacks where fake mammogram images were synthesized by Generative Adversarial Network (GAN) models to mimic positive-looking and negative-looking images
Our study focused on using GAN models to generate highly plausible and highresolution adversarial images with intentional modifications to insert or remove cancerous tissues

Summary

Introduction

While active efforts are advancing medical artificial intelligence (AI) model development and clinical translation, safety issues of the AI models emerge, but little research has been done. We perform a study to investigate the behaviors of an AI diagnosis model under adversarial images generated by Generative Adversarial Network (GAN) models and to evaluate the effects on human experts when visually identifying potential adversarial images. Our GAN model makes intentional modifications to the diagnosis-sensitive contents of mammogram images in deep learning-based computer-aided diagnosis (CAD) of breast cancer. In the current medical diagnosis context, radiologists routinely read or review images in daily practice, and this process can identify potential adversarial samples Another example is attacks that generate entirely new/different images using standard GANs to replace original images. Several recently reported deep learning-based AI-CAD models have shown promising performances[5,14,15,16,17] Adversarial attacks to such AI-CAD models are emerging as a safety concern[11] to patients, health providers, and legislation. We evaluated the effects of expert radiologists in visually identifying these kinds of adversarial images, without and with an educational intervention

Methods

Results

Conclusion