Aligning the domains in cross domain model inversion attack

Ear image analysis is an emerging biometrie application. A method for normalizing ear images and extracting from them a set of measurable features (feature vector) that can be used to identify its owner is proposed. The identification would be made based on the comparison between the feature vector of the input image and all feature vectors of the images in the database we work with. The feature vector is based on the ear contours. One important goal of this paper is to identify the most significant areas in the ear contour for human being identification purpose. Another important contribution of the paper is the combination of active contours techniques and ovoid model ear fitting (used to normalize ear features) and a high accurate invariant approach of internal and external ear contours. Ear geometry is characterized using a set of distances to external and internal contours points. This set of distances, along with six ovoid parameters is considered as the feature vector of the image. To test the method a new ear images database has been created. The proposed method identifies front-parallel views pretty good, even when varying the distance of the individual to the camera or the camera lens.

Color and texture feature vectors of an image are always considered to be an important attribute in content-based image retrieval system. Both of these feature vectors of an image can be combined for the performance enhancement of the content-based image retrieval system. One of the standard ways of extracting color feature from an image is to generate a color histogram. Using Haar wavelet or Daubechies' wavelet the texture feature of an image can be extracted. These two feature vectors and the feature vectors in the database are normalized so that the value of a bin is always between [0,1]. During retrieval, both color and texture feature vectors of query image is combined, weighted and compared with the color and texture feature vectors of each of the database images using Manhattan distance metric. The retrieved result is dependent on the weight given to each of the feature vector. We have done a detailed study of the performance of different combination of weights to color (w/sub c/) and texture (w/sub t/) features on a large database of images. Different combination weights are used in for evaluation and the results shows that texture feature vector weight (W/sub t/) in the range of W/sub c/ /spl plusmn/0.1 to w/sub c/ /spl plusmn/0.2 perform better than the other combinations.

The principal objective of this paper is to develop a face matching method based on facial feature extraction. The first stage to build a robust face matching system is to extract corresponding points between a pair of images. A method based on feature vectors has been used to match images. Since the images illumination, motion, rotation, and scale are different, we have used the SIFT algorithm, which is robust to these variations, for extracting Keypoints. After determining Keypoints for both images and calculating their respective feature vectors, the degree of similarity between two images is evaluated. Besides, the feature vectors of the images are compared with the feature vectors of each reference image to determine the overall similarity between two images. In this paper, we use the SIFT algorithm along with the neural network and the Kepenekci approach and compare the results of these two methods.

Model inversion attacks can reconstruct the training samples of victim deep learning models. The existing efforts heavily rely on auxiliary information of the target samples (prior target information) to achieve their adversarial goals. However, prior target information is hard to obtain in practice. In this paper, we explore the effect of class information in model inversion attacks to reduce the reliance of prior target information. Our contributions on class information exploitation are two-fold. Firstly, we propose a supervised inversion model, Supervised Model Inversion (SMI). The proposed inversion model learns pixel-level features and data-to-class features from the rounded-outputs of the victim model and labeled auxiliary dataset. Secondly, we leverage victim model's rounded-outputs to guide the optimization of reconstructing inversion samples after trained inversion model. Our experimental results show that inversion samples reconstructed by SMI are more visually plausible with more details, comparing to the three representative model inversion attacks. We further perform an extensive study on various auxiliary dataset settings. It is found that the class combination in the auxiliary dataset rather than the number of classes that determines the quality of inversion samples. The ground-truth labels can improve the qualities of inversion samples but not essential to inversion attacks.

This paper proposes a fast and efficient approach for face recognition under non uniform illumination variations. Robust Haar classifiers technique is used for face detection from an image. Since illumination variations lie in low frequency DCT coefficients, illumination variations is removed from detected face by rescaling down an appropriate number of low frequency DCT coefficients while still preserving important facial features. Further, since, important facial features are concentrated in small number of DCT oefficients, face feature vector is generated by discarding high frequency coefficients. K-means clustering is employed to reduce search space complexity. Face recognition is performed by comparing feature vector of test image with feature vector of images in the closest matching cluster using Euclidean distance. Experimental results on Yale database, Caltech database, IMM database and Extended Yale face database B show that the proposed approach improves face recognition rate upto 100% along with significantly reduced search space complexity and low computational cost. Equal error rate (EER) is acquired by plotting false acceptance rate (FAR) and false reject rate (FRR) against different threshold values.

Model Inversion (MI) attacks, which reconstruct the training dataset of neural networks, pose significant privacy concerns in machine learning. Recent MI attacks have managed to reconstruct realistic label-level private data, such as the general appearance of a target person from all training images labeled on him. Beyond label-level privacy, in this paper we show sample-level privacy, the private information of a single target sample, is also important but under-explored in the MI literature due to the limitations of existing evaluation metrics. To address this gap, this study introduces a novel metric tailored for training-sample analysis, namely, the Diversity and Distance Composite Score (DDCS), which evaluates the reconstruction fidelity of each training sample by encompassing various MI attack attributes. This, in turn, enhances the precision of sample-level privacy assessments. Leveraging DDCS as a new evaluative lens, we observe that many training samples remain resilient against even the most advanced MI attack. As such, we further propose a transfer learning framework that augments the generative capabilities of MI attackers through the integration of entropy loss and natural gradient descent. Extensive experiments verify the effectiveness of our framework on improving state-of-the-art MI attacks over various metrics including DDCS, coverage and FID. Finally, we demonstrate that DDCS can also be useful for MI defense, by identifying samples susceptible to MI attacks in an unsupervised manner.

The rapid adoption of deep learning in sensitive domains has brought tremendous benefits. However, this widespread adoption has also given rise to serious vulnerabilities, particularly model inversion (MI) attacks, posing a significant threat to the privacy and integrity of personal data. The increasing prevalence of these attacks in applications such as biometrics, healthcare, and finance has created an urgent need to understand their mechanisms, impacts, and defense methods. This survey aims to fill the gap in the literature by providing a structured and in-depth review of MI attacks and defense strategies. Our contributions include a systematic taxonomy of MI attacks, extensive research on attack techniques and defense mechanisms, and a discussion about the challenges and future research directions in this evolving field. By exploring the technical and ethical implications of MI attacks, this survey aims to offer insights into the impact of AI-powered systems on privacy, security, and trust. In conjunction with this survey, we have developed a comprehensive repository to support research on MI attacks and defenses. The repository includes state-of-the-art research papers, datasets, evaluation metrics, and other resources to meet the needs of both novice and experienced researchers interested in MI attacks and defenses, as well as the broader field of AI security and privacy. The repository will be continuously maintained to ensure its relevance and utility. It is accessible at https://github.com/overgter/Deep-Learning-Model-Inversion-Attacks-and-Defenses.

Model Inversion (MI) attacks, that aim to recover semantically meaningful reconstructions for each target class, have been extensively studied and demonstrated to be successful in the white-box setting. On the other hand, black-box MI attacks demonstrate low performance in terms of both effectiveness, i.e., reconstructing samples which are identifiable as their ground-truth, and efficiency, i.e., time or queries required for completing the attack process. Whether or not effective and efficient black-box MI attacks can be conducted on complex targets, such as Convolutional Neural Networks (CNNs), currently remains unclear. In this paper, we present a feasibility study in regards to the effectiveness and efficiency of MI attacks in the black-box setting. In this context, we introduce Deep-BMI (Deep Black-box Model Inversion), a framework that supports various black-box optimizers for conducting MI attacks on deep CNNs used for image recognition. Deep-BMI’s most efficient optimizer is based on an adaptive hill climbing algorithm, whereas its most effective optimizer is based on an evolutionary algorithm capable of performing an all-class attack and returning a diversity of images in a single run. For assessing the severity of this threat, we utilize all three evaluation approaches found in the literature. In particular, we (a) conduct a user study with human participants, (b) demonstrate our actual reconstructions along with their ground-truth, and (c) use relevant quantitative metrics. Surprisingly, our results suggest that black-box MI attacks, and for complex models, are comparable, in some cases, to those reported so far in the white-box setting.

This paper studies defense mechanisms against model inversion (MI) attacks -- a type of privacy attacks aimed at inferring information about the training data distribution given the access to a target machine learning model. Existing defense mechanisms rely on model-specific heuristics or noise injection. While being able to mitigate attacks, existing methods significantly hinder model performance. There remains a question of how to design a defense mechanism that is applicable to a variety of models and achieves better utility-privacy tradeoff. In this paper, we propose the Mutual Information Regularization based Defense (MID) against MI attacks. The key idea is to limit the information about the model input contained in the prediction, thereby limiting the ability of an adversary to infer the private training attributes from the model prediction. Our defense principle is model-agnostic and we present tractable approximations to the regularizer for linear regression, decision trees, and neural networks, which have been successfully attacked by prior work if not attached with any defenses. We present a formal study of MI attacks by devising a rigorous game-based definition and quantifying the associated information leakage. Our theoretical analysis sheds light on the inefficacy of DP in defending against MI attacks, which has been empirically observed in several prior works. Our experiments demonstrate that MID leads to state-of-the-art performance for a variety of MI attacks, target models and datasets.

In this paper, we investigate approaches to supporting effective and efficient retrieval of image data based on content. We first introduce an effective block-oriented image decomposition structure which can be used to represent image content in image database systems. We then discuss the application of this image data model to content-based image retrieval. Using wavelet transforms to extract image features, significant content features can be extracted from image data through decorrelating the data in their pixel format into frequency domain. Feature vectors of images can then be constructed. Content-based image retrieval is performed by comparing the feature vectors of the query image and the decomposed segments in database images. Our experimental analysis illustrates that the proposed block-oriented image representation offers a novel decomposition structure to be used to facilitate effective and efficient image retrieval.

The field of medical images has been rapidly evolving since the advent of the digital medical information era. However, medical data is susceptible to leaks and hacks during transmission. This paper proposed a robust multi-watermarking algorithm for medical images based on GoogLeNet transfer learning to protect the privacy of patient data during transmission and storage, as well as to increase the resistance to geometric attacks and the capacity of embedded watermarks of watermarking algorithms. First, a pre-trained GoogLeNet network is used in this paper, based on which the parameters of several previous layers of the network are fixed and the network is fine-tuned for the constructed medical dataset, so that the pre-trained network can further learn the deep convolutional features in the medical dataset, and then the trained network is used to extract the stable feature vectors of medical images. Then, a two-dimensional Henon chaos encryption technique, which is more sensitive to initial values, is used to encrypt multiple different types of watermarked private information. Finally, the feature vector of the image is logically operated with the encrypted multiple watermark information, and the obtained key is stored in a third party, thus achieving zero watermark embedding and blind extraction. The experimental results confirm the robustness of the algorithm from the perspective of multiple types of watermarks, while also demonstrating the successful embedding of multiple watermarks for medical images, and show that the algorithm is more resistant to geometric attacks than some conventional watermarking algorithms.

We investigate approaches to support effective and efficient retrieval of image data based on content. We first introduce an effective block-oriented image decomposition structure which can be used to represent image content in image database systems. We then discuss the application of this image data model to content-based image retrieval. Using wavelet transforms to extract image features, significant content features can be extracted from image data through decorrelating the data in their pixel format into the frequency domain. Feature vectors of images can then be constructed. Content-based image retrieval is performed by comparing the feature vectors of the query image and the decomposed segments in database images. Our experimental analysis illustrates that the proposed block-oriented image representation offers a novel decomposition structure to be used to facilitate effective and efficient image retrieval.

Model inversion (MI) attacks, for which effective defense strategies are still lacking, pose significant risks to privacy by reconstructing private training data through access to well-trained classifiers. Addressing this concern, this study introduces TrapNet, designed to defend against advanced MI attacks while maintaining good model utility. TrapNet intentionally injects trapdoors into the classification manifold of the protected target model. In this way, TrapNet can effectively mislead MI attack optimization. Specifically, TrapNet leverages a conditional GAN (cGAN) trained on the private dataset to generate diverse and realistic trapdoor samples. In addition, we propose a graph-matching self-obfuscation strategy and an entropy regularization technique to optimize trapdoor injection while preserving model utility. Compared to the existing defense, TrapNet can provide universal protection to all target classes without access to any auxiliary public data. Extensive experiments on CelebA, VGG-Face, and VGG-Face2 datasets demonstrate TrapNet’s superior performance over existing defenses, including the most advanced NetGuard and BiDO, against state-of-the-art model inversion attacks, i.e., PLG-MI, LOMMA, and Plug&Play.

As machine learning models become integral to security-sensitive applications, concerns over data leakage from adversarial attacks continue to rise. Model Inversion (MI) attacks pose a significant privacy threat by enabling adversaries to reconstruct training data from model outputs. While MI attacks on Artificial Neural Networks (ANNs) have been widely studied, Spiking Neural Networks (SNNs) remain largely unexplored in this context. Due to their event-driven and discrete computations, SNNs introduce fundamental differences in information processing that may offer inherent resistance to such attacks. A critical yet underexplored aspect of this threat lies in black-box settings, where attackers operate through queries without direct access to model parameters or gradients-representing a more realistic adversarial scenario in deployed systems. This work presents the first study of black-box MI attacks on SNNs. We adapt a generative adversarial MI framework to the spiking domain by incorporating rate-based encoding for input transformation and decoding mechanisms for output interpretation. Our results show that SNNs exhibit significantly greater resistance to MI attacks than ANNs, as demonstrated by degraded reconstructions, increased instability in attack convergence, and overall reduced attack effectiveness across multiple evaluation metrics. Further analysis suggests that the discrete and temporally distributed nature of SNN decision boundaries disrupts surrogate modeling, limiting the attacker's ability to approximate the target model.

Model inversion (MI) attacks have raised increasing concerns about privacy, which can reconstruct training data from public models. Indeed, MI attacks can be formalized as an optimization problem that seeks private data in a certain space. Recent MI attacks leverage a generative adversarial network (GAN) as an image prior to narrow the search space, and can successfully reconstruct even the high-dimensional data (e.g., face images). However, these generative MI attacks do not fully exploit the potential capabilities of the target model, still leading to a vague and coupled search space, i.e., different classes of images are coupled in the search space. Besides, the widely used cross-entropy loss in these attacks suffers from gradient vanishing. To address these problems, we propose Pseudo Label-Guided MI (PLG-MI) attack via conditional GAN (cGAN). At first, a top-n selection strategy is proposed to provide pseudo-labels for public data, and use pseudo-labels to guide the training of the cGAN. In this way, the search space is decoupled for different classes of images. Then a max-margin loss is introduced to improve the search process on the subspace of a target class. Extensive experiments demonstrate that our PLG-MI attack significantly improves the attack success rate and visual quality for various datasets and models, notably, 2 ∼ 3× better than state-of-the-art attacks under large distributional shifts. Our code is available at: https://github.com/LetheSec/PLG-MI-Attack.