White-box Attacks Research Articles

Query-Efficient Adversarial Attack With Low Perturbation Against End-to-End Speech Recognition Systems

With the widespread use of automated speech recognition (ASR) systems in modern consumer devices, attack against ASR systems have become an attractive topic in recent years. Although related white-box attack methods have achieved remarkable success in fooling neural networks, they rely heavily on obtaining full access to the details of the target models. Due to the lack of prior knowledge of the victim model and the inefficiency in utilizing query results, most of the existing black-box attack methods for ASR systems are query-intensive. In this paper, we propose a new black-box attack called the Monte Carlo gradient sign attack (MGSA) to generate adversarial audio samples with substantially fewer queries. It updates an original sample based on the elements obtained by a Monte Carlo tree search. We attribute its high query efficiency to the effective utilization of the dominant gradient phenomenon, which refers to the fact that only a few elements of each origin sample have significant effect on the output of ASR systems. Extensive experiments are performed to evaluate the efficiency of MGSA and the stealthiness of the generated adversarial examples on the DeepSpeech system. The experimental results show that MGSA achieves 98% and 99% attack success rates on the LibriSpeech and Mozilla Common Voice datasets, respectively. Compared with the state-of-the-art methods, the average number of queries is reduced by 27% and the signal-to-noise ratio is increased by 31%.

Transformer Based Defense GAN Against Palm-Vein Adversarial Attacks

Vein biometrics is a high security and privacy preserving identification technology that has attracted increasing attention over the last decade. Deep neural networks (DNNs), such as convolutional neural networks (CNN), have shown strong capabilities for robust feature representation, and have achieved, as a result, state-of-the-art performance on various vision tasks. Inspired by their success, deep learning models have been widely investigated for vein recognition and have shown significant improvement of identification accuracy compared to handcrafted models. Existing deep learning models, however, are vulnerable to adversarial perturbation attacks, where thoughtfully crafted small perturbations can cause misclassification of legitimate images, degrading, thereby, the efficiency of vein recognition systems. To address this problem, we propose, in this paper, VeinGuard, a novel defense framework to defend deep learning classifiers against adversarial palm-vein image attacks, composed of a local transformer-based GAN and a purifier. VeinGuard comprises two components: a local transformer-based GAN (LTGAN) that learns the distribution of unperturbed vein images and generates high-quality palm-vein images, and a purifier consisting of a trainable residual network and of a pre-trained generator from LTGAN that automatically removes a wide variety of adversarial perturbations. The resulting clean images are fed to vein classifiers for identification, thereby avoiding adversarial attacks. We evaluate VeinGuard on three public vein datasets in terms of white-box attacks, black-box attacks, ablation experiments, and computation time. The experimental results show that VeinGuard allows filtering the perturbations and enables the classifiers to achieve state-of-the-art recognition results for different adversarial attacks.

One-Variable Attack on the Industrial Fault Classification System and Its Defense

Recently developed fault classification methods for industrial processes are mainly data-driven. Notably, models based on deep neural networks have significantly improved fault classification accuracy owing to the inclusion of a large number of data patterns. However, these data-driven models are vulnerable to adversarial attacks; thus, small perturbations on the samples can cause the models to provide incorrect fault predictions. Several recent studies have demonstrated the vulnerability of machine learning methods and the existence of adversarial samples. This paper proposes a black-box attack method with an extreme constraint for a safe-critical industrial fault classification system: Only one variable can be perturbed to craft adversarial samples. Moreover, to hide the adversarial samples in the visualization space, a Jacobian matrix is used to guide the perturbed variable selection, making the adversarial samples in the dimensional reduction space invisible to the human eye. Using the one-variable attack (OVA) method, we explore the vulnerability of industrial variables and fault types, which can help understand the geometric characteristics of fault classification systems. Based on the attack method, a corresponding adversarial training defense method is also proposed, which efficiently defends against an OVA and improves the prediction accuracy of the classifiers. In experiments, the proposed method was tested on two datasets from the Tennessee–Eastman process (TEP) and Steel Plates (SP). We explore the vulnerability and correlation within variables and faults and verify the effectiveness of OVAs and defenses for various classifiers and datasets. For industrial fault classification systems, the attack success rate of our method is close to (on TEP) or even higher than (on SP) the current most effective first-order white-box attack method, which requires perturbation of all variables.

One4All: Manipulate one agent to poison the cooperative multi-agent reinforcement learning

Reinforcement Learning (RL) has achieved a plenty of breakthroughs in the past decade. Notably, existing studies have shown that RL is suffered from poisoning attack, which results in failure or even catastrophe in decision processes. However, these studies almost focus on single-agent RL setting, the cooperative Multi-Agent Reinforcement Learning (c-MARL) setting is less explored, which is a generalization of the single-agent RL setting and has achieved great success in many areas. As a sub-field of RL setting, c-MARL also faces some security issues, e.g., poisoning attack.In this paper, we introduce two novel poisoning attack techniques, i.e., State Noise Poisoning Attack (SNPA) and Target Action Poisoning Attack (TAPA), to attack the c-MARL setting stealthily and efficiently, which achieves the goal that poisoning the c-MARL setting while only manipulate one agent. The first attack technique, termed SNPA, a black-box attack method, modifies the state observation data of one agent, which results in the c-MARL setting performance degradation. The second attack technique, termed TAPA, a white-box attack method, injects a backdoor and triggers the target action by manipulating the action and reward function of one agent. The extensive experiments are conducted in two popular c-MARL games, i.e., MCSPT and MCHT. The experiment results show that the presented novel poisoning attacks, SNPA and TAPA, are effective on c-MARL scenarios. Specifically, the total reward is reduced by 1/3 and the winning rate of team drops down to 0 under the two proposed attacks. Furthermore, the TAPA experiments verify that the victim agent executes the target action once the backdoor is triggered. It is worth noting that the trigger rate of target action rises up to 99.31% and 99.01% in two c-MARL games.

Edge enhancement improves adversarial robustness in image classification

Imperceptible adversarial examples are capable of deceiving the deep neural networks with high confidence. Recent studies show that it is particularly effective to control the attack space to low-frequency components of the image on the basis of adversarial training. However, those methods face the problem of losing valuable knowledge, especially shape information, which is vital for classification and robustness. To alleviate this issue, we propose a new method based on edge detection, named Edge Enhancement (EE), which can explicitly make up for the missing shape information in frequency constraints and further enhance the adversarial robustness. Specifically, we first employ a traditional edge detection algorithm called Canny to obtain shape information due to its simplicity and intrinsic robustness. Then, we augment the low-frequency space via obtained shape features, with the weighting operation carried on. This operation can be regarded as an emphasis on shape information, which could mitigate the texture bias of deep neural networks, thereby further serving the robustness. Finally, we feed the augmented features into the deep neural network. It is worth noting that these modules are optimized along with the deep neural network, which enables an end-to-end training fashion. Experimental results show that our proposed model can significantly improve adversarial robustness over the state-of-the-art methods on three benchmark datasets, including MNIST, Tiny ImageNet, and particularly ImageNet. For example, our method achieves 51.66% accuracy on ImageNet under 10-iteration targeted PGD white-box attack where the prior art has 36.94% accuracy. Code is available at https://github.com/Aiqz/Edge-Enhancement.

Improving transferability of adversarial examples with powerful affine-shear transformation attack

Image classification models based on deep neural networks have made great improvements on various tasks, but they are still vulnerable to adversarial examples that could increase the possibility of misclassification. Various methods are proposed to generate adversarial examples under white-box attack circumstances that have achieved a high success rate. However, most existing adversarial attacks only achieve poor transferability when attacking other unknown models with the black-box scenario settings. In this paper, we propose a new method that generates adversarial examples based on affine-shear transformation from the perspective of deep model input layers and maximizes the loss function during each iteration. This method could improve the transferability and the input diversity of adversarial examples, and we also optimize the above adversarial examples generation process with Nesterov accelerated gradient. Extensive experiments on ImageNet Dataset indicate that our proposed method could exhibit higher transferability and achieve higher attack success rates on both single model settings and ensemble-model settings. It can also combine with other gradient-based methods and image transformation-based methods to further build more powerful attacks.

Formalizing and Estimating Distribution Inference Risks

Distribution inference, sometimes called property inference, infers statistical properties about a training set from access to a model trained on that data. Distribution inference attacks can pose serious risks when models are trained on private data, but are difficult to distinguish from the intrinsic purpose of statistical machine learning—namely, to produce models that capture statistical properties about a distribution. Motivated by Yeom et al.’s membership inference framework, we propose a formal definition of distribution inference attacks general enough to describe a broad class of attacks distinguishing between possible training distributions. We show how our definition captures previous ratio-based inference attacks as well as new kinds of attack including revealing the average node degree or clustering coefficient of training graphs. To understand distribution inference risks, we introduce a metric that quantifies observed leakage by relating it to the leakage that would occur if samples from the training distribution were provided directly to the adversary. We report on a series of experiments across a range of different distributions using both novel black-box attacks and improved versions of the state-of-the-art white-box attacks. Our results show that inexpensive attacks are often as effective as expensive meta-classifier attacks, and that there are surprising asymmetries in the effectiveness of attacks.

On Noise Stability and Robustness of Adversarially Trained Networks on NVM Crossbars

Applications based on deep neural networks (DNNs) have grown exponentially in the past decade. To match their increasing computational needs, several nonvolatile memory (NVM) crossbar-based accelerators have been proposed. Recently, researchers have shown that apart from improved energy efficiency and performance, such approximate hardware also possess intrinsic robustness for defense against adversarial attacks. Prior works have focused on quantifying this intrinsic robustness for vanilla networks, that is DNNs trained on unperturbed inputs. However, adversarial training of DNNs, i.e., training with adversarially perturbed images, is the benchmark technique for robustness, and sole reliance on intrinsic robustness of the hardware may not be sufficient. In this work, we explore the design of robust DNNs through the amalgamation of adversarial training and the intrinsic robustness offered by NVM crossbar-based analog hardware. First, we study the noise stability of such networks on unperturbed inputs and observe that internal activations of adversarially trained networks have lower signal-to-noise ratio (SNR), and are sensitive to noise compared to vanilla networks. As a result, they suffer significantly higher performance degradation due to the approximate computations on analog hardware; on an average <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$2\times $ </tex-math></inline-formula> accuracy drop. Noise stability analyses clearly show the instability of adversarially trained DNNs. On the other hand, for adversarial images generated using Square Black Box attacks, ResNet-10/20 adversarially trained on CIFAR-10/100 display a robustness improvement of 20%–30% under high <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\epsilon _{\mathrm{ attack}}$ </tex-math></inline-formula> (degree of input perturbation). For adversarial images generated using projected-gradient-descent (PGD) White-Box attacks, the adversarially trained DNNs present a 5%–10% gain in robust accuracy due to the underlying NVM crossbar when <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\epsilon _{\mathrm{ attack}}$ </tex-math></inline-formula> is greater than the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">epsilon</i> of the adversarial training ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\epsilon _{\mathrm{ train}}$ </tex-math></inline-formula> ). Our results indicate that implementing adversarially trained networks on analog hardware requires careful calibration between hardware nonidealities and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\epsilon _{\mathrm{ train}}$ </tex-math></inline-formula> to achieve optimum robustness and performance.

An Approximated Gradient Sign Method Using Differential Evolution for Black-Box Adversarial Attack

Recent studies show that deep neural networks are vulnerable to adversarial attacks in the form of subtle perturbations to the input image, which leads the model to output wrong prediction. Such an attack can easily succeed by the existing white-box attack methods, where the perturbation is calculated based on the gradient of the target network. Unfortunately, the gradient is often unavailable in the real-world scenarios, which makes the black-box adversarial attack problems practical and challenging. In fact, they can be formulated as high-dimensional black-box optimization problems at the pixel level. Although evolutionary algorithms are well known for solving black-box optimization problems, they cannot efficiently deal with the high-dimensional decision space. Therefore, we propose an approximated gradient sign method using differential evolution (DE) for solving black-box adversarial attack problems. Unlike most existing methods, it is novel that the proposed method searches the gradient sign rather than the perturbation by a DE algorithm. Also, we transform the pixel-based decision space into a dimension-reduced decision space by combining the pixel differences from the input image to neighbor images, and two different techniques for selecting neighbor images are introduced to build the transferred decision space. In addition, six variants of the proposed method are designed according to the different neighborhood selection and optimization search strategies. Finally, the performance of the proposed method is compared with a number of the state-of-the-art adversarial attack algorithms on CIFAR-10 and ImageNet datasets. The experimental results suggest that the proposed method shows superior performance for solving black-box adversarial attack problems, especially nontargeted attack problems.

Adversarial Deep Learning for Indoor Localization With Channel State Information Tensors

Fingerprinting-based indoor localization has been a research focus for GPS denied areas. The development of neural networks has greatly promoted its application in indoor localization systems. However, recent studies showed that the machine learning models, including state-of-the-art neural networks, are vulnerable to adversarial examples, and thus, neural network-based indoor localization systems are also under the threat of adversarial attacks. To investigate the effect of adversarial attacks on indoor localization systems and to make such systems resilient to adversarial attacks, we propose AdvLoc, an adversarial deep learning for indoor localization system. With the proposed AdvLoc system, the effect of adversarial attacks on indoor localization is studied under six types of adversarial attack methods in both black-box attack and white-box attack scenarios. Furthermore, adversarial training is utilized in offline training of the proposed AdvLoc system, which is effective against first-order adversarial attacks. The proposed AdvLoc system is implemented with commodity WiFi devices and evaluated with extensive experiments in two representative indoor environments. The experimental results verify the robustness of the proposed system against first-order adversarial attacks in representative indoor environments.

Unauthorized AI cannot recognize me: Reversible adversarial example

In this study, we propose a new methodology to control how user’s data is recognized and used by AI via exploiting the properties of adversarial examples. For this purpose, we propose reversible adversarial example (RAE), a new type of adversarial example. A remarkable feature of RAE is that the image can be correctly recognized and used by the AI model specified by the user because the authorized AI can recover the original image from the RAE exactly by eliminating adversarial perturbation. On the other hand, other unauthorized AI models cannot recognize it correctly because it functions as an adversarial example. Moreover, RAE can be considered as one type of encryption to computer vision since reversibility guarantees the decryption. To realize RAE, we combine three technologies, adversarial example, reversible data hiding for exact recovery of adversarial perturbation, and encryption for selective control of AIs who can remove adversarial perturbation. Experimental results show that the proposed method can achieve comparable attack ability with the corresponding adversarial attack method and similar visual quality with the original image, including white-box attacks and black-box attacks.

Generating Adversarial Examples With Shadow Model

The reduction in the number of queries to the object model is a hot topic in the current research of black-box adversarial attack methods. To solve this problem, in this article, we propose generating adversarial examples with shadow model (GASM) that shifts the number of queries to the object model to the shadow model. The method first determines the shadow model based on the robustness and transferability of classifiers and fine-tunes the decision boundary of the shadow model by constructing adversarial datasets. Second, accesses the shadow model and constructs adversarial examples by maximizing the output probability of the targeted class (any class other than the current one) to modify the image gradient information. Finally, the results show that GASM has the strongest transferability and outperforms white-box attacks when AlexNet (MNIST), VGG-19 (CIFAR10), and MobileNet v2 (Tiny ImageNet) are selected as shadow models.

Network Traffic Obfuscation against Traffic Classification

In recent years, tremendous progress has been made in network traffic classification and its use has become ubiquitous in many emerging applications, such as Internet censorship in many countries and ISP traffic engineering. However, the traffic analysis of intermediaries brings the risk of privacy disclosure to users. This paper presents a network traffic obfuscation technology to resist traffic classification. It deceives the machine learning and deep learning models by generating adversarial samples. The adversarial samples generation algorithm includes a white-box attack algorithm based on fuzzy strategy and a black-box attack algorithm based on smote data enhancement. Experiments based on the ISCXTor2016 public data set show that the MIM algorithm has the best performance in white-box attacks, and the obfuscation success rate of DNN and LSTM models is 90%. In the black-box attack, the obfuscation effect of LSTM is the best, while CNN has stronger robustness.

Diminishing-feature attack: The adversarial infiltration on visual tracking

Deeply understanding machine learning is a challenging task because its backbone technology is still obscure. The adversarial attack is established toward the vulnerability of deep models and can be utilized for understanding deep neural networks and enhancing model robustness. Adversarial attacks on tracking models can generate imperceptible noise to an input and consequently increase the cumulative error of tracking displacement. In general, existing adversarial visual-attacking models are white-box attacks. Although it successfully deceives the baseline tracker, the training model’s attacking effect is insufficiently transferred to blind trackers. Here, we present a new approach, the diminishing-feature attack, that generates a subtle perturbation into the input frame. The malicious noise in the proposed model is generated to disturb the feature heatmap to distract the classification score and annihilate bounding-box prediction. The adversarial noise generator is solely trained from SiamRPN++ ResNet features of template frames without search frames or other parameters. Our method requires fewer computing resources than other visual tracking attackers when malfunctioning visual trackers. To validate the proposed model, we benchmark the adversarial input frame with the tracker on four tracking datasets: OTB100, VOT2018, LaSOT and UAV123. Our attacking method achieves a high performance drop that is comparable to the other existing adversarial attackers in visual tracking, yet our adversarial noise is lower than the aforementioned attackers. Furthermore, the transferability attacking of our algorithm is excellent on state-of-the-art visual tracking, for example, SiamRPN, DaSiam and DiMP.

Adversarial Analysis of ML-Based Anomaly Detection in Multi-Layer Network Automation

The fast development of multi-layer packet-over-optical networks has made network monitoring and troubleshooting increasingly complicated. This has stimulated people to combine machine learning (ML) and software-defined networking (SDN) to realize multi-layer network automation. Despite its initial successes, the vulnerabilities of multi-layer network automation have not been fully explored. This work studies how to mislead the ML-based classifiers for anomaly detection. Specifically, we design two adversarial-sample-based attack schemes based on the white-box attack (WBA) and black-box attack (BBA) strategies, respectively, to eavesdrop and tamper legitimate telemetry data samples and generate adversarial samples adaptively, for disturbing ML-based classifiers and in turn misleading network automation to make incorrect decisions. Compared with WBA, BBA makes the attack scheme more practical by minimizing the dependency on pre-knowledge of the target ML-based classifiers. Considering different types of ML-based classifiers, we build a real-world packet-over-optical testbed and leverage the telemetry samples collected in it to demonstrate that our proposed BBA scheme can interact with the network quietly to train itself, generate well-crafted adversarial samples to tamper legitimate telemetry samples in the hard-to-detect way, and mislead ML-based classifiers in the network automation system to severely affect their performance on anomaly detection.

Defending against Adversarial Attacks in Deep Learning with Robust Auxiliary Classifiers Utilizing Bit-plane Slicing

Deep Neural Networks (DNNs) have been widely used in variety of fields with great success. However, recent research indicates that DNNs are susceptible to adversarial attacks, which can easily fool the well-trained DNN-based classifiers without being detected by human eyes. In this article, we propose to integrate the target DNN model with our robust bit-plane classifiers to defend against adversarial attacks. The bit-plane classifiers take bit-planes of input images for convolution, which is motivated by our observation that successful attacks aim to generate imperceptible perturbations, and they mainly affect the low-order bits of pixels in clean images when adding the perturbations. We also propose two metrics, bit-plane perturbation rate and channel modification rate, to further explain the robustness of bit-plane classifiers. We discuss potential adaptive attack and find that our defense can be effective as long as the adversarial examples are qualified. We conduct experiments on dataset CIFAR-10 and GTSRB under white-box attack and black-box attack. The results show that our defense method can effectively increase the average model accuracy from 16.23% to 83.53% under white-box attack and from 40.65% to 88.14% under black-box attack on CIFAR-10 without sacrificing the accuracy of clean images.

DI-AA: An interpretable white-box attack for fooling deep neural networks

White-box adversarial example (AE) attacks on deep neural networks (DNNs) have a more powerful destructive capacity than black-box attacks using AE strategies. However, few studies have been conducted on the generation of low-perturbation adversarial examples from the interpretability perspective. Specifically, adversaries who conducted attacks lacked interpretation from the point of view of DNNs, and the perturbation was not further considered. To address these, we propose an interpretable white-box AE attack approach, DI-AA, which not only explores the application of the interpretable method of deep Taylor decomposition in selecting the most contributing features but also adopts the Lagrangian relaxation optimization of the logit output and Lp norm to make the perturbation more unnoticeable. We compare DI-AA with eight baseline attacks on four representative datasets. Experimental results reveal that our approach can (1) attack nonrobust models with low perturbation, where the perturbation is closer to or lower than that of the state-of-the-art white-box AE attacks; (2) evade the detection of the adversarial-training robust models with the highest success rate; (3) be flexible in the degree of AE generation saturation. Additionally, the AE generated by DI-AA can reduce the accuracy of the robust black-box models by 16–31 % in the black-box manner.

Regularization Meets Enhanced Multi-Stage Fusion Features: Making CNN More Robust against White-Box Adversarial Attacks.

Regularization has become an important method in adversarial defense. However, the existing regularization-based defense methods do not discuss which features in convolutional neural networks (CNN) are more suitable for regularization. Thus, in this paper, we propose a multi-stage feature fusion network with a feature regularization operation, which is called Enhanced Multi-Stage Feature Fusion Network (EMSF2Net). EMSF2Net mainly combines three parts: multi-stage feature enhancement (MSFE), multi-stage feature fusion (MSF2), and regularization. Specifically, MSFE aims to obtain enhanced and expressive features in each stage by multiplying the features of each channel; MSF2 aims to fuse the enhanced features of different stages to further enrich the information of the feature, and the regularization part can regularize the fused and original features during the training process. EMSF2Net has proved that if the regularization term of the enhanced multi-stage feature is added, the adversarial robustness of CNN will be significantly improved. The experimental results on extensive white-box attacks on the CIFAR-10 dataset illustrate the robustness and effectiveness of the proposed method.

Sparse-RS: A Versatile Framework for Query-Efficient Sparse Black-Box Adversarial Attacks

We propose a versatile framework based on random search, Sparse-RS, for score-based sparse targeted and untargeted attacks in the black-box setting. Sparse-RS does not rely on substitute models and achieves state-of-the-art success rate and query efficiency for multiple sparse attack models: L0-bounded perturbations, adversarial patches, and adversarial frames. The L0-version of untargeted Sparse-RS outperforms all black-box and even all white-box attacks for different models on MNIST, CIFAR-10, and ImageNet. Moreover, our untargeted Sparse-RS achieves very high success rates even for the challenging settings of 20x20 adversarial patches and 2-pixel wide adversarial frames for 224x224 images. Finally, we show that Sparse-RS can be applied to generate targeted universal adversarial patches where it significantly outperforms the existing approaches. Our code is available at https://github.com/fra31/sparse-rs.

Transferability analysis of adversarial attacks on gender classification to face recognition: Fixed and variable attack perturbation

Most deep learning-based image classification models are vulnerable to adversarial attacks that introduce imperceptible changes to the input images for the purpose of model misclassification. It has been demonstrated that these attacks, targeting a specific model, are transferable among models performing the same task. However, models performing different tasks but sharing the same input space and model architecture were never considered in the transferability scenarios presented in the literature. In this paper, this phenomenon was analysed in the context of VGG16-based and ResNet50-based biometric classifiers. The authors investigate the impact of two white-box attacks on a gender classifier and contrast a defence method as a countermeasure. Then, using adversarial images generated by the attacks, a pre-trained face recognition classifier is attacked in a black-box fashion. Two verification comparison settings are employed, in which images perturbed with the same and different magnitude of the perturbation are compared. The authors’ results indicate transferability in the fixed perturbation setting for a Fast Gradient Sign Method attack and non-transferability in a pixel-guided denoiser attack setting. The interpretation of this non-transferability can support the use of fast and train-free adversarial attacks targeting soft biometric classifiers as means to achieve soft biometric privacy protection while maintaining facial identity as utility.