Drop In Accuracy Research Articles

A Robust DCT-Based Scheme for Watermarking Deep Neural Networks

This paper proposes a new robust scheme to provide copy protection services to Deep Nural Network (DNN) models using digital watermarks. The rationale of the proposed technique is that robustness can be achieved by embedding a large watermark that can span the whole watermark. If an adversary attempts to destroy the watermark, he will end up destroying the DNN model. However, maximizing the embedding capacity will strongly degrade the performance of the DNN model. To control the performance-capacity-robustness trade-of, the proposed scheme utilizes the Discrete Cosine Transform (DCT) due to its strong compaction property, which becomes even stronger when applied to correlated segments of data. Therefore, the proposed technique extracts the DNN weights and groups them within correlated segments before applying the DCT. The experimental results have proven the effectiveness of the proposed DNN watermarking scheme as it was able to embed 1.73 Mb of data with only a 1.4% drop in the classification accuracy of a standard Residual Neural Network with 21 Convolutional layers.

Bearing Health Evaluation Model using Segmentive Technique and Cosine KNN in Different Loading Situations

Bearing faults are a common cause of machinery failure, and bearing vibration analysis is critical in preventing any unacceptable consequences of such failures. Advancements in smart data and computing make Artificial Intelligence (AI) preferable for bearing vibration analysis. Typically, signal processing and feature engineering are essential for achieving satisfactory classification accuracy. Additionally, a drop in classification accuracy is commonly observed during different loading situations due to the vastly varying vibration characteristics under different loads. This paper evaluates an AI model in variable loading situations using raw vibration signals, devoid of signal processing and feature engineering. The proposed AI model, Segmentive Cosine K-Nearest Neighbours (SCosKNN), demonstrated a higher overall classification accuracy of 90.6–94.3% in same loading situations, and 72.1–84.2% in different loading situations. An improvement of around 9% in same loadings and 10–14% in different loadings were observed compared to a model without Segmentive Technique

3D Knee and Hip Angle Estimation With Reduced Wearable IMUs via Transfer Learning During Yoga, Golf, Swimming, Badminton, and Dance.

Wearable lower-limb joint angle estimation using a reduced inertial measurement unit (IMU) sensor set could enable quick, economical sports injury risk assessment and motion capture; however the vast majority of existing research requires a full IMU set attached to every related body segment and is implemented in only a single movement, typically walking. We thus implemented 3-dimensional knee and hip angle estimation with a reduced IMU sensor set during yoga, golf, swimming (simulated lower body swimming in a seated posture), badminton, and dance movements. Additionally, current deep-learning models undergo an accuracy drop when tested with new and unseen activities, which necessitates collecting large amounts of data for the new activity. However, collecting large datasets for every new activity is time-consuming and expensive. Thus, a transfer learning (TL) approach with long short-term memory neural networks was proposed to enhance the model's generalization ability towards new activities while minimizing the need for a large new-activity dataset. This approach could transfer the generic knowledge acquired from training the model in the source-activity domain to the target-activity domain. The maximum improvement in estimation accuracy (RMSE) achieved by TL is 23.6 degrees for knee flexion/extension and 22.2 degrees for hip flexion/extension compared to without TL. These results extend the application of motion capture with reduced sensor configurations to a broader range of activities relevant to injury prevention and sports training. Moreover, they enhance the capacity of data-driven models in scenarios where acquiring a substantial amount of training data is challenging.

Target Before Shooting: Accurate Anomaly Detection and Localization Under One Millisecond via Cascade Patch Retrieval.

In this work, by re-examining the "matching" nature of Anomaly Detection (AD), we propose a novel AD framework that simultaneously enjoys new records of AD accuracy and dramatically high running speed. In this framework, the anomaly detection problem is solved via a cascade patch retrieval procedure that retrieves the nearest neighbors for each test image patch in a coarse-to-fine fashion. Given a test sample, the top-K most similar training images are first selected based on a robust histogram matching process. Secondly, the nearest neighbor of each test patch is retrieved over the similar geometrical locations on those "most similar images", by using a carefully trained local metric. Finally, the anomaly score of each test image patch is calculated based on the distance to its "nearest neighbor" and the "non-background" probability. The proposed method is termed "Cascade Patch Retrieval" (CPR) in this work. Different from the previous patch-matching-based AD algorithms, CPR selects proper "targets" (reference images and patches) before "shooting" (patch-matching). On the well-acknowledged MVTec AD, BTAD and MVTec-3D AD datasets, the proposed algorithm consistently outperforms all the comparing SOTA methods by remarkable margins, measured by various AD metrics. Furthermore, CPR is extremely efficient. It runs at the speed of 113 FPS with the standard setting while its simplified version only requires less than 1 ms to process an image at the cost of a trivial accuracy drop. The code of CPR is available at https://github.com/flyinghu123/CPR.

Vina-FPGA-Cluster: Multi-FPGA Based Molecular Docking Tool with High-Accuracy and Multi-Level Parallelism.

AutoDock Vina (Vina) stands out among numerous molecular docking tools due to its precision and comparatively high speed, playing a key role in the drug discovery process. Hardware acceleration of Vina on FPGA platforms offers a high energy-efficiency approach to speed up the docking process. However, previous FPGA-based Vina accelerators exhibit several shortcomings: 1) Simple uniform quantization results in inevitable accuracy drop; 2) Due to Vina's complex computing process, the evaluation and optimization phase for hardware design becomes extended; 3) The iterative computations in Vina constrain the potential for further parallelization. 4) The system's scalability is limited by its unwieldy architecture. To address the above challenges, we propose Vina-FPGA-cluster, a multi-FPGA-based molecular docking tool enabling high-accuracy and multi-level parallel Vina acceleration. Standing upon the shoulders of Vina-FPGA, we first adapt hybrid fixed-point quantization to minimize accuracy loss. We then propose a SystemC-based model, accelerating the hardware accelerator architecture design evaluation. Next, we propose a novel bidirectional AG module for data-level parallelism. Finally, we optimize the system architecture for scalable deployment on multiple Xilinx ZCU104 boards, achieving task-level parallelism. Vina-FPGA-cluster is tested on three representative molecular docking datasets. The experiment results indicate that in the context of RMSD (for successful docking outcomes with metrics below 2Å), Vina-FPGA-cluster shows a mere 0.2% lose. Relative to CPU and Vina-FPGA, Vina-FPGA-cluster achieves 27.33× and 7.26× speedup, respectively. Notably, Vina-FPGA-cluster is able to deliver the 1.38× speedup as GPU implementation (Vina-GPU), with just the 28.99% power consumption.

SEDMA: Self-Distillation with Model Aggregation for Membership Privacy

Membership inference attacks (MIAs) are important measures to evaluate potential risks of privacy leakage from machine learning (ML) models. State-of-the-art MIA defenses have achieved favorable privacy-utility trade-offs using knowledge distillation on split training datasets. However, such defenses increase computational costs as a large number of the ML models must be trained on the split datasets. In this study, we proposed a new MIA defense, called SEDMA, based on self-distillation using model aggregation to mitigate the MIAs, inspired by the model parameter averaging as used in federated learning. The key idea of SEDMA is to split the training dataset into several parts and aggregate multiple ML models trained on each split for self-distillation. The intuitive explanation of SEDMA is that model aggregation prevents model over-fitting by smoothing information related to the training data among the multiple ML models and preserving the model utility, such as in federated learning. Through our experiments on major benchmark datasets (Purchase100, Texas100, and CIFAR100), we show that SEDMA outperforms state-of-the-art MIA defenses in terms of membership privacy (MIA accuracy), model accuracy, and computational costs. Specifically, SEDMA incurs at most approximately 3 - 5% model accuracy drop, while achieving the lowest MIA accuracy in state-of-the-art empirical MIA defenses. For computational costs, SEDMA takes significantly less processing time than a defense with the state-of-the-art privacy-utility trade-offs in previous defenses. SEDMA achieves both favorable privacy-utility trade-offs and low computational costs.

Online assessment of transient power angle & voltage in power system: Realizing fast prediction and fast update in any scenarios

Transient voltage and transient power angle are in the same time scale, both of which are the basis of safety operation of the power system. However, there are few studies on the integration of power angle and voltage at present. How to quickly realize the integrated assessment of voltage & power angle and how to shorten the update period when the model accuracy drops are the problems that need to be solved urgently. Therefore, this paper proposes a fast assessment scheme that considers both transient voltage and transient power angle. In this study, the stable boundary between the transient power angle and the transient voltage is first constructed from the time dimension by the variable step dichotomy. And then, a convolutional neural network with multiscale residual squeeze excitation (MRSE-CNN) is proposed, which can assess the voltage and power angle without post-fault-clearance data. It only takes three sampling points to accurately learn the mapping between the input feature and the stable boundary. At the same time, the results of whether the transient voltage and the transient power angle are stable, and the corresponding margin are output. By introducing the improved Huber loss function to dynamically adjust the cost of misjudgment and missing judgment, the reliability of the model is further enhanced. In the online application, a pool-based active transfer learning is proposed for the unlearned scenarios under load, topology, and renewable energy, which greatly reduces the adaptive update time of the model in unlearned scenarios. The model is verified in the improved IEEE 39 bus system and provincial interconnection system. It shows that the proposed method can quickly and accurately realize the integrated adaptive assessment of transient power angle and voltage in any scenarios.

Variance-Guided Structured Sparsity in Deep Neural Networks

The success of deep neural networks, especially convolutional neural networks in various applications, has greatly been possible by the presence of an enormous number of learnable parameters. These parameters increase the learning capacity of the model, but at the same time, it also significantly increases the computational and memory costs. This severely hinders the scalability of these models to limited resource environments, like IoT devices. The majority of the network weights are known to be redundant and can be removed from the network. This paper introduces a regularization scheme, which is the combination of structured sparsity regularization and variance regularization. It simultaneously helps to obtain computationally sparse models by making the majority of parameter groups zero and increasing the variance of non-zero groups to compensate for the accuracy drop. We use sparse group lasso, group sparsity variant of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\ell _{1}$</tex-math></inline-formula> (lasso) regularization, which removes redundant connections and unnecessary neurons from the network. For variance regularization, the KL divergence of the current parameter distribution and the target distribution is minimized, which aims to have the concentration of weights towards zero and a high variance of non-zero weights (skewed distribution). To check the effectiveness of the proposed regularizer, the experiments are performed on various benchmark datasets and it is noticed that variance regularization helps to reduce the accuracy drop caused by sparsity regularization. On MNIST, the trainable parameters are reduced from 331984 (baseline model) to 57327 and managed to obtain better accuracy than the baseline (99.6%). Also, on Fashion-MNIST, Cifar-10 and ImageNet the proposed scheme achieved state-of-the-art sparsity with almost no drop in accuracy.

ResMon: Domain-Adaptive Wireless Respiration State Monitoring via Few-Shot Bayesian Deep Learning

Under the outbreak of the COVID-19 pandemic, respiration state monitoring plays an important role in assisting respiratory disease diagnosis and treatment. Thanks to the non-intrusive nature and low deployment cost, Wi-Fi based wireless respiration state monitoring methods have gained increasing popularity. By analyzing the variation of channel state information (CSI) of Wi-Fi signals, the respiration states of a target person under the wireless coverage, such as cough, sneeze, and yawn, can be accurately detected. A major problem of the current wireless respiration state monitoring methods is being overly domain-dependent. That is, a sensing algorithm fine-tuned to a specific device placement and background setting (i.e., a domain) can result in drastic drop in detection accuracy when applied to a dissimilar new domain. To enhance the robustness of wireless sensing and reduce the sensing cost across different domains, we propose in this paper a domain-adaptive Respiration state Monitoring system (ResMon) that achieves highly accurate cross-domain detection performance while requiring very limited labeled samples in the new domain. In a nutshell, the proposed ResMon consists of a source domain meta-training stage and a target domain meta-testing stage. In the meta-training stage, we leverage the rich source domain labeled dataset to train an embedding model as a feature extractor of high-dimensional CSI data measurements. In particular, we apply the statistical Bayesian deep learning technique to improve the generalization performance of the embedding model in cross-domain applications. In the meta-testing stage, we combine the embedding model with a few-shot learning technique to train a domain-specific classifier using very limited labeled samples in the target domain. Experiment results show that the proposed ResMon can achieve on average 87.26% cross-domain detection accuracy in a 4-class respiration state classification task using only 5 labeled samples per class, which significantly outperforms the considered benchmark methods.

Handling Data Handoff of AI-Based Applications in Edge Computing Systems

Edge computing aims at better supporting low-latency applications. One of its key techniques is computation offloading, the process that outsources computing tasks from resourced-constrained mobile devices and moves them to edge data centers. In this paper, we tackle an emerging problem within the umbrella of computation offloading, i.e., migration of offloaded inference tasks of Artificial Intelligence (AI) trained models. Such context tailors migration aspects of data-sensitive services where i) the value of the updates is inversely proportional to the data age and ii) outage is highly detrimental to accuracy. To tackle this challenge, we propose Mobile Edge Data-handoff (MED) a framework able to relocate inference or online training tasks from one edge datacenter to another by moving only the necessary data to minimize any accuracy drop during the process. We implemented MED in a well-known edge computing emulator, openLEON, and experimentally verified its performance with an AI-based Industry 4.0 application that forecasts the gas flow in a chemical plant. For our experiments, we use a real, open-source dataset that contains sensors readings. Collected results show that MED, employing proactive data handoff algorithms, is able to minimize the packet loss during the handoff thereby providing guarantees on the inference accuracy.

AdvRain: Adversarial Raindrops to Attack Camera-Based Smart Vision Systems

Vision-based perception modules are increasingly deployed in many applications, especially autonomous vehicles and intelligent robots. These modules are being used to acquire information about the surroundings and identify obstacles. Hence, accurate detection and classification are essential to reach appropriate decisions and take appropriate and safe actions at all times. Current studies have demonstrated that “printed adversarial attacks”, known as physical adversarial attacks, can successfully mislead perception models such as object detectors and image classifiers. However, most of these physical attacks are based on noticeable and eye-catching patterns for generated perturbations making them identifiable/detectable by the human eye, in-field tests, or in test drives. In this paper, we propose a camera-based inconspicuous adversarial attack (AdvRain) capable of fooling camera-based perception systems over all objects of the same class. Unlike mask-based FakeWeather attacks that require access to the underlying computing hardware or image memory, our attack is based on emulating the effects of a natural weather condition (i.e., Raindrops) that can be printed on a translucent sticker, which is externally placed over the lens of a camera whenever an adversary plans to trigger an attack. Note, such perturbations are still inconspicuous in real-world deployments and their presence goes unnoticed due to their association with a natural phenomenon. To accomplish this, we develop an iterative process based on performing a random search aiming to identify critical positions to make sure that the performed transformation is adversarial for a target classifier. Our transformation is based on blurring predefined parts of the captured image corresponding to the areas covered by the raindrop. We achieve a drop in average model accuracy of more than 45% and 40% on VGG19 for ImageNet dataset and Resnet34 for Caltech-101 dataset, respectively, using only 20 raindrops.

A time series and deep fusion framework for rotating machinery fault diagnosis

For shafting rotation equipment, in the fault diagnosis based on vibration analysis, the sampling signal is a waveform of a short moment, which is quasi-static information at the corresponding moment. A single quasi-static piece of information does not necessarily contain obvious fault information, however, in a longer period much larger than the sampling period, the time series information composed of multiple quasi-static combinations may well-characterize the fault. Traditional deep learning algorithms often focus on fault features in quasi-static information, ignoring time series features, which results in low diagnostic accuracy. To solve the above problems, this paper firstly presents a fault diagnosis framework of time series and deep fusion network (TDFN). Then, a method to extract time series information and quasi-static information based on expert experience knowledge is proposed, and finally studies the method of using this framework to train the network and fuse the two types of features. Different from existing deep learning methods, TDFN includes time series blocks, depth blocks and fusion blocks. This paper uses laboratory data and industrial rotor data for verification, the results show that the overall accuracy of TDFN reached 93.8% and it does not experience a drop in accuracy after fusion. The diagnosis accuracy is higher than traditional networks, providing guiding suggestions for the management and operation of on-site industrial equipment.

Adversarial Attacks in Underwater Acoustic Target Recognition with Deep Learning Models

Deep learning models can produce unstable results by introducing imperceptible perturbations that are difficult for humans to recognize. This can have a significant impact on the accuracy and security of deep learning applications due to their poorly understood interpretability. As a field critical to security research, this problem clearly exists in underwater acoustic target recognition for ocean sensing. To address this issue, this article investigates the reliability of state-of-the-art deep learning models by exploring adversarial attack methods that add small, exquisite perturbations on acoustic Mel-spectrograms to generate adversarial spectrograms. Experimental results based on real-world datasets reveal that these models can be forced to learn unexpected features when subjected to adversarial spectrograms, resulting in significant accuracy drops. Specifically, when employing the iterative attack method, the overall accuracy of all models experiences a significant decrease of approximately 70% for two datasets under stronger perturbations.

Synaptic Characteristics and Vector‐Matrix Multiplication Operation in Highly Uniform and Cost‐Effective Four‐Layer Vertical RRAM Array

AbstractThis study implements a highly uniform 3D vertically stack resistive random‐access memory (VRRAM) with a four‐layer contact hole structure. The fabrication process of a four‐layer VRRAM is demonstrated, and its physical and electrical properties are thoroughly examined. X‐ray photoelectron spectroscopy and transmission electron microscopy are employed to analyze the chemical distribution and physical structure of the VRRAM device. Multilevel capability, reliable endurance (&gt;104 cycles), and retention (104 s) are successfully obtained. Synaptic memory plasticity, such as spike time‐dependent plasticity, spike rate‐dependent plasticity, excitatory post‐synaptic current, paired‐pulse facilitation, and long‐term potentiation and depression is presented. Finally, the vector‐matrix multiplication (VMM) operation is conducted on a 4 × 12 VRRAM array, according to the low resistance state ratio. It is ascertained that the accuracy drop, which can occur due to VMM error, can be limited to a decrease of less than 0.44% point. Utilizing the high‐density, multilevel, and biological characteristics of VRRAM, it is possible to implement high‐performance neuromorphic systems that require densely integrated synaptic devices.

Beam search decoder for enhancing sequence decoding speed in single-molecule peptide sequencing data.

Next-generation single-molecule protein sequencing technologies have the potential to significantly accelerate biomedical research. These technologies offer sensitivity and scalability for proteomic analysis. One auspicious method is fluorosequencing, which involves: cutting naturalized proteins into peptides, attaching fluorophores to specific amino acids, and observing variations in light intensity as one amino acid is removed at a time. The original peptide is classified from the sequence of light-intensity reads, and proteins can subsequently be recognized with this information. The amino acid step removal is achieved by attaching the peptides to a wall on the C-terminal and using a process called Edman Degradation to remove an amino acid from the N-Terminal. Even though a framework (Whatprot) has been proposed for the peptide classification task, processing times remain restrictive due to the massively parallel data acquisicion system. In this paper, we propose a new beam search decoder with a novel state formulation that obtains considerably lower processing times at the expense of only a slight accuracy drop compared to Whatprot. Furthermore, we explore how our novel state formulation may lead to even faster decoders in the future.

Simplifying YOLOv5 for deployment in a real crop monitoring setting

Deep learning-based object detection models have become a preferred choice for crop detection tasks in crop monitoring activities due to their high accuracy and generalization capabilities. However, their high computational demand and large memory footprint pose a challenge for use on mobile embedded devices deployed in crop monitoring settings. Various approaches have been taken to minimize the computational cost and reduce the size of object detection models such as channel and layer pruning, detection head searching, backbone optimization, etc. In this work, we approached computational lightening, model compression, and speed improvement by discarding one or more of the three detection scales of the YOLOv5 object detection model. Thus, we derived up to five separate fast and light models, each with only one or two detection scales. To evaluate the new models for a real crop monitoring use case, the models were deployed on NVIDIA Jetson nano and NVIDIA Jetson Orin devices. The new models achieved up to 21.4% reduction in giga floating-point operations per second (GFLOPS), 31.9% reduction in number of parameters, 30.8% reduction in model size, 28.1% increase in inference speed, with only a small average accuracy drop of 3.6%. These new models are suitable for crop detection tasks since the crops are usually of similar sizes due to the high likelihood of being in the same growth stage, thus, making it sufficient to detect the crops with just one or two detection scales.

DualConv: Dual Convolutional Kernels for Lightweight Deep Neural Networks.

Convolutional neural network (CNN) architectures are generally heavy on memory and computational requirements which make them infeasible for embedded systems with limited hardware resources. We propose dual convolutional kernels (DualConv) for constructing lightweight deep neural networks. DualConv combines 3×3 and 1×1 convolutional kernels to process the same input feature map channels simultaneously and exploits the group convolution technique to efficiently arrange convolutional filters. DualConv can be employed in any CNN model such as VGG-16 and ResNet-50 for image classification, you only look once (YOLO) and R-CNN for object detection, or fully convolutional network (FCN) for semantic segmentation. In this work, we extensively test DualConv for classification since these network architectures form the backbone for many other tasks. We also test DualConv for image detection on YOLO-V3. Experimental results show that, combined with our structural innovations, DualConv significantly reduces the computational cost and number of parameters of deep neural networks while surprisingly achieving slightly higher accuracy than the original models in some cases. We use DualConv to further reduce the number of parameters of the lightweight MobileNetV2 by 54% with only 0.68% drop in accuracy on CIFAR-100 dataset. When the number of parameters is not an issue, DualConv increases the accuracy of MobileNetV1 by 4.11% on the same dataset. Furthermore, DualConv significantly improves the YOLO-V3 object detection speed and improves its accuracy by 4.4% on PASCAL visual object classes (VOC) dataset.

Pruning Networks With Cross-Layer Ranking & k-Reciprocal Nearest Filters.

This article focuses on filter-level network pruning. A novel pruning method, termed CLR-RNF, is proposed. We first reveal a "long-tail" pruning problem in magnitude-based weight pruning methods and then propose a computation-aware measurement for individual weight importance, followed by a cross-layer ranking (CLR) of weights to identify and remove the bottom-ranked weights. Consequently, the per-layer sparsity makes up the pruned network structure in our filter pruning. Then, we introduce a recommendation-based filter selection scheme where each filter recommends a group of its closest filters. To pick the preserved filters from these recommended groups, we further devise a k -reciprocal nearest filter (RNF) selection scheme where the selected filters fall into the intersection of these recommended groups. Both our pruned network structure and the filter selection are nonlearning processes, which, thus, significantly reduces the pruning complexity and differentiates our method from existing works. We conduct image classification on CIFAR-10 and ImageNet to demonstrate the superiority of our CLR-RNF over the state-of-the-arts. For example, on CIFAR-10, CLR-RNF removes 74.1% FLOPs and 95.0% parameters from VGGNet-16 with even 0.3% accuracy improvements. On ImageNet, it removes 70.2% FLOPs and 64.8% parameters from ResNet-50 with only 1.7% top-five accuracy drops. Our project is available at https://github.com/lmbxmu/CLR-RNF.

Investigating Lightweight Transformer Models for Defect Detection

In industrial production, product defect detection is vital for quality control. Traditional manual inspection is inefficient and error-prone. Deep learning, particularly in image processing, has enabled computer-based automated defect detection. This paper proposes a Visual Transformer-based model to overcome limitations in industrial anomaly detection. Leveraging pretrained Vision Transformer and Point Transformer models, it extracts features from RGB images and point cloud data. Multimodal feature fusion enhances anomaly perception, with residual connections mitigating feature loss. On the MVTec AD dataset, it achieves 96.3% AU PRO for anomaly detection and 99.3% Pixel ROCAUC for anomaly segmentation. To enable deployment on devices like Raspberry Pi, the paper introduces a lightweight model via post-training quantization and pruning. This results in a 28.52% inference speedup with only a 1.08% average detection accuracy drop, facilitating practical industrial applications on compact devices.

Texture-aware re-parameterization to mitigate accuracy drop after quantization for 4K/8K image super-resolution

Texture-aware re-parameterization to mitigate accuracy drop after quantization for 4K/8K image super-resolution