Image Classification Neural Networks Research Articles

Can deep neural networks work with amplitude and phase input of defocused images?

Deep neural network (DNN) models, particularly convolutional neural networks (CNNs), have demonstrated remarkable performance in biomedical image classification due to their ability to automatically learn features from large datasets. One common challenge in the preparation of large, microscopic datasets for DNN tasks is sample defocusing, potentially impairing the model performance. To handle defocusing, computational imaging, or specifically quantitative phase imaging (QPI), performs digital refocusing by using both the phase and the amplitude of the complex optical field. This leads us to investigate whether feeding the complex field into DNN would potentially address the defocusing problem as in-focus information is implicitly encoded in the complex field. In this paper, we assess the feasibility of employing neural networks to directly process full amplitude and phase data from a defocus plane without digital refocusing. Our specific focus lies in understanding the tolerance for defocus in image classification neural networks when amplitude and phase are taken as inputs. To accomplish this, we used Fourier ptychography microscopy (FPM) to acquire in-focus phase and amplitude images of two distinct object types – normal red blood cells and echinocytes. We then digitally propagate the complex field to generate progressively defocused images of the samples to serve as training and testing datasets for image classification neural networks. While the digitally defocused images contain sufficient information to recover the original in-focus images, we observed that current standard implementations of deep learning models are unable to effectively utilize the defocused field to distinguish between the two cell types. We conclude that the physical-model-based digital refocusing capability of QPI remains indispensable for overcoming defocusing issues in current standard DNN models.

Detection and coverage estimation of purple nutsedge in turf with image classification neural networks.

Accurate detection of weeds and estimation of their coverage is crucial for implementing precision herbicide applications. Deep learning (DL) techniques are typically used for weed detection and coverage estimation by analyzing information at the pixel or individual plant level, which requires a substantial amount of annotated data for training. This study aims to evaluate the effectiveness of using image-classification neural networks (NNs) for detecting and estimating weed coverage in bermudagrass turf. Weed-detection NNs, including DenseNet, GoogLeNet and ResNet, exhibited high overall accuracy and F1 scores (≥0.971) throughout the k-fold cross-validation. DenseNet outperformed GoogLeNet and ResNet with the highest overall accuracy and F1 scores (0.977). Among the evaluated NNs, DenseNet showed the highest overall accuracy and F1 scores (0.996) in the validation and testing data sets for estimating weed coverage. The inference speed of ResNet was similar to that of GoogLeNet but noticeably faster than DenseNet. ResNet was the most efficient and accurate deep convolution neural network for weed detection and coverage estimation. These results demonstrated that the developed NNs could effectively detect weeds and estimate their coverage in bermudagrass turf, allowing calculation of the herbicide requirements for variable-rate herbicide applications. The proposed method can be employed in a machine vision-based autonomous site-specific spraying system of smart sprayers. © 2024 Society of Chemical Industry.

Robustness and Transferability of Adversarial Attacks on Different Image Classification Neural Networks

Recent works demonstrated that imperceptible perturbations to input data, known as adversarial examples, can mislead neural networks’ output. Moreover, the same adversarial sample can be transferable and used to fool different neural models. Such vulnerabilities impede the use of neural networks in mission-critical tasks. To the best of our knowledge, this is the first paper that evaluates the robustness of emerging CNN- and transformer-inspired image classifier models such as SpinalNet and Compact Convolutional Transformer (CCT) against popular white- and black-box adversarial attacks imported from the Adversarial Robustness Toolbox (ART). In addition, the adversarial transferability of the generated samples across given models was studied. The tests were carried out on the CIFAR-10 dataset, and the obtained results show that the level of susceptibility of SpinalNet against the same attacks is similar to that of the traditional VGG model, whereas CCT demonstrates better generalization and robustness. The results of this work can be used as a reference for further studies, such as the development of new attacks and defense mechanisms.

Powering AI at the edge: A robust, memristor-based binarized neural network with near-memory computing and miniaturized solar cell

Memristor-based neural networks provide an exceptional energy-efficient platform for artificial intelligence (AI), presenting the possibility of self-powered operation when paired with energy harvesters. However, most memristor-based networks rely on analog in-memory computing, necessitating a stable and precise power supply, which is incompatible with the inherently unstable and unreliable energy harvesters. In this work, we fabricated a robust binarized neural network comprising 32,768 memristors, powered by a miniature wide-bandgap solar cell optimized for edge applications. Our circuit employs a resilient digital near-memory computing approach, featuring complementarily programmed memristors and logic-in-sense-amplifier. This design eliminates the need for compensation or calibration, operating effectively under diverse conditions. Under high illumination, the circuit achieves inference performance comparable to that of a lab bench power supply. In low illumination scenarios, it remains functional with slightly reduced accuracy, seamlessly transitioning to an approximate computing mode. Through image classification neural network simulations, we demonstrate that misclassified images under low illumination are primarily difficult-to-classify cases. Our approach lays the groundwork for self-powered AI and the creation of intelligent sensors for various applications in health, safety, and environment monitoring.

Deep Learning Detection of Aneurysm Clips for Magnetic Resonance Imaging Safety.

Flagging the presence of metal devices before a head MRI scan is essential to allow appropriate safety checks. There is an unmet need for an automated system which can flag aneurysm clips prior to MRI appointments. We assess the accuracy with which a machine learning model can classify the presence or absence of an aneurysm clip on CT images. A total of 280 CT head scans were collected, 140 with aneurysm clips visible and 140 without. The data were used to retrain a pre-trained image classification neural network to classify CT localizer images. Models were developed using fivefold cross-validation and then tested on a holdout test set. A mean sensitivity of 100% and a mean accuracy of 82% were achieved. Predictions were explained using SHapley Additive exPlanations (SHAP), which highlighted that appropriate regions of interest were informing the models. Models were also trained from scratch to classify three-dimensional CT head scans. These did not exceed the sensitivity of the localizer models. This work illustrates an application of computer vision image classification to enhance current processes and improve patient safety.

A novel text representation which enables image classifiers to also simultaneously classify text, applied to name disambiguation

We introduce a novel method for converting text data into abstract image representations, which allows image-based processing techniques (e.g. image classification networks) to be applied to text-based comparison problems. We apply the technique to entity disambiguation of inventor names in US patents, obtaining a list of IDs which identify individual inventors with high accuracy. The method involves converting text from each pairwise comparison between two inventor name records into a 2D RGB (stacked) image representation. We then train an image classification neural network to discriminate between such pairwise comparison images. The trained neural network then labels each pair of records as either matched (same inventor) or non-matched (different inventors), producing highly accurate results. Our new text-to-image representation method could also be used more broadly for other text comparison problems, such as entity disambiguation of academic publications, or for problems that require simultaneous classification of both text and image datasets.

Accurate Image Multi-Class Classification Neural Network Model with Quantum Entanglement Approach.

Quantum machine learning (QML) has attracted significant research attention over the last decade. Multiple models have been developed to demonstrate the practical applications of the quantum properties. In this study, we first demonstrate that the previously proposed quanvolutional neural network (QuanvNN) using a randomly generated quantum circuit improves the image classification accuracy of a fully connected neural network against the Modified National Institute of Standards and Technology (MNIST) dataset and the Canadian Institute for Advanced Research 10 class (CIFAR-10) dataset from 92.0% to 93.0% and from 30.5% to 34.9%, respectively. We then propose a new model referred to as a Neural Network with Quantum Entanglement (NNQE) using a strongly entangled quantum circuit combined with Hadamard gates. The new model further improves the image classification accuracy of MNIST and CIFAR-10 to 93.8% and 36.0%, respectively. Unlike other QML methods, the proposed method does not require optimization of the parameters inside the quantum circuits; hence, it requires only limited use of the quantum circuit. Given the small number of qubits and relatively shallow depth of the proposed quantum circuit, the proposed method is well suited for implementation in noisy intermediate-scale quantum computers. While promising results were obtained by the proposed method when applied to the MNIST and CIFAR-10 datasets, a test against a more complicated German Traffic Sign Recognition Benchmark (GTSRB) dataset degraded the image classification accuracy from 82.2% to 73.4%. The exact causes of the performance improvement and degradation are currently an open question, prompting further research on the understanding and design of suitable quantum circuits for image classification neural networks for colored and complex data.

Detecting Android Malware With Pre-Existing Image Classification Neural Networks

Detecting Android Malware With Pre-Existing Image Classification Neural Networks

Classification of neck tissues in OCT images by using convolutional neural network

Identification and classification of surrounding neck tissues are very important in thyroid surgery. The advantages of optical coherence tomography (OCT), high resolution, non-invasion, and non-destruction make it have great potential in identifying different neck tissues during thyroidectomy. We studied the automatic classification for neck tissues in OCT images based on convolutional neural network in this paper. OCT images of five kinds of neck tissues were collected firstly by our home-made swept source (SS-OCT) system, and a dataset was built for neural network training. Three image classification neural networks: LeNet, VGGNet, and ResNet, were used to train and test the dataset. The impact of transfer learning on the classification of neck tissue OCT images was also studied. Through the comparison of accuracy, it was found that ResNet has the best classification accuracy among the three networks. In addition, transfer learning did not significantly improve the accuracy, but it can somewhat accelerate the convergence of the network and shorten the network training time.

Drought stress impact on the performance of deep convolutional neural networks for weed detection in Bahiagrass

AbstractMachine vision‐based weed detection relies on features such as plant colour, leaf texture, shape, and patterns. Drought stress in plants can alter leaf colour and morphological features, which may in turn affect the reliability of machine vision‐based weed detection. The objective of this research was to evaluate the feasibility of using deep convolutional neural networks for the detection of Florida pusley (Richardia scabra L.) growing in drought stressed and unstressed bahiagrass (Paspalum natatum Flugge). The object detection neural networks you only look once (YOLO)v3, faster region‐based convolutional network (Faster R‐CNN), and variable filter net (VFNet) failed to effectively detect Florida pusley growing in drought stressed or unstressed bahiagrass, with F1 scores ≤0.54 in the testing dataset. Nevertheless, the use of the image classification neural networks AlexNet, GoogLeNet, and Visual Geometry Group‐Network (VGGNet) was highly effective and achieved high (≥0.97) F1 scores and recall values (≥0.98) in detecting images containing Florida pusley growing in drought stressed or unstressed bahiagrass. Overall, these results demonstrated the effectiveness of using an image classification convolutional neural network for detecting Florida pusley in drought stressed or unstressed bahiagrass. These findings illustrate the broad applicability of these neural networks for weed detection.

Investigating the reliability impacts of neutron-induced soft errors in aerial image classification CNNs implemented in a softcore SRAM-based FPGA GPU

This work investigates the impacts of neutron-induced soft errors on the reliability of aerial image classification neural networks running on a softcore GPU implemented in an SRAM-based FPGA. We designed and trained fixed-point and floating-point all-convolutional neural networks to classify four-channel aerial images from the SAT-6 dataset, extracted from the U.S. National Agriculture Imagery Program, and implemented on FGPU, a configurable open-source GPU-like processor with floating-point arithmetic hardware. Results from fast neutron and thermal neutron irradiation experiments coupled with configuration bitstream fault injection campaigns show that the impact of soft errors in the aerial image classification must be taken care of with hardening techniques.

A deep learning-based method for classification, detection, and localization of weeds in turfgrass.

Precision spraying of synthetic herbicides can reduce herbicide input. Previous research demonstrated the effectiveness of using image classification neural networks for detecting weeds growing in turfgrass, but did not attempt to discriminate weed species and locate the weeds on the input images. The objectives of this research were to: (i) investigate the feasibility of training deep learning models using grid cells (subimages) to detect the location of weeds on the image by identifying whether or not the grid cells contain weeds; and (ii) evaluate DenseNet, EfficientNetV2, ResNet, RegNet and VGGNet to detect and discriminate multiple weed species growing in turfgrass (multi-classifier) and detect and discriminate weeds (regardless of weed species) and turfgrass (two-classifier). The VGGNet multi-classifier exhibited an F1 score of 0.950 when used to detect common dandelion and achieved high F1 scores of ≥0.983 to detect and discriminate the subimages containing dallisgrass, purple nutsedge and white clover growing in bermudagrass turf. DenseNet, EfficientNetV2 and RegNet multi-classifiers exhibited high F1 scores of ≥0.984 for detecting dallisgrass and purple nutsedge. Among the evaluated neural networks, EfficientNetV2 two-classifier exhibited the highest F1 scores (≥0.981) for exclusively detecting and discriminating subimages containing weeds and turfgrass. The proposed method can accurately identify the grid cells containing weeds and thus precisely locate the weeds on the input images. Overall, we conclude that the proposed method can be used in the machine vision subsystem of smart sprayers to locate weeds and make the decision for precision spraying herbicides onto the individual map cells. © 2022 Society of Chemical Industry.

Neural Network Detection of Pacemakers for MRI Safety

Flagging the presence of cardiac devices such as pacemakers before an MRI scan is essential to allow appropriate safety checks. We assess the accuracy with which a machine learning model can classify the presence or absence of a pacemaker on pre-existing chest radiographs. A total of 7973 chest radiographs were collected, 3996 with pacemakers visible and 3977 without. Images were identified from information available on the radiology information system (RIS) and correlated with report text. Manual review of images by two board certified radiologists was performed to ensure correct labeling. The data set was divided into training, validation, and a hold-back test set. The data were used to retrain a pre-trained image classification neural network. Final model performance was assessed on the test set. Accuracy of 99.67% on the test set was achieved. Re-testing the final model on the full training and validation data revealed a few additional misclassified examples which are further analyzed. Neural network image classification could be used to screen for the presence of cardiac devices, in addition to current safety processes, providing notification of device presence in advance of safety questionnaires. Computational power to run the model is low. Further work on misclassified examples could improve accuracy on edge cases. The focus of many healthcare applications of computer vision techniques has been for diagnosis and guiding management. This work illustrates an application of computer vision image classification to enhance current processes and improve patient safety.

Adversarial Deep Learning: A Survey on Adversarial Attacks and Defense Mechanisms on Image Classification

The popularity of adapting deep neural networks (DNNs) in solving hard problems has increased substantially. Specifically, in the field of computer vision, DNNs are becoming a core element in developing many image and video classification and recognition applications. However, DNNs are vulnerable to adversarial attacks, in which, given a well-trained image classification model, a malicious input can be crafted by adding mere perturbations to misclassify the image. This phenomena raise many security concerns in utilizing DNNs in critical life applications which attracts the attention of academic and industry researchers. As a result, multiple studies have proposed discussing novel attacks that can compromise the integrity of state-of-the-art image classification neural networks. The raise of these attacks urges the research community to explore countermeasure methods to mitigate these attacks and increase the reliability of adapting DDNs in different major applications. Hence, various defense strategies have been proposed to protect DNNs against adversarial attacks. In this paper, we thoroughly review the most recent and state-of-the-art adversarial attack methods by providing an in-depth analysis and explanation of the working process of these attacks. In our review, we focus on explaining the mathematical concepts and terminologies of the adversarial attacks, which provide a comprehensive and solid survey to the research community. Additionally, we provide a comprehensive review of the most recent defense mechanisms and discuss their effectiveness in defending DNNs against adversarial attacks. Finally, we highlight the current challenges and open issues in this field as well as future research directions.

What You See is What it Means! Semantic Representation Learning of Code based on Visualization and Transfer Learning

Recent successes in training word embeddings for Natural Language Processing ( NLP ) tasks have encouraged a wave of research on representation learning for source code, which builds on similar NLP methods. The overall objective is then to produce code embeddings that capture the maximum of program semantics. State-of-the-art approaches invariably rely on a syntactic representation (i.e., raw lexical tokens, abstract syntax trees, or intermediate representation tokens) to generate embeddings, which are criticized in the literature as non-robust or non-generalizable. In this work, we investigate a novel embedding approach based on the intuition that source code has visual patterns of semantics. We further use these patterns to address the outstanding challenge of identifying semantic code clones. We propose the WySiWiM ( ‘ ‘What You See Is What It Means ” ) approach where visual representations of source code are fed into powerful pre-trained image classification neural networks from the field of computer vision to benefit from the practical advantages of transfer learning. We evaluate the proposed embedding approach on the task of vulnerable code prediction in source code and on two variations of the task of semantic code clone identification: code clone detection (a binary classification problem), and code classification (a multi-classification problem). We show with experiments on the BigCloneBench (Java), Open Judge (C) that although simple, our WySiWiM approach performs as effectively as state-of-the-art approaches such as ASTNN or TBCNN. We also showed with data from NVD and SARD that WySiWiM representation can be used to learn a vulnerable code detector with reasonable performance (accuracy ∼90%). We further explore the influence of different steps in our approach, such as the choice of visual representations or the classification algorithm, to eventually discuss the promises and limitations of this research direction.

Evaluation of different deep convolutional neural networks for detection of broadleaf weed seedlings in wheat.

In-field weed detection in wheat (Triticum aestivum L.) is challenging due to the occurrence of weeds in close proximity with the crop. The objective of this research was to evaluate the feasibility of using deep convolutional neural networks for detecting broadleaf weed seedlings growing in wheat. The object detection neural networks, including CenterNet, Faster R-CNN, TridenNet, VFNet, and You Only Look Once Version 3 (YOLOv3) were insufficient for weed detection in wheat because the recall never exceeded 0.58 in the testing dataset. The image classification neural networks including AlexNet, DenseNet, ResNet, and VGGNet were trained with small (5500 negative and 5500 positive images) or large training datasets (11 000 negative and 11 000 positive images) and three training image sizes (200 × 200, 300 × 300, and 400 × 400 pixels). For the small training dataset, increasing image sizes decreased the F1 scores of AlexNet and VGGNet but generally increased the F1 scores of DenseNet and ResNet. For the large training dataset, no obvious difference was detected between the training image sizes since all neural networks exhibited remarkable classification accuracies with high F1 scores (≥0.96). All image classification neural networks exhibited high F1 scores (≥0.99) when trained with the large training dataset and the training images of 200 × 200 pixels. CenterNet, Faster R-CNN, TridentNet, VFNet, and YOLOv3 were insufficient, while AlexNet, DenseNet, ResNet, and VGGNet trained with a large training dataset were highly effective for detection of broadleaf weed seedlings in wheat. © 2021 Society of Chemical Industry.

SDLV: Verification of Steering Angle Safety for Self-Driving Cars

AbstractSelf-driving cars over the last decade have achieved significant progress like driving millions of miles without any human intervention. However, behavioral safety in applying deep-neural-network-based (DNN based) systems for self-driving cars could not be guaranteed. Several real-world accidents involving self-driving cars have already happened, some of which have led to fatal collisions. In this paper, we present a novel and automated technique for verifying steering angle safety for self-driving cars. The technique is based on deep learning verification (DLV), which is an automated verification framework for safety of image classification neural networks. We extend DLV by leveraging neuron coverage and slack relationship to solve the judgement problem of predicted behaviors, and thus, to achieve verification of steering angle safety for self-driving cars. We evaluate our technique on the NVIDIA’s end-to-end self-driving architecture, which is a crucial ingredient in many modern self-driving cars. Experimental results show that our technique can successfully find adversarial misclassifications (i.e., incorrect steering decisions) within given regions if they exist. Therefore, we can achieve safety verification (if no misclassification is found for all DNN layers, in which case the network can be said to be stable or reliable w.r.t. steering decisions) or falsification (in which case the adversarial examples can be used to fine-tune the network).

A dedicated hardware accelerator for real-time acceleration of YOLOv2

In recent years, dedicated hardware accelerators for the acceleration of the convolutional neural network (CNN) have been extensively studied. Although many studies have presented efficient designs on FPGAs for image classification neural network models such as AlexNet and VGG, there are still little implementations for CNN-based object detection applications. This paper presents an OpenCL-based high-throughput FPGA accelerator for the YOLOv2 object detection algorithm on Arria-10 GX1150 FPGA. The proposed hardware architecture adopts a scalable pipeline design to support multi-resolution input image and full 8-bit fixed-point datapath to improve hardware resource utilization. Layer fusion technology that merges the convolution, batch normalization and Leaky-ReLU is also developed to avoid transmission of intermediate data between FPGA and external memory. Experimental results show that the final design achieves a peak throughput of 566 GOP/s under the working frequency of 190 MHz. The accelerator can execute YOLOv2 inference computation ( $$288\times 288$$ resolution) and tiny YOLOv2 ( $$416\times 416$$ resolution) at the speed of 35 and 71 FPS, respectively.

An Investigation of Surrogate Models for Efficient Performance-Based Decoding of 3D Point Clouds

Abstract This work investigates surrogate modeling techniques for learning to approximate a computationally expensive function evaluation of 3D models. While in the past, 3D point clouds have been a data format that is too high dimensional for surrogate modeling, by leveraging advances in 3D object autoencoding neural networks, these point clouds can be mapped to a one-dimensional latent space. This leads to the fundamental research question: what surrogate modeling technique is most suitable for learning relationships between the 3D geometric features of the objects captured in the encoded latent vector and the physical phenomena captured in the evaluation software? Radial basis functions (RBFs), Kriging, and shallow 1D analogs of popular deep 2D image classification neural networks are investigated in this work. We find the nonintuitive result that departing from neural networks to decode latent representations of 3D objects into performance predictions is far more efficient than using a neural network decoder. In test cases using datasets of aircraft and watercraft 3D models, the non-neural network surrogate models achieve comparable accuracy to the neural network models. We find that an RBF surrogate model is able to approximate the lift and drag coefficients of 234 aircraft models with a mean absolute error of 1.97 × 10−3 and trains in only 3 seconds. Furthermore, the RBF surrogate model is able to rank a set of designs with an average percentile error of less than 8%. In comparison, a 1D ResNet achieves an average absolute error of 1.35 × 103 in 38 min for the same test case. We validate the comparable accuracy of the four techniques through a test case involving 214 3D watercraft models, but we also find that the distribution of the performance values of the data, in particular the presence of many outliers, has a significant negative impact on accuracy. These results contradict a common perception of neural networks as an efficient “one-size-fits-all” solution for learning black-box functions and suggests that even within systems that utilize multiple neural networks, potentially more efficient alternatives should be considered for each network in the system. Depending on the required accuracy of the application, this surrogate modeling approach could be used to approximate an expensive simulation software, or if the tolerance for error is low, it serves as a first pass which can narrow down the number of candidate designs to be analyzed more thoroughly.

Which Has Better Visual Quality: The Clear Blue Sky or a Blurry Animal?

Image content variation is a typical and challenging problem in no-reference image-quality assessment (NR-IQA). This work pays special attention to the impact of image content variation on NR-IQA methods. To better analyze this impact, we focus on blur-dominated distortions to exclude the impacts of distortion-type variations. We empirically show that current NR-IQA methods are inconsistent with human visual perception when predicting the relative quality of image pairs with different image contents. In view of deep semantic features of pretrained image classification neural networks always containing discriminative image content information, we put forward a new NR-IQA method based on semantic feature aggregation (SFA) to alleviate the impact of image content variation. Specifically, instead of resizing the image, we first crop multiple overlapping patches over the entire distorted image to avoid introducing geometric deformations. Then, according to an adaptive layer selection procedure, we extract deep semantic features by leveraging the power of a pretrained image classification model for its inherent content-aware property. After that, the local patch features are aggregated using several statistical structures. Finally, a linear regression model is trained for mapping the aggregated global features to image-quality scores. The proposed method, SFA, is compared with nine representative blur-specific NR-IQA methods, two general-purpose NR-IQA methods, and two extra full-reference IQA methods on Gaussian blur images (with and without Gaussian noise/JPEG compression) and realistic blur images from multiple databases, including LIVE, TID2008, TID2013, MLIVE1, MLIVE2, BID, and CLIVE. Experimental results show that SFA is superior to the state-of-the-art NR methods on all seven databases. It is also verified that deep semantic features play a crucial role in addressing image content variation, and this provides a new perspective for NR-IQA.