Common Convolutional Neural Network Research Articles

EcoFlow: Efficient Convolutional Dataflows on Low-Power Neural Network Accelerators

Dilated and transposed convolutions are widely used in modern convolutional neural networks (CNNs). These kernels are used extensively during CNN training and inference of applications such as image segmentation and high-resolution image generation. We find that commonly-used low-power CNN inference accelerators are <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">not</i> optimized for both these convolutional kernels. Dilated and transposed convolutions introduce significant zero padding when mapped to the underlying spatial architecture, significantly degrading performance and energy efficiency. Existing approaches that address this issue require significant design changes to the otherwise simple, efficient, and well-adopted architectures used to compute direct convolutions. To address this challenge, we propose EcoFlow, a new set of dataflows and mapping algorithms for dilated and transposed convolutions. These algorithms are tailored to execute efficiently on existing low-cost, small-scale spatial architectures and requires minimal changes to existing accelerators. At its core, EcoFlow eliminates zero padding through careful dataflow orchestration and data mapping tailored to the spatial architecture. We evaluate EcoFlow on CNN training workloads and Generative Adversarial Network (GAN) workloads. Experiments in our new cycle-accurate simulator show that, using a common CNN inference accelerator, EcoFlow 1) reduces end-to-end CNN training time between 7-85%, and 2) improves end-to-end GAN training performance between 29-42%, compared to state-of-the-art CNN dataflows.

Brain Cognition-Inspired Dual-Pathway CNN Architecture for Image Classification.

Inspired by the global-local information processing mechanism in the human visual system, we propose a novel convolutional neural network (CNN) architecture named cognition-inspired network (CogNet) that consists of a global pathway, a local pathway, and a top-down modulator. We first use a common CNN block to form the local pathway that aims to extract fine local features of the input image. Then, we use a transformer encoder to form the global pathway to capture global structural and contextual information among local parts in the input image. Finally, we construct the learnable top-down modulator where fine local features of the local pathway are modulated by global representations of the global pathway. For ease of use, we encapsulate the dual-pathway computation and modulation process into a building block, called the global-local block (GL block), and a CogNet of any depth can be constructed by stacking a necessary number of GL blocks one after another. Extensive experimental evaluations have revealed that the proposed CogNets have achieved the state-of-the-art performance accuracies on all the six benchmark datasets and are very effective for overcoming the "texture bias" and the "semantic confusion" problems faced by many CNN models.

ADS-CNN: Adaptive Dataflow Scheduling for lightweight CNN accelerator on FPGAs

Lightweight convolutional neural networks (CNNs) enable lower inference latency and data traffic, facilitating deployment on resource-constrained edge devices such as field-programmable gate arrays (FPGAs). However, CNNs inference requires access to off-chip synchronous dynamic random-access memory (SDRAM), which significantly degrades inference speed and system power efficiency. In this paper, we propose an adaptive dataflow scheduling method for lightweight CNN accelerator on FPGAs named ADS-CNN. The key idea of ADS-CNN is to efficiently utilize on-chip resources and reduce the amount of SDRAM access. To achieve the reuse of logical resources, we design a time division multiplexing calculation engine to be integrated in ADS-CNN. We implement a configurable module for the convolution controller to adapt to the data reuse of different convolution layers, thus reducing the off-chip access. Furthermore, we exploit on-chip memory blocks as buffers based on the configuration of different layers in lightweight CNNs. On the resource-constrained Intel CycloneV SoC 5CSEBA6 FPGA platform, we evaluated six common lightweight CNN models to demonstrate the performance advantages of ADS-CNN. The evaluation results indicate that, compared with accelerators that use traditional tiling strategy dataflow, our ADS-CNN can achieve up to 1.29× speedup with the overall dataflow scale compression of 23.7%.

Stable Low-Rank CP Decomposition for Compression of Convolutional Neural Networks Based on Sensitivity

Modern convolutional neural networks (CNNs) play a crucial role in computer vision applications. The intricacy of the application scenarios and the growing dataset both significantly raise the complexity of CNNs. As a result, they are often overparameterized and have significant computational costs. One potential solution for optimizing and compressing the CNNs is to replace convolutional layers with low-rank tensor decomposition. The most suitable technique for this is Canonical Polyadic (CP) decomposition. However, there are two primary issues with CP decomposition that lead to a significant loss in accuracy. Firstly, the selection of tensor ranks for CP decomposition is an unsolved issue. Secondly, degeneracy and instability are common problems in the CP decomposition of contractional tensors, which makes fine-tuning the compressed model difficult. In this study, a novel approach was proposed for compressing CNNs by using CP decomposition. The first step involves using the sensitivity of convolutional layers to determine the tensor ranks for CP decomposition effectively. Subsequently, to address the degeneracy issue and enhance the stability of the CP decomposition, two novel techniques were incorporated: optimization with sensitivity constraints and iterative fine-tuning based on sensitivity order. Finally, the proposed method was examined on common CNN structures for image classification tasks and demonstrated that it provides stable performance and significantly fewer reductions in classification accuracy.

A Rotating Object Detector with Convolutional Dynamic Adaptive Matching

Standard convolution sliding along a fixed direction in common convolutional neural networks (CNNs) is inconsistent with the direction of aerial targets, making it difficult to effectively extract features with high-aspect-ratio and arbitrary directional targets. To this end, We have fully considered the dynamic adaptability of remote sensing (RS) detectors in feature extraction and the balance of sample gradients during training and designed a plug-and-play dynamic rotation convolution with an adaptive alignment function. Specifically, we design dynamic convolutions in the backbone network that can be closely coupled with the spatial features of aerial targets. We design a network that can capture the rotation angle of aerial targets and dynamically adjust the spatial sampling position of the convolution to reduce the difference between the convolution and the target in directional space. In order to improve the stability of the network, a gradient adaptive equalization loss function is designed during training. The loss function we designed strengthens the gradient of high-quality samples, dynamically balancing the gradients of samples of different qualities to achieve stable training of the network. Sufficient experiments were conducted on the DOTA, HRSC-2016, and UCAS-AOD datasets to demonstrate the effectiveness of the proposed method and to achieve an effective balance between complexity and accuracy.

Recognition of wheat rusts in a field environment based on improved DenseNet

Currently, the main methods for detecting plant diseases are sampling and manual visual inspection. However, these methods are time-consuming, laborious and prone to misinterpretation. Rapid advances in Deep Learning (DL) techniques offer new possibilities. This study focused on analysing the confounding factors among three types of wheat rust (stripe rust, leaf rust and stem rust) and aimed to achieve higher classification accuracy. The following approaches were used: (1) Images were collected from several crops and diseases: Wheat Rusts Dataset (WRD), Wheat Common Disease Dataset (WDD), and Common Poaceae Disease Dataset (PDD); (2) Seven common convolutional neural network (CNN) models were made and their performance compared. DenseNet121 was selected as the base model, and its classification results further analysed. The results of the above analyses were then considered using phenotypic morphology and model structure analysis, as well as potential confounder discussions; (3) Adjustments and optimisations were made based on the identified confounding factors. The final improved model, designated Imp-DenseNet, achieved the following accuracies with different datasets: Top-1 accuracy = 98.32% (WRD), Top-3 accuracy = 97.30% (WDD) and Top-5 accuracy = 96.60% (PDD) (Top-x Accuracy refers to the accuracy of the top-ranked category that matches or containing the actual results). The study revealed the potential factors contributing to the confusion among the three wheat rusts and successfully achieved higher accuracy. It can provide a new perspective for future research on other diseases of wheat or other crops.

Which Convolutional Neural Network Architecture is the Best at Detecting Alzheimer’s Disease: A head‐to‐head Comparison Using Relevance Maps

AbstractBackgroundAlzheimer’s disease (AD) is a neurodegenerative disorder which causes gradual and irreversible damage to the brain. These patterns of brain damage can be detected using T1‐weighted MRI scans by modern neuroimaging methods. Recent developments in convolutional neural networks (CNN) have achieved promising results in various computer‐vision tasks. Feature attribution methods such as layer‐wise relevance propagation (LRP) allow tracing back the information flow in CNNs to derive relevance heatmaps, which approximate the contribution of the input image regions on the model decision. We addressed the open question, which of the most common CNN architectures is best suited for medical image detection, i.e. AD classification based on MRI data.MethodFour CNN architectures are widely used in the literature: AlexNet, VGG, ResNet, and DenseNet. We adapted these CNN architectures to be used with 3D brain MRI data and trained the models on a heterogeneous dataset with N&gt;2200 from four large studies. We applied tenfold cross‐validation and additionally evaluated the results on an independent test dataset.ResultThe more complex CNN architectures DenseNet and ResNet with ‘skip connections’, provided the best results for both group separation tasks of ‐ AD vs Cognitively Normal (CN), and Mild Cognitive Impairment (MCI) vs CN, although the overall differences in accuracy did not reach statistical significance. From the mean relevance maps obtained using the LRP relevance propagation method (see Fig. 1.), we found that DenseNet focused primarily on the medial temporal lobe and posterior cingulate cortex atrophy. ResNet’s mean relevance maps were more noisy and heterogeneous.ConclusionDenseNet provided the most focused relevance maps, best matching apriori expectations of brain regions contributing to the detection of AD. Its ‘dense connections’ between the layers enabled a highly efficient information flow at various scales. Our evaluation setup also demonstrated the value of a holistic evaluation of models, where relevance maps are used in combination with classical performance metrics.

Which Convolutional Neural Network Architecture is the Best at Detecting Alzheimer’s Disease: A head‐to‐head Comparison Using Relevance Maps

AbstractBackgroundAlzheimer’s disease (AD) is a neurodegenerative disorder which causes gradual and irreversible damage to the brain. These patterns of brain damage can be detected using T1‐weighted MRI scans by modern neuroimaging methods. Recent developments in convolutional neural networks (CNN) have achieved promising results in various computer‐vision tasks. Feature attribution methods such as layer‐wise relevance propagation (LRP) allow tracing back the information flow in CNNs to derive relevance heatmaps, which approximate the contribution of the input image regions on the model decision. We addressed the open question, which of the most common CNN architectures is best suited for medical image detection, i.e. AD classification based on MRI data.MethodFour CNN architectures are widely used in the literature: AlexNet, VGG, ResNet, and DenseNet. We adapted these CNN architectures to be used with 3D brain MRI data and trained the models on a heterogeneous dataset with N&gt;2200 from four large studies. We applied tenfold cross‐validation and additionally evaluated the results on an independent test dataset.ResultThe more complex CNN architectures DenseNet and ResNet with ‘skip connections’, provided the best results for both group separation tasks of ‐ AD vs Cognitively Normal (CN), and Mild Cognitive Impairment (MCI) vs CN, although the overall differences in accuracy did not reach statistical significance. From the mean relevance maps obtained using the LRP relevance propagation method (see Fig. 1.), we found that DenseNet focused primarily on the medial temporal lobe and posterior cingulate cortex atrophy. ResNet’s mean relevance maps were more noisy and heterogeneous.ConclusionDenseNet provided the most focused relevance maps, best matching apriori expectations of brain regions contributing to the detection of AD. Its ‘dense connections’ between the layers enabled a highly efficient information flow at various scales. Our evaluation setup also demonstrated the value of a holistic evaluation of models, where relevance maps are used in combination with classical performance metrics.

RIC-CNN: Rotation-Invariant Coordinate Convolutional Neural Network

Due to the lack of rotation invariance in traditional convolution operations, even acting a slight rotation on the input can severely degrade the performance of Convolutional Neural Networks (CNNs). To address this, we propose a Rotation-Invariant Coordinate Convolution (RIC-C), which achieves natural invariance to arbitrary rotations around the input center without additional trainable parameters or data augmentation. We first evaluate the rotational invariance of RIC-C using the MNIST dataset and compare its performance with most previous rotation-invariant CNN models. RIC-C achieves state-of-the-art classification on the MNIST-rot test set without data augmentation and with lower computational costs. Then, the interchangeability of RIC-C with traditional convolution operations is demonstrated by seamlessly integrating it into common CNN models like VGG, ResNet, and DenseNet. We conduct remote sensing image classification on the NWPU VHR-10, MTARSI and AID datasets and patch matching experiments on the UBC benchmark dataset, showing that RIC-C significantly enhances the performance of CNN models across different applications, especially when training data is limited. Our codes can be downloaded from https://github.com/HanlinMo/Rotation-Invariant-Coordinate-Convolutional-Neural-Network.git.

Image Based Search Engine Using Deep Learning

Abstract: Machine the learning's field of image recognition as well as classification is one that is expanding quickly. The business ramifications of object identification, which is a crucial component of picture categorization, are enormous. A procedure to identify and recognize an item or property in a digital video or picture, image recognition is a subset of artificial intelligence. A larger phrase used to describe techniques for obtaining, processing, and analyzing data from the actual environment is computer vision. The highly dimensional data generates judgements that are expressed as numerical or pictorial information. Artificial intelligence (AI) also encompasses event detection, object identification, learning, picture the rebuilding process, and tracking of video in addition to image classification. This project outlines a methodical strategy to organizing using machine learning. digital photos. Convolutional neural network (CNN) and deep neural networks are two classifiers that may be combined to improve classification performance. Over the past several years, Convolutional Neural Networks (CNNs) have emerged as the leading approach for image classification and object recognition tasks. On several of the picture categorization databases, they now outperform humans. Most of these datasets are built around the idea of tangible classes; photographs are categorized according to the kind of item they include. Using Abstract classes, this project will propose a unique picture classifying dataset that should be simple for humans to solve but difficult for CNNs to fully understand. This dataset and potential variants of the dataset are used to assess the classifications performance of common CNN designs. Interesting topics for future study are found

Multiple-signal defect identification of hydraulic pump using an adaptive normalized model and S transform

Axial piston pump plays a pivotal role in a hydraulic transmission system since it can supply the core power source. The complexity of structure and the invisibility of failure feature bring more difficulties and challenges to fault identification of an axial piston pump. It is of great meaning to exploit an effective and feasible fault identification method for the safe and stable system operation. An improved convolutional neural network (CNN) is constructed by mining and utilizing the interdisciplinary advantages of artificial intelligence and machinery engineering. First, the modeling takes the batch normalization strategy into account due to its capability of decreasing the data distribution differences. Second, Bayesian algorithm is used for intelligent tuning of hyperparameters. Third, the improved CNN is applied to fault identification based on the S transform of multiple source signals. The performance of the proposed method is demonstrated by the experiments on an axial piston pump. The identification accuracies based on three signals reach 88.17%, 95.94%, and 99.44%. Results show that it can automatically recognize typical faults of the axial piston pump, and present the superiority to common CNNs.

Underwater Acoustic Target Recognition Based on Data Augmentation and Residual CNN

In the field of underwater acoustic recognition, machine learning methods rely on a large number of datasets to achieve high accuracy, while the actual collected signal samples are often very scarce, which has a great impact on the recognition performance. This paper presents a recognition method of an underwater acoustic target by the data augmentation technique and the residual convolutional neural network (CNN) model, which is used to expand training samples to improve recognition performance. As a representative model in residual CNN, the ResNet18 model is used for recognition. The whole process mainly includes mel-frequency cepstral coefficient (MFCC) feature extraction, data augmentation processing, and ResNet18 model recognition. On the base of the traditional data augmentation, this study used the deep convolutional generative adversarial network (DCGAN) model to realize the expansion of underwater acoustic samples and compared the recognition performance of support vector machine (SVM), common CNN, VGG19, and ResNet18. The recognition results of the MFCC, constant Q transform (CQT), and low-frequency analyzer and recorder (LOFAR) spectrum were also analyzed and compared. Experimental results showed that the recognition accuracy of the MFCC feature was better than that of other features at the same method, and using the data augmentation method could obviously improve the recognition performance. Moreover, the recognition performance of ResNet18 using data enhancement technology was better than that of other models, which was due to the combination of the data expansion advantage of data augmentation technology and the deep feature extracting ability of the residual CNN model. In addition, although this method was used for ship recognition in this paper, it is not limited to this. This method is also applicable to other target voice recognition, such as natural sound and underwater voice biometrics.

Comparative assessment of common pre-trained CNNs for vision-based surface defect detection of machined components

Small and Medium Enterprises (SMEs) and Micro, Small, and Medium Enterprises (MSMEs) contemplate integrating machine vision with high throughput manufacturing lines to ensure a consistent quality of standardized components. The inspection productivity can improve considerably by substituting machine vision with manual activities. The pre-trained Convolutional Neural Networks (CNNs) can facilitate enhanced machine vision capabilities compared to the rule-based classical image processing algorithms. However, the non-availability of labeled datasets and lack of expertise in model development restricts their utilities for SMEs and MSMEs. The present work examines the practicality of utilizing publicly available labeled datasets while developing surface defect detection algorithms using pre-trained CNNs considering case studies of typical machined components - flat washers and tapered rollers. It is shown that the publicly available surface defect datasets are ineffective for specific-case such as machined surfaces of flat washers and tapered rollers. The explicitly labeled image datasets can offer better prediction abilities in such cases. A comparative assessment of common pre-trained CNNs is conducted to identify an appropriate network while developing a surface defect detection framework for machined components. The common pre-trained CNNs VGG-19, GoogLeNet, ResNet-50, EfficientNet-b0, and DenseNet-201 showing prediction abilities for similar classification tasks have been examined. The pre-trained CNNs developed using explicit image datasets were implemented to segregate defective flat washers and tapered rollers as sample components manufactured by SMEs and MSMEs. The performance assessment was accomplished using parameters estimated from the confusion matrix. It is observed that EfficientNet-b0 outperforms other networks on most parameters, and it can be preferred while developing a surface defect detection algorithm. The outcomes of the present study form the basis for developing an integrated vision-based expert system for surface defect detection tasks.

Understanding of Convolutional Neural Network (CNN): A Review

The application of deep learning technology has increased rapidly in recent years. Technologies in deep learning increasingly emulate natural human abilities, such as knowledge learning, problem-solving, and decision-making. In general, deep learning can carry out self-training without repetitive programming by humans. Convolutional neural networks (CNNs) are deep learning algorithms commonly used in wide applications. CNN is often used for image classification, segmentation, object detection, video processing, natural language processing, and speech recognition. CNN has four layers: convolution layer, pooling layer, fully connected layer, and non-linear layer. The convolutional layer uses kernel filters to calculate the convolution of the input image by extracting the fundamental features. The pooling layer combines two successive convolutional layers. The third layer is the fully connected layer, commonly called the convolutional output layer. The activation function defines the output of a neural network, such as 'yes' or 'no'. The most common and popular CNN activation functions are Sigmoid, Tanh, ReLU, Leaky ReLU, Noisy ReLU, and Parametric Linear Units. The organization and function of the visual cortex greatly influence CNN architecture because it is designed to resemble the neuronal connections in the human brain. Some of the popular CNN architectures are LeNet, AlexNet and VGGNet.

A Lightweight CNN for Wind Turbine Blade Defect Detection Based on Spectrograms

Since wind turbines are exposed to harsh working environments and variable weather conditions, wind turbine blade condition monitoring is critical to prevent unscheduled downtime and loss. Realizing that common convolutional neural networks are difficult to use in embedded devices, a lightweight convolutional neural network for wind turbine blades (WTBMobileNet) based on spectrograms is proposed, reducing computation and size with a high accuracy. Compared to baseline models, WTBMobileNet without data augmentation has an accuracy of 97.05%, a parameter of 0.315 million, and a computation of 0.423 giga floating point operations (GFLOPs), which is 9.4 times smaller and 2.7 times less computation than the best-performing model with only a 1.68% decrease in accuracy. Then, the impact of difference data augmentation is analyzed. The WTBMobileNet with augmentation has an accuracy of 98.1%, and the accuracy of each category is above 95%. Furthermore, the interpretability and transparency of WTBMobileNet are demonstrated through class activation mapping for reliable deployment. Finally, WTBMobileNet is explored in drones image classification and spectrogram object detection, whose accuracy and mAP@[0.5, 0.95] are 89.55% and 70.7%, respectively. This proves that WTBMobileNet not only has a good performance in spectrogram classification, but also has good application potential in drone image classification and spectrogram object detection.

Red Flag/Blue Flag visualization of a common CNN for text classification.

A shallow convolutional neural network (CNN), TextCNN, has become nearly ubiquitous for classification among clinical and medical text. This research presents a novel eXplainable-AI (X-AI) software, Red Flag/Blue Flag (RFBF), designed for binary classification with TextCNN. RFBF visualizes each convolutional filter's discriminative capability. This is a more informative approach than direct assessment of logit contribution, features that overfit to train set nuances on smaller datasets may indiscriminately activate large logits on validation samples from both classes. RFBF enables model diagnosis, term feature verification, and overfit prevention. We present 3 use cases of (1) filter consistency assessment; (2) predictive performance improvement; and (3) estimation of information leakage between train and holdout sets. The use cases derive from experiments on TextCNN for binary prediction of surgical misadventure outcomes from physician-authored operative notes. Due to TextCNN's prevalence, this X-AI can benefit clinical text research, and hence improve patient outcomes.

Debris flow susceptibility mapping in mountainous area based on multi-source data fusion and CNN model – taking Nujiang Prefecture, China as an example

ABSTRACT Efforts to evaluate the susceptibility of debris flows in large areas, especially in mountainous regions, are often hampered by the alpine and canyon terrain. This paper proposes a convolution neural network (CNN) model named dense residual shuffle net (DRSNet). It is successfully applied to Nujiang Prefecture in Yunnan Province of China, a typical alpine area with frequent debris flows. DRSNet uses digital elevation model, remote sensing, lithology, soil type and precipitation data as input. First, dense connection and residual structure were used to extract the shallow features of various data. Next, channel shuffle, fuse block and fully connection were applied to strengthen the correlation between different shallow features and give inner danger scores. Finally, precipitation as the activation factor was introduced giving the valleys susceptibility. To verify the feasibility of DRSNet, comparative tests were conducted on 7 CNN models and 3 other machine learning (ML) methods. Experimental results show that DRSNet can achieve 78.6% accuracy in debris flow valley classification, which is at least 7.4% higher than common CNN models and 15.2% higher than other ML methods. This article provides new ideas for debris flow susceptibility evaluation.

CNN-based MRI analysis of Alzheimer's disease

The dementia known as Alzheimer's disease (AD) damages the brain. It is very common and almost irreversible in the population, and to date, there is no definitive diagnosis and no effective treatment. Despite the lack of comprehensive treatments, studies have shown that early detection can reduce the severity of the disease and lengthen the patient's life by delaying the development of the illness. Machine learning (ML), often known as deep learning, is a fast-growing science that makes extensive use of convolutional neural networks (CNNs). CNNs can be used for image recognition. One of the important features of AD can be observed by Magnetic resonance imaging (MRI): damage to brain cells and partial atrophy. For the training of CNNs, it is possible to achieve high accuracy in the diagnostic classification of AD. This article reviews the latest related studies, analyses the cases using three common CNN methods, ResNet, VGG-16 and GoogLeNet, and evaluates the use of migration methods. In order to help AD patients, get diagnosed sooner and spend longer in remission, this article assesses the use of multiple ML approaches throughout the diagnostic classification of AD and chooses the most accurate method.

Quantized convolutional neural networks through the lens of partial differential equations

Quantization of convolutional neural networks (CNNs) is a common approach to ease the computational burden involved in the deployment of CNNs, especially on low-resource edge devices. However, fixed-point arithmetic is not natural to the type of computations involved in neural networks. In this work, we explore ways to improve quantized CNNs using PDE-based perspective and analysis. First, we harness the total variation (TV) approach to apply edge-aware smoothing to the feature maps throughout the network. This aims to reduce outliers in the distribution of values and promote piecewise constant maps, which are more suitable for quantization. Secondly, we consider symmetric and stable variants of common CNNs for image classification and graph convolutional networks for graph node classification. We demonstrate through several experiments that the property of forward stability preserves the action of a network under different quantization rates. As a result, stable quantized networks behave similarly to their non-quantized counterparts even though they rely on fewer parameters. We also find that at times, stability even aids in improving accuracy. These properties are of particular interest for sensitive, resource-constrained, low-power or real-time applications like autonomous driving.

A deep learning-based diagnostic pattern for ultrasound breast imaging: can it reduce unnecessary biopsy?

BackgroundEarly studies have demonstrated the potential of deep learning in bringing revolutionary changes in medical analysis. However, it is unknown which deep learning based diagnostic pattern is more effective for differentiating malignant and benign breast lesions (BLs) and can assist radiologists to reduce unnecessary biopsies.MethodsA total of 506 malignant BLs and 557 benign BLs were enrolled in this study after excluding incomplete ultrasound images. 396 malignant BLs and 447 benign BLs were included in the training cohort while 110 malignant and 110 benign BLs were included in the validation cohort. All BLs in the training and validation cohort were biopsy-proven. The most common convolutional neural networks (VGG-16 and VGG-19) were applied to identify malignant and benign BLs using grey-scale ultrasound images. Two radiologists determined the malignant (suggestion for biopsy) and benign (suggestion for follow-up) BLs with a 2-step reading session. The first step was based on conventional ultrasound (US) images alone to make a biopsy or follow-up decision. The second step was to take deep learning results into account for the decision adjustment. If a deep learning result of a first-classified benign BL was above the cut-off value, then it was re-classified as malignant.ResultsIn terms of area under the curve (AUC), the VGG-19 model yielded the best diagnostic performance in both training [0.939, 95% confidence interval (CI): 0.924–0.954] and testing dataset (0.959, 95% CI: 0.937–0.982). With the aid of deep learning models, the AUC of radiologists improved from 0.805 (95% CI: 0.744–0.865) to 0.827 (95% CI: 0.771–0.875, VGG-16) and 0.914 (95% CI: 0.871–0.957, VGG-19). The unnecessary biopsies decreased from 10.0% (11/110) to 8.2% (9/110) (assisted by VGG-16) and 0.9% (1/110) (assisted by VGG-19).ConclusionsThe application of deep learning patterns in breast US may improve the diagnostic performance of radiologists by offering a second opinion. And thus, the assist of deep learning algorithm can considerably reduce the unnecessary biopsy rate in the clinical management of breast lesions.