Backbone Convolutional Neural Network Research Articles

PND-Net: plant nutrition deficiency and disease classification using graph convolutional network

Crop yield production could be enhanced for agricultural growth if various plant nutrition deficiencies, and diseases are identified and detected at early stages. Hence, continuous health monitoring of plant is very crucial for handling plant stress. The deep learning methods have proven its superior performances in the automated detection of plant diseases and nutrition deficiencies from visual symptoms in leaves. This article proposes a new deep learning method for plant nutrition deficiencies and disease classification using a graph convolutional network (GNN), added upon a base convolutional neural network (CNN). Sometimes, a global feature descriptor might fail to capture the vital region of a diseased leaf, which causes inaccurate classification of disease. To address this issue, regional feature learning is crucial for a holistic feature aggregation. In this work, region-based feature summarization at multi-scales is explored using spatial pyramidal pooling for discriminative feature representation. Furthermore, a GCN is developed to capacitate learning of finer details for classifying plant diseases and insufficiency of nutrients. The proposed method, called Plant Nutrition Deficiency and Disease Network (PND-Net), has been evaluated on two public datasets for nutrition deficiency, and two for disease classification using four backbone CNNs. The best classification performances of the proposed PND-Net are as follows: (a) 90.00% Banana and 90.54% Coffee nutrition deficiency; and (b) 96.18% Potato diseases and 84.30% on PlantDoc datasets using Xception backbone. Furthermore, additional experiments have been carried out for generalization, and the proposed method has achieved state-of-the-art performances on two public datasets, namely the Breast Cancer Histopathology Image Classification (BreakHis 40×\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\ imes $$\\end{document}: 95.50%, and BreakHis 100×\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\ imes $$\\end{document}: 96.79% accuracy) and Single cells in Pap smear images for cervical cancer classification (SIPaKMeD: 99.18% accuracy). Also, the proposed method has been evaluated using five-fold cross validation and achieved improved performances on these datasets. Clearly, the proposed PND-Net effectively boosts the performances of automated health analysis of various plants in real and intricate field environments, implying PND-Net’s aptness for agricultural growth as well as human cancer classification.

AOPPI: An adaptive obstruction perception pedestrian identifier for complex environments

Pedestrian detection plays a crucial role in ensuring traffic safety within the domain of computer vision. However, accurately detecting pedestrians in complex environments proves to be a challenge due to issues such as occlusion. To address this issue, this paper presents an end-to-end pedestrian detection model founded on the DEtection TRansformer (DETR) architecture, effectively managing occlusion scenarios. The proposed model comprises a backbone Convolutional Neural Network (CNN) and a Transformer network. The backbone CNN incorporates variable convolution and U-Net design principles to enhance feature extraction capabilities, particularly for occluded pedestrians. Additionally, our innovative Adaptive Occlusion-Aware Attention Mechanism (AOAM) is embedded within the Transformer network, allowing the model to dynamically adjust attention weights and enhance the localization and identification of occluded pedestrians. Extensive experiments on the Caltech and ETH datasets demonstrate the superior performance of our model compared to state-of-the-art approaches across four key evaluation metrics. This study provides effective methodologies and theoretical foundations for pedestrian detection in complex environments.

Transformer-Augmented Network With Online Label Correction for Facial Expression Recognition

Facial expression recognition (FER) in the wild is extremely challenging due to occlusions, variant head poses under unconstrained conditions and incorrect annotations (e.g., label noise). In this paper, we aim to improve the performance of in-the-wild FER with Transformers and online label correction. Different from pure CNNs based methods, we propose a Transformer-augmented network (TAN) to dynamically capture the relationships within each facial patch and across the facial patches. Specifically, the TAN translates a number of facial patch images into a set of visual feature sequences by a backbone convolutional neural network. The intra-patch Transformer is subsequently utilized to capture the most discriminative features within each visual feature sequence. The position-disentangled attention mechanism of the intra-patch Transformer is proposed to better incorporate the positional information for feature sequences. Furthermore, we propose the inter-patch Transformer to model the dependencies across these feature sequences. More importantly, we present the online label correction (OLC) framework to correct suspicious hard labels and accumulate soft labels based on the predictions of the model, which strengthens the robustness of our model against label noise. We validate our method on several widely-used datasets (RAF-DB, FERPlus, AffectNet), realistic occlusion and pose variation datasets, and synthetic noisy datasets. Extensive experiments on these benchmarks demonstrate that the proposed method performs favorably against state-of-the-art methods. The source code will be made publicly available.

Convolutional neural network-based magnetic resonance image differentiation of filum terminale ependymomas from schwannomas

PurposePreoperative diagnosis of filum terminale ependymomas (FTEs) versus schwannomas is difficult but essential for surgical planning and prognostic assessment. With the advancement of deep-learning approaches based on convolutional neural networks (CNNs), the aim of this study was to determine whether CNN-based interpretation of magnetic resonance (MR) images of these two tumours could be achieved.MethodsContrast-enhanced MRI data from 50 patients with primary FTE and 50 schwannomas in the lumbosacral spinal canal were retrospectively collected and used as training and internal validation datasets. The diagnostic accuracy of MRI was determined by consistency with postoperative histopathological examination. T1-weighted (T1-WI), T2-weighted (T2-WI) and contrast-enhanced T1-weighted (CE-T1) MR images of the sagittal plane containing the tumour mass were selected for analysis. For each sequence, patient MRI data were randomly allocated to 5 groups that further underwent fivefold cross-validation to evaluate the diagnostic efficacy of the CNN models. An additional 34 pairs of cases were used as an external test dataset to validate the CNN classifiers.ResultsAfter comparing multiple backbone CNN models, we developed a diagnostic system using Inception-v3. In the external test dataset, the per-examination combined sensitivities were 0.78 (0.71–0.84, 95% CI) based on T1-weighted images, 0.79 (0.72–0.84, 95% CI) for T2-weighted images, 0.88 (0.83–0.92, 95% CI) for CE-T1 images, and 0.88 (0.83–0.92, 95% CI) for all weighted images. The combined specificities were 0.72 based on T1-WI (0.66–0.78, 95% CI), 0.84 (0.78–0.89, 95% CI) based on T2-WI, 0.74 (0.67–0.80, 95% CI) for CE-T1, and 0.81 (0.76–0.86, 95% CI) for all weighted images. After all three MRI modalities were merged, the receiver operating characteristic (ROC) curve was calculated, and the area under the curve (AUC) was 0.93, with an accuracy of 0.87.ConclusionsCNN based MRI analysis has the potential to accurately differentiate ependymomas from schwannomas in the lumbar segment.

Deep Learning for Plant Identification and Disease Classification from Leaf Images: Multi-prediction Approaches

Deep learning (DL) plays an important role in modern agriculture, especially in plant pathology using leaf images where convolutional neural networks (CNN) are attracting a lot of attention. While numerous reviews have explored the applications of DL within this research domain, there remains a notable absence of an empirical study to offer insightful comparisons due to the employment of varied datasets in the evaluation. Furthermore, a majority of these approaches tend to address the problem as a singular prediction task, overlooking the multifaceted nature of predicting various aspects of plant species and disease types. Lastly, there is an evident need for a more profound consideration of the semantic relationships that underlie plant species and disease types. In this article, we start our study by surveying current DL approaches for plant identification and disease classification. We categorise the approaches into multi-model, multi-label, multi-output, and multi-task, in which different backbone CNNs can be employed. Furthermore, based on the survey of existing approaches in plant pathology and the study of available approaches in machine learning, we propose a new model named Generalised Stacking Multi-output CNN (GSMo-CNN). To investigate the effectiveness of different backbone CNNs and learning approaches, we conduct an intensive experiment on three benchmark datasets Plant Village, Plant Leaves, and PlantDoc. The experimental results demonstrate that InceptionV3 can be a good choice for a backbone CNN as its performance is better than AlexNet, VGG16, ResNet101, EfficientNet, MobileNet, and a custom CNN developed by us. Interestingly, there is empirical evidence to support the hypothesis that using a single model for both tasks can be comparable or better than using two models, one for each task. Finally, we show that the proposed GSMo-CNN achieves state-of-the-art performance on three benchmark datasets.

Cross-domain open-set rolling bearing fault diagnosis based on feature improvement adversarial network under noise condition

Domain adaptation (DA) technology has the ability to solve fault diagnosis (FD) problems under variable operating conditions. However, DA technology faces two issues: (1) in general, vibration signals inevitably contain noise, which makes it difficult to extract discriminant features.(2) there are unknown fault types in target domain. These issues will lead to poor diagnostic performance. To solve above issues, a new cross-domain open-set transfer FD method called feature improvement adversarial network (FIAN) is proposed in this article. Specifically, to alleviate noise interference, a feature improvement module (FIM) is proposed and embedded into the backbone convolutional neural network to form new feature extractor. FIM uses soft threshold function to enhance important information and suppresses redundant information. Furthermore,open-set DA by back-propagation (OSBP) is introduced into FIAN. OSBP can predict the probability that a target domain sample belongs to an unknown category, so that it can effectively identify unknown and known category samples. Experimental results demonstrated its effectiveness and superiority in two bearing datasets.

One-Vote Veto: Semi-Supervised Learning for Low-Shot Glaucoma Diagnosis.

Convolutional neural networks (CNNs) are a promising technique for automated glaucoma diagnosis from images of the fundus, and these images are routinely acquired as part of an ophthalmic exam. Nevertheless, CNNs typically require a large amount of well-labeled data for training, which may not be available in many biomedical image classification applications, especially when diseases are rare and where labeling by experts is costly. This article makes two contributions to address this issue: 1) It extends the conventional Siamese network and introduces a training method for low-shot learning when labeled data are limited and imbalanced, and 2) it introduces a novel semi-supervised learning strategy that uses additional unlabeled training data to achieve greater accuracy. Our proposed multi-task Siamese network (MTSN) can employ any backbone CNN, and we demonstrate with four backbone CNNs that its accuracy with limited training data approaches the accuracy of backbone CNNs trained with a dataset that is 50 times larger. We also introduce One-Vote Veto (OVV) self-training, a semi-supervised learning strategy that is designed specifically for MTSNs. By taking both self-predictions and contrastive predictions of the unlabeled training data into account, OVV self-training provides additional pseudo labels for fine-tuning a pre-trained MTSN. Using a large (imbalanced) dataset with 66,715 fundus photographs acquired over 15 years, extensive experimental results demonstrate the effectiveness of low-shot learning with MTSN and semi-supervised learning with OVV self-training. Three additional, smaller clinical datasets of fundus images acquired under different conditions (cameras, instruments, locations, populations) are used to demonstrate the generalizability of the proposed methods.

Deep Learning-Based Weed Detection Using UAV Images: A Comparative Study

Semantic segmentation has been widely used in precision agriculture, such as weed detection, which is pivotal to increasing crop yields. Various well-established and swiftly evolved AI models have been developed of late for semantic segmentation in weed detection; nevertheless, there is insufficient information about their comparative study for optimal model selection in terms of performance in this field. Identifying such a model helps the agricultural community make the best use of technology. As such, we perform a comparative study of cutting-edge AI deep learning-based segmentation models for weed detection using an RGB image dataset acquired with UAV, called CoFly-WeedDB. For this, we leverage AI segmentation models, ranging from SegNet to DeepLabV3+, combined with five backbone convolutional neural networks (VGG16, ResNet50, DenseNet121, EfficientNetB0 and MobileNetV2). The results show that UNet with EfficientNetB0 as a backbone CNN is the best-performing model compared with the other candidate models used in this study on the CoFly-WeedDB dataset, imparting Precision (88.20%), Recall (88.97%), F1-score (88.24%) and mean Intersection of Union (56.21%). From this study, we suppose that the UNet model combined with EfficientNetB0 could potentially be used by the concerned stakeholders (e.g., farmers, the agricultural industry) to detect weeds more accurately in the field, thereby removing them at the earliest point and increasing crop yields.

A Multi-Scaling Reinforcement Learning Trading System Based on Multi-Scaling Convolutional Neural Networks

Advancements in machine learning have led to an increased interest in applying deep reinforcement learning techniques to investment decision-making problems. Despite this, existing approaches often rely solely on single-scaling daily data, neglecting the importance of multi-scaling information, such as weekly or monthly data, in decision-making processes. To address this limitation, a multi-scaling convolutional neural network for reinforcement learning-based stock trading, termed multi-scaling convolutional neural network SARSA (state, action, reward, state, action), is proposed. Our method utilizes a multi-scaling convolutional neural network to obtain multi-scaling features of daily and weekly financial data automatically. This involves using a convolutional neural network with several filter sizes to perform a multi-scaling extraction of temporal features. Multiple-scaling feature mining allows agents to operate over longer time scaling, identifying low stock positions on the weekly line and avoiding daily fluctuations during continuous declines. This mimics the human approach of considering information at varying temporal and spatial scaling during stock trading. We further enhance the network’s robustness by adding an average pooling layer to the backbone convolutional neural network, reducing overfitting. State, action, reward, state, action, as an on-policy reinforcement learning method, generates dynamic trading strategies that combine multi-scaling information across different time scaling, while avoiding dangerous strategies. We evaluate the effectiveness of our proposed method on four real-world datasets (Dow Jones, NASDAQ, General Electric, and AAPLE) spanning from 1 January 2007 to 31 December 2020, and demonstrate its superior profits compared to several baseline methods. In addition, we perform various comparative and ablation tests in order to demonstrate the superiority of the proposed network architecture. Through these experiments, our proposed multi-scaling module yields better results compared to the single-scaling module.

Spatial Correlation Module for Classification of Multi-Label Ocular Diseases Using Color Fundus Images

To prevent irreversible damage to one’s eyesight, ocular diseases (ODs) need to be recognized and treated immediately. Color fundus imaging (CFI) is a screening technology that is both effective and economical. According to CFIs, the early stages of the disease are characterized by a paucity of observable symptoms, which necessitates the prompt creation of automated and robust diagnostic algorithms. The traditional research focuses on image-level diagnostics that attend to the left and right eyes in isolation without making use of pertinent correlation data between the two sets of eyes. In addition, they usually only target one or a few different kinds of eye diseases at the same time. In this study, we design a patient-level multi-label OD (PLML_ODs) classification model that is based on a spatial correlation network (SCNet). This model takes into consideration the relevance of patient-level diagnosis combining bilateral eyes and multi-label ODs classification. PLML_ODs is made up of three parts: a backbone convolutional neural network (CNN) for feature extraction i.e., DenseNet-169, a SCNet for feature correlation, and a classifier for the development of classification scores. The DenseNet-169 is responsible for retrieving two separate sets of attributes, one from each of the left and right CFI. After then, the SCNet will record the correlations between the two feature sets on a pixel-by-pixel basis. After the attributes have been analyzed, they are integrated to provide a representation at the patient level. Throughout the whole process of ODs categorization, the patient-level representation will be used. The efficacy of the PLML_ODs is examined using a soft margin loss on a dataset that is readily accessible to the public, and the results reveal that the classification performance is significantly improved when compared to several baseline approaches.

Patch attention convolutional vision transformer for facial expression recognition with occlusion

Despite substantial progress in Facial Expression Recognition (FER) in recent decades, most previous methods have been developed to recognize constrained facial expressions. Real-world occlusions lead to invisible facial regions and contaminated facial features, which undoubtedly increase the difficulty of FER in the wild. Therefore, a Patch Attention Convolutional Vision Transformer (PACVT) is proposed to tackle the occlusion FER problem. The backbone convolutional neural network is used to extract facial feature maps, which are cropped into multiple regional patches to extract local and global features. The Patch Attention Unit (PAU) is designed to perceive occluded regions by adaptively calculating the patch-level attention weights of local features for expression recognition. The facial patches are mapped into sequences of visual tokens, and the Vision Transformer (ViT) is employed to capture the interactions and correlations between these visual tokens from a global perspective. The self-attention in ViT enables the PACVT to focus on the salient patches with discriminative features and ignore the occlusion. Experiments are conducted on three widely used expression datasets and their occlusion subsets, and the results demonstrate that the proposed PACVT outperforms state-of-the-art methods on occlusion FER. Cross-dataset experiment results evidence the generalization ability of the PACVT.

GCENet: Global contextual exploration network for RGB-D salient object detection

Representing contextual features at multiple scales is important for RGB-D SOD. Recently, due to advances in backbone convolutional neural networks (CNNs) revealing stronger multi-scale representation ability, many methods achieved comprising performance. However, most of them represent multi-scale features in a layer-wise manner, which ignores the fine-grained global contextual cues in a single layer. In this paper, we propose a novel global contextual exploration network (GCENet) to explore the performance gain of multi-scale contextual features in a fine-grained manner. Concretely, a cross-modal contextual feature module (CCFM) is proposed to represent the multi-scale contextual features at a single fine-grained level, which can enlarge the range of receptive fields for each network layer. Furthermore, we design a multi-scale feature decoder (MFD) that integrates fused features from CCFM in a top-down way. Extensive experiments on five benchmark datasets demonstrate that the proposed GCENet outperforms the other state-of-the-art (SOTA) RGB-D SOD methods.

A Novel CNN pooling layer for breast cancer segmentation and classification from thermograms.

Breast cancer is the second most frequent cancer worldwide, following lung cancer and the fifth leading cause of cancer death and a major cause of cancer death among women. In recent years, convolutional neural networks (CNNs) have been successfully applied for the diagnosis of breast cancer using different imaging modalities. Pooling is a main data processing step in CNN that decreases the feature maps' dimensionality without losing major patterns. However, the effect of pooling layer was not studied efficiently in literature. In this paper, we propose a novel design for the pooling layer called vector pooling block (VPB) for the CCN algorithm. The proposed VPB consists of two data pathways, which focus on extracting features along horizontal and vertical orientations. The VPB makes the CNNs able to collect both global and local features by including long and narrow pooling kernels, which is different from the traditional pooling layer, that gathers features from a fixed square kernel. Based on the novel VPB, we proposed a new pooling module called AVG-MAX VPB. It can collect informative features by using two types of pooling techniques, maximum and average pooling. The VPB and the AVG-MAX VPB are plugged into the backbone CNNs networks, such as U-Net, AlexNet, ResNet18 and GoogleNet, to show the advantages in segmentation and classification tasks associated with breast cancer diagnosis from thermograms. The proposed pooling layer was evaluated using a benchmark thermogram database (DMR-IR) and its results compared with U-Net results which was used as base results. The U-Net results were as follows: global accuracy = 96.6%, mean accuracy = 96.5%, mean IoU = 92.07%, and mean BF score = 78.34%. The VBP-based results were as follows: global accuracy = 98.3%, mean accuracy = 97.9%, mean IoU = 95.87%, and mean BF score = 88.68% while the AVG-MAX VPB-based results were as follows: global accuracy = 99.2%, mean accuracy = 98.97%, mean IoU = 98.03%, and mean BF score = 94.29%. Other network architectures also demonstrate superior improvement considering the use of VPB and AVG-MAX VPB.

CNN and transformer framework for insect pest classification

Insect pests pose a significant and increasing threat to agricultural production worldwide. However, most existing recognition methods are built upon well-known convolutional neural networks, which limits the possibility of improving pest recognition accuracies. This research attempts to overcome this challenge from a novel perspective, constructing a simplified but very useful network for effective insect pest recognition by combining transformer architecture and convolution blocks. First, the representative features are extracted from the input image using a backbone convolutional neural network. Second, a new transformer attention-based classification head is proposed to sufficiently utilize spatial data from the features. With that, we explore different combinations for each module in our model and abstract our model into a simple and scalable architecture; we introduce more effective training strategies, pretrained models and data augmentation methods. Our models performance was evaluated on the IP102 benchmark dataset and achieved classification accuracies of 74.897% and 75.583% with minimal implementation costs at image resolutions of 224 × 224 pixels and 480 × 480 pixels, respectively. Our model also attains accuracies of 99.472% and 97.935% on the D0 dataset and Li's dataset, respectively, with an image resolution of 224 × 224 pixels. The experimental results demonstrate that our method is superior to the state-of-the-art methods on these datasets. Accordingly, the proposed model can be deployed in practice and provides additional insights into the related research.

Prediction of response to neoadjuvant chemotherapy in breast cancer with recurrent neural networks and raw ultrasound signals

Objective. Prediction of the response to neoadjuvant chemotherapy (NAC) in breast cancer is important for patient outcomes. In this work, we propose a deep learning based approach to NAC response prediction in ultrasound (US) imaging. Approach. We develop recurrent neural networks that can process serial US imaging data to predict chemotherapy outcomes. We present models that can process either raw radio-frequency (RF) US data or regular US images. The proposed approach is evaluated based on 204 sequences of US data from 51 breast cancers. Each sequence included US data collected before the chemotherapy and after each subsequent dose, up to the 4th course. We investigate three pre-trained convolutional neural networks (CNNs) as back-bone feature extractors for the recurrent network. The CNNs were pre-trained using raw US RF data, US b-mode images and RGB images from the ImageNet dataset. The first two networks were developed using US data collected from malignant and benign breast masses. Main results. For the pre-treatment data, the better performing network, with back-bone CNN pre-trained on US images, achieved area under the receiver operating curve (AUC) of 0.81 (±0.04). Performance of the recurrent networks improved with each course of the chemotherapy. For the 4th course, the better performing model, based on the CNN pre-trained with RGB images, achieved AUC value of 0.93 (±0.03). Statistical analysis based on the DeLong test presented that there were no significant differences in AUC values between the pre-trained networks at each stage of the chemotherapy (p-values > 0.05). Significance. Our study demonstrates the feasibility of using recurrent neural networks for the NAC response prediction in breast cancer US.

Deep Transfer Learning for the Multilabel Classification of Chest X-ray Images.

Chest X-ray (CXR) is widely used to diagnose conditions affecting the chest, its contents, and its nearby structures. In this study, we used a private data set containing 1630 CXR images with disease labels; most of the images were disease-free, but the others contained multiple sites of abnormalities. Here, we used deep convolutional neural network (CNN) models to extract feature representations and to identify possible diseases in these images. We also used transfer learning combined with large open-source image data sets to resolve the problems of insufficient training data and optimize the classification model. The effects of different approaches of reusing pretrained weights (model finetuning and layer transfer), source data sets of different sizes and similarity levels to the target data (ImageNet, ChestX-ray, and CheXpert), methods integrating source data sets into transfer learning (initiating, concatenating, and co-training), and backbone CNN models (ResNet50 and DenseNet121) on transfer learning were also assessed. The results demonstrated that transfer learning applied with the model finetuning approach typically afforded better prediction models. When only one source data set was adopted, ChestX-ray performed better than CheXpert; however, after ImageNet initials were attached, CheXpert performed better. ResNet50 performed better in initiating transfer learning, whereas DenseNet121 performed better in concatenating and co-training transfer learning. Transfer learning with multiple source data sets was preferable to that with a source data set. Overall, transfer learning can further enhance prediction capabilities and reduce computing costs for CXR images.

Deep learning characterization of surface defects in the selective laser melting process

Surface defects in the selective laser melting (SLM) process adversely affect the surface quality of additive manufacturing workpieces. Although some studies have been conducted to classify or detect defects, the investigation of the distribution of defects has received relatively little attention. Currently, there are many studies which focus on the counting of objects and they typically adopt a backbone convolutional neural network to obtain an initial feature map, which contains more semantic information but loses some geometric details. In this paper, the distribution of surface defects and defect count estimation of additive manufactured components are first studied as well as presentation of a developed deep learning characterization method (DLCM) based on a detail-aware dilated convolutional neural network (DDCNN) incorporated with a fine details feature map extractor designed to obtain final fine semantic features. It features two dilated convolutional layer combination blocks (DCBs), which are proposed to fuse low-level features with semantic features. Data are acquired from the surface of a workpiece fabricated by SLM. A series of experiments have been conducted to validate the performance of the DLCM. Compared with the other main state-of-the-art methods, the proposed DLCM yields better results.

Scale and Background Aware Asymmetric Bilateral Network for Unconstrained Image Crowd Counting

This paper attacks the two challenging problems of image-based crowd counting, that is, scale variation and complex background. To that end, we present a novel crowd counting method, called the Scale and Background aware Asymmetric Bilateral Network (SBAB-Net), which is able to handle scale variation and background noise in a unified framework. Specifically, the proposed SBAB-Net contains three main components, a pre-trained backbone convolutional neural network (CNN) as the feature extractor and two asymmetric branches to generate a density map. These two asymmetric branches have different structures and use features from different semantic layers. One branch is densely connected stacked dilated convolution (DCSDC) sub-network with different dilation rates, which relies on one deep feature layer and can handle scale variation. The other branch is parameter-free densely connected stacked pooling (DCSP) sub-network with various pooling kernels and strides, which relies on shallow feature and can fuse features with several receptive fields to reduce the impact of background noise. Two sub-networks are fused by the attention mechanism to generate the final density map. Extensive experimental results on three widely-used benchmark datasets have demonstrated the effectiveness and superiority of our proposed method: (1) We achieve competitive counting performance compared to state-of-the-art methods; (2) Compared with baseline, the MAE and MSE are decreased by at least 6.3% and 11.3%, respectively.

A vision-based hand hygiene monitoring approach using self-attention convolutional neural network

Automatic hand hygiene monitoring is an effective and necessary practice in healthcare. The main objective is to automate the quality control of the hand hygiene process by recognizing handwashing actions. This paper proposes a new computer vision-based hand hygiene monitoring approach, with the aim to recognize handwashing actions from the input videos. The proposed approach consists of the following three modules. First, in the feature extraction module, a backbone convolutional neural network (CNN) model (i.e., a ResNet model) is used to extract features from each frame of the input video. Second, in the feature aggregation module, the self-attention mechanism is introduced to form six masked self-attention blocks that are stacked to aggregate features from multiple frames. Third, in the feature classification module, the self-attentive aggregated feature representation is used for handwashing action recognition. The proposed approach outperforms a few state-of-the-art handwashing action recognition approaches. The proposed approach is able to perform effective and accurate handwashing action recognition for automatic hand hygiene monitoring, as verified by three benchmark datasets.

Hybrid FPGA-CPU-Based Architecture for Object Recognition in Visual Servoing of Arm Prosthesis.

The present paper proposes an implementation of a hybrid hardware–software system for the visual servoing of prosthetic arms. We focus on the most critical vision analysis part of the system. The prosthetic system comprises a glass-worn eye tracker and a video camera, and the task is to recognize the object to grasp. The lightweight architecture for gaze-driven object recognition has to be implemented as a wearable device with low power consumption (less than 5.6 W). The algorithmic chain comprises gaze fixations estimation and filtering, generation of candidates, and recognition, with two backbone convolutional neural networks (CNN). The time-consuming parts of the system, such as SIFT (Scale Invariant Feature Transform) detector and the backbone CNN feature extractor, are implemented in FPGA, and a new reduction layer is introduced in the object-recognition CNN to reduce the computational burden. The proposed implementation is compatible with the real-time control of the prosthetic arm.