Three-dimensional Convolutional Network Research Articles

Predicting multiple linear stapler firings in double stapling technique with an MRI-based deep-learning model

Multiple linear stapler firings is a risk factor for anastomotic leakage (AL) in laparoscopic low anterior resection (LAR) using double stapling technique (DST) anastomosis. In this study, our objective was to establish the risk factors for ≥ 3 linear stapler firings, and to create and validate a predictive model for ≥ 3 linear stapler firings in laparoscopic LAR using DST anastomosis. We retrospectively enrolled 328 mid–low rectal cancer patients undergoing laparoscopic LAR using DST anastomosis. With a split ratio of 4:1, patients were randomly divided into 2 sets: the training set (n = 260) and the testing set (n = 68). A clinical predictive model of ≥ 3 linear stapler firings was constructed by binary logistic regression. Based on three-dimensional convolutional networks, we built an image model using only magnetic resonance (MR) images segmented by Mask region-based convolutional neural network, and an integrated model based on both MR images and clinical variables. Area under the curve (AUC), sensitivity, specificity, accuracy, positive predictive value (PPV), and Youden index were calculated for each model. And the three models were validated by an independent cohort of 128 patients. There were 17.7% (58/328) patients received ≥ 3 linear stapler firings. Tumor size ≥ 5 cm (odds ratio (OR) = 2.54, 95% confidence interval (CI) = 1.15–5.60, p = 0.021) and preoperative carcinoma embryonic antigen (CEA) level > 5 ng/mL [OR = 2.20, 95% CI = 1.20–4.04, p = 0.011] were independent risk factors associated with ≥ 3 linear stapler firings. The integrated model (AUC = 0.88, accuracy = 94.1%) performed better on predicting ≥ 3 linear stapler firings than the clinical model (AUC = 0.72, accuracy = 86.7%) and the image model (AUC = 0.81, accuracy = 91.2%). Similarly, in the validation set, the integrated model (AUC = 0.84, accuracy = 93.8%) performed better than the clinical model (AUC = 0.65, accuracy = 65.6%) and the image model (AUC = 0.75, accuracy = 92.1%). Our deep-learning model based on pelvic MR can help predict the high-risk population with ≥ 3 linear stapler firings in laparoscopic LAR using DST anastomosis. This model might assist in determining preoperatively the anastomotic technique for mid–low rectal cancer patients.

Detection of solidification crack formation in laser beam welding videos of sheet metal using neural networks

AbstractLaser beam welding has become widely applied in many industrial fields in recent years. Solidification cracks remain one of the most common welding faults that can prevent a safe welded joint. In civil engineering, convolutional neural networks (CNNs) have been successfully used to detect cracks in roads and buildings by analysing images of the constructed objects. These cracks are found in static objects, whereas the generation of a welding crack is a dynamic process. Detecting the formation of cracks as early as possible is greatly important to ensure high welding quality. In this study, two end-to-end models based on long short-term memory and three-dimensional convolutional networks (3D-CNN) are proposed for automatic crack formation detection. To achieve maximum accuracy with minimal computational complexity, we progressively modify the model to find the optimal structure. The controlled tensile weldability test is conducted to generate long videos used for training and testing. The performance of the proposed models is compared with the classical neural network ResNet-18, which has been proven to be a good transfer learning model for crack detection. The results show that our models can detect the start time of crack formation earlier, while ResNet-18 only detects cracks during the propagation stage.

Detecting abnormal behavior in megastore for intelligent surveillance through 3D deep convolutional model

Abstract The use of neural networks in a range of academic and scientific pursuits has introduced a great interest in modeling human behavior and activity patterns to recognize particular events. Various methods have so far been proposed for building expert vision systems to understand the scene and draw true semantic inferences from the observed dynamics. However, classifying abnormal or unusual activities in real-time video sequences is still challenging, as the details in video sequences have a time continuity constraint. A cost-effective approach is still demanding and so this work presents an advanced three-dimensional convolutional network (A3DConvNet) for detecting abnormal behavior of persons by analyzing their actions. The network proposed is 15 layers deep that uses 18 convolutional operations to effectively analyze the video contents and produces spatiotemporal features. The integrated dense layer uses these features for the efficient learning process and the softmax layer is used as the output layer for labeling the sequences. Additionally, we have created a dataset that carries video clips to represent abnormal behaviors of humans in megastores/shops, which is a consequent contribution of this paper. The dataset includes five complicated activities in the shops/megastores: normal, shoplifting, drinking, eating, and damaging. By analyzing human actions, the proposed algorithm produces an alert if anything like abnormalities is found. The extensive experiments performed on the synthesized dataset demonstrate the effectiveness of our method, with achieved accuracy of up to 90.90%.

Applicability of UAV-based optical imagery and classification algorithms for detecting pine wilt disease at different infection stages

ABSTRACT As a quarantine disease with a rapid spread tendency in the context of climate change, accurate detection and location of pine wilt disease (PWD) at different infection stages is critical for maintaining forest health and being highly productivity. In recent years, unmanned aerial vehicle (UAV)-based optical remote-sensing images have provided new instruments for timely and accurate PWD monitoring. Numerous corresponding analysis algorithms have been proposed for UAV-based image classification, but their applicability of detecting different PWD infection stages has not yet been evaluated under a uniform conditions and criteria. This research aims to systematically assess the performance of multi-source images for detecting different PWD infection stages, analyze effective classification algorithms, and further analyze the validity of thermal images for early detection of PWD. In this study, PWD infection was divided into four stages: healthy, chlorosis, red and gray, and UAV-based hyperspectral (HSI), multispectral (MSI), and MSI with a thermal band (MSI&TIR) datasets were used as the data sources. Spectral analysis, support vector machine (SVM), random forest (RF), two- and three-dimensional convolutional network (2D- and 3D-CNN) algorithms were applied to these datasets to compare their classification abilities. The results were as follows: (I) The classification accuracy of the healthy, red, and gray stages using the MSI dataset was close to that obtained when using the MSI&TIR dataset with the same algorithms, whereas the HSI dataset displayed no obvious advantages. (II) The RF and 3D-CNN algorithms were the most accurate for all datasets (RF: overall accuracy = 94.26%, 3D-CNN: overall accuracy = 93.31%), while the spectral analysis method is also valid for the MSI&TIR dataset. (III) Thermal band displayed significant potential in detection of the chlorosis stage, and the MSI&TIR dataset displayed the best performance for detection of all infection stages. Considering this, we suggest that the MSI&TIR dataset can essentially satisfy PWD identification requirements at various stages, and the RF algorithm provides the best choice, especially in actual forest investigations. In addition, the performance of thermal imaging in the early monitoring of PWD is worthy of further investigation. These findings are expected to provide insight into future research and actual surveys regarding the selection of both remote sensing datasets and data analysis algorithms for detection requirements of different PWD infection stages to detect the disease earlier and prevent losses.

Automatic Description Method of Floor Exercise Video Based on Three-dimensional Convolutional Network and Multi-label Classification

Automatic Description Method of Floor Exercise Video Based on Three-dimensional Convolutional Network and Multi-label Classification

Multi-input dense convolutional network for classification of hepatocellular carcinoma and intrahepatic cholangiocarcinoma

Primary liver cancer is one of the leading causes of cancer deaths worldwide. The most common types of primary liver cancer are hepatocellular carcinoma (HCC) and intrahepatic cholangiocarcinoma (ICC). Influencing factors and treatment of HCC and ICC are different in clinical. However, due to common radiographic features of them, differentiating the two types of primary liver cancer is still very challenging. We aim to propose a method based on deep learning for the classification of HCC and ICC. In the first step, we adopt a modified U-Net to segment the liver cancer lesions from preprocessed enhanced CT images and we take the segmentation regions. And then, we propose a multi-input dense convolutional network (MIDC-net) to classify hepatocellular carcinoma and intrahepatic cholangiocarcinoma. Considering the spatial characteristics of medical images, we use a three-dimensional convolutional network to classify them. Meanwhile, in order to learn the characteristic differences of multi-stage images, arterial and venous images were used as the input of MIDC-net. The experiment result shows that the ROC curve of the method on the test data is over 0.96, and the accuracy is over 0.91.

Lite-3DCNN Combined with Attention Mechanism for Complex Human Movement Recognition.

Three-dimensional convolutional network (3DCNN) is an essential field of motion recognition research. The research work of this paper optimizes the traditional three-dimensional convolution network, introduces the self-attention mechanism, and proposes a new network model to analyze and process complex human motion videos. In this study, the average frame skipping sampling and scaling and the one-hot encoding are used for data pre-processing to retain more features in the limited data. The experimental results show that this paper innovatively designs a lightweight three-dimensional convolutional network combined with an attention mechanism framework, and the number of parameters of the model is reduced by more than 90% to only about 1.7 million. This study compared the performance of different models in different classifications and found that the model proposed in this study performed well in complex human motion video classification. Its recognition rate increased by 1%–8% compared with the C3D model.

Spatial–temporal interaction module for action recognition

Video action recognition methods based on deep learning can be divided into two types: two-dimensional convolutional networks (2D-ConvNets) relied and three-dimensional convolutional networks (3D-ConvNets) relied. 2D-ConvNets are more efficient to learn spatial features, but cannot capture temporal relationships directly. 3D-ConvNets can jointly learn spatial–temporal features, but their learning is time-consuming because of a large number of networks’ parameters. We therefore propose an effective spatial–temporal interaction (STI) module. The 2D spatial convolution and the one-dimensional temporal convolution are combined through attention mechanism in STI to learn the spatial–temporal information effectively and efficiently. The computation cost of the proposed method is far less than 3D convolution. The proposed STI module can be combined with 2D-ConvNets to obtain the effect of 3D-ConvNets with far fewer parameters, and it can also be inserted into 3D-ConvNets to improve their ability to learn spatial–temporal features, so as to improve the recognition accuracy. Experimental results show that the proposed method outperforms the existing counterparts on benchmark datasets.

Alzheimer's Disease Classification Through Imaging Genetic Data With IGnet.

The application of deep learning techniques to the detection and automated classification of Alzheimer's disease (AD) has recently gained considerable attention. The rapid progress in neuroimaging and sequencing techniques has enabled the generation of large-scale imaging genetic data for AD research. In this study, we developed a deep learning approach, IGnet, for automated AD classification using both magnetic resonance imaging (MRI) data and genetic sequencing data. The proposed approach integrates computer vision (CV) and natural language processing (NLP) techniques, with a deep three-dimensional convolutional network (3D CNN) being used to handle the three-dimensional MRI input and a Transformer encoder being used to manage the genetic sequence input. The proposed approach has been applied to the Alzheimer's Disease Neuroimaging Initiative (ADNI) data set. Using baseline MRI scans and selected single-nucleotide polymorphisms on chromosome 19, it achieved a classification accuracy of 83.78% and an area under the receiver operating characteristic curve (AUC-ROC) of 0.924 with the test set. The results demonstrate the great potential of using multi-disciplinary AI approaches to integrate imaging genetic data for the automated classification of AD.

Hidden Markov Model-Based Video Recognition for Sports

In this paper, we conduct an in-depth study and analysis of sports video recognition by improved hidden Markov model. The feature module is a complex gesture recognition module based on hidden Markov model gesture features, which applies the hidden Markov model features to gesture recognition and performs the recognition of complex gestures made by combining simple gestures based on simple gesture recognition. The combination of the two modules forms the overall technology of this paper, which can be applied to many scenarios, including some special scenarios with high-security levels that require real-time feedback and some public indoor scenarios, which can achieve different prevention and services for different age groups. With the increase of the depth of the feature extraction network, the experimental effect is enhanced; however, the two-dimensional convolutional neural network loses temporal information when extracting features, so the three-dimensional convolutional network is used in this paper to extract features from the video in time and space. Multiple binary classifications of the extracted features are performed to achieve the goal of multilabel classification. A multistream residual neural network is used to extract features from video data of three modalities, and the extracted feature vectors are fed into the attention mechanism network, then, the more critical information for video recognition is selected from a large amount of spatiotemporal information, further learning the temporal dependencies existing between consecutive video frames, and finally fusing the multistream network outputs to obtain the final prediction category. By training and optimizing the model in an end-to-end manner, recognition accuracies of 92.7% and 64.4% are achieved on the dataset, respectively.

Depression Speech Recognition With a Three-Dimensional Convolutional Network.

Depression has become one of the main afflictions that threaten people's mental health. However, the current traditional diagnosis methods have certain limitations, so it is necessary to find a method of objective evaluation of depression based on intelligent technology to assist in the early diagnosis and treatment of patients. Because the abnormal speech features of patients with depression are related to their mental state to some extent, it is valuable to use speech acoustic features as objective indicators for the diagnosis of depression. In order to solve the problem of the complexity of speech in depression and the limited performance of traditional feature extraction methods for speech signals, this article suggests a Three-Dimensional Convolutional filter bank with Highway Networks and Bidirectional GRU (Gated Recurrent Unit) with an Attention mechanism (in short 3D-CBHGA), which includes two key strategies. (1) The three-dimensional feature extraction of the speech signal can timely realize the expression ability of those depression signals. (2) Based on the attention mechanism in the GRU network, the frame-level vector is weighted to get the hidden emotion vector by self-learning. Experiments show that the proposed 3D-CBHGA can well establish mapping from speech signals to depression-related features and improve the accuracy of depression detection in speech signals.

Research on Driver Status Recognition System of Intelligent Vehicle Terminal Based on Deep Learning

Automobile safety driving technology is a hot topic in today’s society, which is very significant to the social transportation system. Vehicle driving behavior monitoring is the foundation and core of safe driving techniques. The research on existing vehicle safety technology can not only improve the understanding of current safe driving research progress, but also provide reference for future researchers. This paper proposes a state recognition system based on a three-dimensional convolutional neural network, which can identify several improper states frequently encountered by drivers during driving, including drinking, making phone calls, and smoking, and can also issue alarm interventions. The system takes the collected continuous video frame information as the input of the three-dimensional convolutional network, carries out multi-level feature extraction and spatio-temporal information fusion, and identifies the driver state according to the extracted spatio-temporal features. The state is judged by the facial feature points of the video stream, and the design of the video surveillance driver state recognition system is completed. Then, the driver status recognition is improved and optimized, and finally, the actual deployment of the driver status recognition system on the mobile terminal is completed. A large number of experimental results show that the driver status recognition system proposed in this paper has achieved upper identification accuracy.

Automatic Construction and Extraction of Sports Moment Feature Variables Using Artificial Intelligence

In this paper, we study the automatic construction and extraction of feature variables of sports moments and construct the extraction of the specific variables by artificial intelligence. In this paper, support vector machines, which have better performance in the case of small samples, are selected as classifiers, and multiclass classifiers are constructed in a one-to-one manner to achieve the classification and recognition of human sports postures. The classifier for a single decomposed action is constructed to transform the automatic description problem of free gymnastic movements into a multilabel classification problem. With the increase in the depth of the feature extraction network, the experimental effect is enhanced; however, the two-dimensional convolutional neural network loses temporal information when extracting features, so the three-dimensional convolutional network is used in this paper for spatial-temporal feature extraction of the video. The extracted features are binary classified several times to achieve the goal of multilabel classification. To form a comparison experiment, the results of the classification are randomly combined into a sentence and compared with the results of the automatic description method to verify the effectiveness of the method. The multiclass classifier constructed in this paper is used for human motion pose classification and recognition tests, and the experimental results show that the human motion pose recognition algorithm based on multifeature fusion can effectively improve the recognition accuracy and perform well in practical applications.

Aerobics Action Recognition Algorithm Based on Three-Dimensional Convolutional Neural Network and Multilabel Classification

In the context of modern people increasingly paying attention to health and promoting aerobics, the amount of data and audiences of aerobics videos has grown rapidly, and its potential application value has attracted widespread attention from scientific research and industry perspectives. This article has integrated computer vision and deep learning related knowledge to realize the intelligent recognition and representation of specific human movements in aerobics video sequences. The study proposes an automatic recognition method for floor exercise videos based on three-dimensional convolutional networks and multilabel classification. Since two-dimensional convolutional neural networks (CNNs) lose time information when extracting features, so to overcome this, the proposed research uses three-dimensional convolutional networks to perform video recognition. The feature is taken in time and space, and the extracted features are subjected to multiple binary classifications to achieve the goal of multilabel classification. Various comparison and simulation experiments are conducted for the proposed research, and the experimental results prove the effectiveness and superiority of the approach.

Three-dimensional convolutional neural network–based underground object classification using three-dimensional ground penetrating radar data

Ground-penetrating radar is a typical sensor system for analyzing underground facilities such as pipelines and rebars. The technique also can be used to detect an underground cavity, which is a potential sign of urban sinkholes. Multichannel ground-penetrating radar devices are widely used to detect underground cavities thanks to the capacity of informative three-dimensional data. Nevertheless, the three-dimensional ground-penetrating radar data interpretation is unclear and complicated when recognizing underground cavities because similar ground-penetrating radar data reflected from different underground objects are often mixed with the cavities. As it is prevalently known that the deep learning algorithm-based techniques are powerful at image classification, deep learning-based techniques for underground object detection techniques using two-dimensional GPR (ground-penetrating radar) radargrams have been researched upon in recent years. However, spatial information of underground objects can be characterized better in three-dimensional ground-penetrating radar voxel data than in two-dimensional ground-penetrating radar images. Therefore, in this study, a novel underground object classification technique is proposed by applying deep three-dimensional convolutional neural network on three-dimensional ground-penetrating radar data. First, a deep convolutional neural network architecture was developed using three-dimensional convolutional networks for recognizing spatial underground objects such as, pipe, cavity, manhole, and subsoil. The framework of applying the three-dimensional convolutional neural network into three-dimensional ground-penetrating radar data was then proposed and experimentally validated using real three-dimensional ground-penetrating radar data. In order to do that, three-dimensional ground-penetrating radar block data were used to train the developed three-dimensional convolutional neural network and to classify unclassified three-dimensional ground-penetrating radar data collected from urban roads in Seoul, South Korea. The validation results revealed that four underground objects (pipe, cavity, manhole, and subsoil) are successfully classified, and the average classification accuracy was 97%. In addition, a false alarm was rarely indicated.

Sparse Coding Guided Spatiotemporal Feature Learning for Abnormal Event Detection in Large Videos

Abnormal event detection in large videos is an important task in research and industrial applications, which has attracted considerable attention in recent years. Existing methods usually solve this problem by extracting local features and then learning an outlier detection model on training videos. However, most previous approaches merely employ hand-crafted visual features, which is a clear disadvantage due to their limited representation capacity. In this paper, we present a novel unsupervised deep feature learning algorithm for the abnormal event detection problem. To exploit the spatiotemporal information of the inputs, we utilize the deep three-dimensional convolutional network (C3D) to perform feature extraction. Then, the key problem is how to train the C3D network without any category labels. Here, we employ the sparse coding results of the hand-crafted features generated from the inputs to guide the unsupervised feature learning. Specifically, we define a multilevel similarity relationship between these inputs according to the statistical information of the shared atoms. In the following, we introduce the quadruplet concept to model the multilevel similarity structure, which could be used to construct a generalized triplet loss for training the C3D network. Furthermore, the C3D network could be utilized to generate the features for sparse coding again, and this pipeline could be iterated for several times. By jointly optimizing between the sparse coding and the unsupervised feature learning, we can obtain robust and rich feature representations. Based on the learned representations, the sparse reconstruction error is applied to predicting the anomaly score of each testing input. Experiments on several publicly available video surveillance datasets in comparison with a number of existing works demonstrate that the proposed algorithm performs favorably against the state-of-the-art methods.