Three-dimensional Convolutional Neural Network Research Articles

Architecture of a three-dimensional convolutional neural network for detecting the fact of falsification of a video sequence

The article reflects the use of neural network technologies to determine the facts of falsification of the contents of video sequences. In the modern world, new technologies have become an integral part of the multimedia environment, but their proliferation has also created a new threat - the possibility of misuse to falsify the contents of video sequences. This leads to serious problems, such as the spread of fake news and misinformation of society. The scientific article examines this problem and determines the need to use neural networks to solve it. In comparison with other existing models and approaches, neural networks have high efficiency and accuracy in detecting video data falsification due to their ability to extract complex features and learn from large amounts of source data, which is especially important when reducing the resolution of the analyzed video sequence. Within the framework of this work, a mathematical model for identifying the falsification of audio and video sequences in video recordings is presented, as well as a model based on a three-dimensional convolutional neural network to determine the fact of falsification of a video sequence by analyzing the contents of individual frames. Within the framework of this work, it was proposed to consider the problem of identifying falsifications in video recordings as a joint solution to two problems: identification of falsification of audio and video sequences, and the resulting problem itself was transformed into a classical classification problem. Any video recording can be assigned to one of the four groups described in the work. Only the videos belonging to the first group are considered authentic, and all the others are fabricated. To increase the flexibility of the model, probabilistic classifiers have been added, which allows to take into account the degree of confidence in the predictions. The peculiarity of the resulting solution is the ability to adjust the threshold values, which allows to adapt the model to different levels of rigor depending on the task. The architecture of a three-dimensional convolutional neural network, including a preprocessing layer and a neural network layer, is proposed to determine fabricated photoreceads. The resulting model has a sufficient degree of accuracy in determining falsified video sequences, taking into account a significant decrease in frame resolution. Testing of the model on a training dataset showed the proportion of correct detection of video sequence falsification above 70%, which is noticeably better than guessing. Despite the sufficient accuracy, the model can be refined to more significantly increase the proportion of correct predictions.

Investigating the interpretability of schizophrenia EEG mechanism through a 3DCNN-based hidden layer features aggregation framework

Background and objectiveElectroencephalogram (EEG) signals record brain activity, with growing interest in quantifying neural activity through complexity analysis as a potential biological marker for schizophrenia. Presently, EEG complexity analysis primarily relies on manual feature extraction, which is subjective and yields varied findings in studies involving schizophrenia and healthy controls. MethodsThis study aims to leverage deep learning methods for enhanced EEG complexity exploration, aiding early schizophrenia screening and diagnosis. Our proposed approach utilizes a three-dimensional Convolutional Neural Network (3DCNN) to extract enhanced data features for early schizophrenia identification and subsequent complexity analysis. Leveraging the spatiotemporal capabilities of 3DCNN, we extract advanced latent features and employ knowledge distillation to reintegrate these features into the original channels, creating feature-enhanced data. ResultsWe employ a 10-fold cross-validation strategy, achieving the average accuracies of 99.46% and 98.06% in subject-dependent experiments on Dataset 1(14SZ and 14HC) and Dataset 2 (45SZ and 39HC). The average accuracy for subject-independent is 96.04% and 92.67% on both datasets. Feature extraction and classification are conducted on both the re-aggregated data and the original data. Our results demonstrate that re-aggregated data exhibit superior classification performance and a more stable training process after feature extraction. In the complexity analysis of re-aggregated data, we observe lower entropy features in schizophrenic patients compared to healthy controls, with more pronounced differences in the temporal and frontal lobes. Analyzing Katz's Fractal Dimension (KFD) across three sub-bands of lobe channels reveals the lowest α band KFD value in schizophrenia patients. ConclusionsThis emphasizes the ability of our method to enhance the discrimination and interpretability in schizophrenia detection and analysis. Our approach enhances the potential for EEG-based schizophrenia diagnosis by leveraging deep learning, offering superior discrimination capabilities and richer interpretive insights.

White matter brain age as a biomarker of cerebrovascular burden in the ageing brain.

As the brain ages, it almost invariably accumulates vascular pathology, which differentially affects the cerebral white matter. A rich body of research has investigated the link between vascular risk factors and the brain. One of the less studied questions is that among various modifiable vascular risk factors, which is the most debilitating one for white matter health? A white matter specific brain age was developed to evaluate the overall white matter health from diffusion weighted imaging, using a three-dimensional convolutional neural network deep learning model in both cross-sectional UK biobank participants (n = 37,327) and a longitudinal subset (n = 1409). White matter brain age gap (WMBAG) was the difference between the white matter age and the chronological age. Participants with one, two, and three or more vascular risk factors, compared to those without any, showed an elevated WMBAG of 0.54, 1.23, and 1.94years, respectively. Diabetes was most strongly associated with an increased WMBAG (1.39years, p < 0.001) among all risk factors followed by hypertension (0.87years, p < 0.001) and smoking (0.69years, p < 0.001). Baseline WMBAG was associated significantly with processing speed, executive and global cognition. Significant associations of diabetes and hypertension with poor processing speed and executive function were found to be mediated through the WMBAG. White matter specific brain age can be successfully targeted for the examination of the most relevant risk factors and cognition, and for tracking an individual's cerebrovascular ageing process. It also provides clinical basis for the better management of specific risk factors.

Deep learning based assessment of hemodynamics in the coarctation of the aorta: comparison of bidirectional recurrent and convolutional neural networks.

The utilization of numerical methods, such as computational fluid dynamics (CFD), has been widely established for modeling patient-specific hemodynamics based on medical imaging data. Hemodynamics assessment plays a crucial role in treatment decisions for the coarctation of the aorta (CoA), a congenital heart disease, with the pressure drop (PD) being a crucial biomarker for CoA treatment decisions. However, implementing CFD methods in the clinical environment remains challenging due to their computational cost and the requirement for expert knowledge. This study proposes a deep learning approach to mitigate the computational need and produce fast results. Building upon a previous proof-of-concept study, we compared the effects of two different artificial neural network (ANN) architectures trained on data with different dimensionalities, both capable of predicting hemodynamic parameters in CoA patients: a one-dimensional bidirectional recurrent neural network (1D BRNN) and a three-dimensional convolutional neural network (3D CNN). The performance was evaluated by median point-wise root mean square error (RMSE) for pressures along the centerline in 18 test cases, which were not included in a training cohort. We found that the 3D CNN (median RMSE of 3.23mmHg) outperforms the 1D BRNN (median RMSE of 4.25mmHg). In contrast, the 1D BRNN is more precise in PD prediction, with a lower standard deviation of the error (±7.03mmHg) compared to the 3D CNN (±8.91mmHg). The differences between both ANNs are not statistically significant, suggesting that compressing the 3D aorta hemodynamics into a 1D centerline representation does not result in the loss of valuable information when training ANN models. Additionally, we evaluated the utility of the synthetic geometries of the aortas with CoA generated by using a statistical shape model (SSM), as well as the impact of aortic arch geometry (gothic arch shape) on the model's training. The results show that incorporating a synthetic cohort obtained through the SSM of the clinical cohort does not significantly increase the model's accuracy, indicating that the synthetic cohort generation might be oversimplified. Furthermore, our study reveals that selecting training cases based on aortic arch shape (gothic versus non-gothic) does not improve ANN performance for test cases sharing the same shape.

The identification of minor impact collisions in a long video for detecting property damages caused by fleeing vehicles using three-dimensional convolutional neural network

Abstract A parked vehicle damaged by a hit-and-run can only be repaired at the expense of the owner, unless the fleeing vehicle is identified and the driver apprehended. Identifying the fleeing vehicle involves using a video investigation method that searches for perpetrators through CCTV footage of the crime scene. When the length of the recorded video is long, the investigation may require an extended amount of time from the investigator, resulting in an added burden on their daily work. Some commercial companies are using object recognition and tracking technology to detect hit-and-run incidents; however, detecting small movements of a vehicle during a minor collision still remains a challenge. Therefore, there is a need for a system that can detect small movement in a vehicle in a lengthy video. Automatic recognition and tracking require a sufficient amount of training dataset. However, such a dataset for hit-and-run incidents is not publicly available. One of the reasons behind this scarcity is that it may violate personal information protection acts. On the other hand, instead of using real accident videos, we could use actors to simulate such accident scenes. Although this may be feasible, creating such a dataset would require substantial costs. In this paper, we describe a new dataset for hit-and-run incidents. We collected 833 hit-and-run videos by recreating a parking lot using miniaturized cars. This dataset has been made publicly available through Kaggle. We used three-dimensional convolution neural network, which is frequently used in the field of action recognition, to detect small movements of vehicles during hit-and-run incidents. In addition, the proportion of the area that surrounds the target vehicle to the min-max box of the vehicle itself and the length of the input frame are varied to compare the accuracy. As a result, we were able to achieve better accuracy by using the lowest proportion and the shortest input frame.

Coupling video vision transformer (ViVit) into land change simulation: a comparison with three-dimensional convolutional neural network (3DCNN)

ABSTRACT To enhance land use/cover change (LUCC) simulation accuracy, we introduced ViViT-ANN-CA, blending video vision transformer’s spatio-temporal features extraction ability, artificial neural network‘s (ANN) non-linearity computing ability, and CA’s spatial computing. Compared to 3DCNN-ANN-CA, ViViT-ANN-CA showed higher accuracy in simulating water bodies and vegetation, with overall improvements in Hailing District and Wuxi City. ViViT demonstrates comparable spatio-temporal feature extraction ability to three-dimensional convolutional neural network (3DCNN), promising for future ynamic LUCC simulations.

Deep Subsurface Pseudo-Lithostratigraphic Modeling Based on Three-Dimensional Convolutional Neural Network (3D CNN) Using Inversed Geophysical Properties and Shallow Subsurface Geological Model

Abstract Lithostratigraphic modeling holds a vital role in mineral resource exploration and geological studies. In this study, we introduce a novel approach for automating pseudo-lithostratigraphic modeling in the deep subsurface, leveraging inversed geophysical properties. We propose a three-dimensional convolutional neural network with adaptive moment estimation (3D Adam-CNN) to achieve this objective. Our model employs 3D geophysical properties as input features for training, concurrently reconstructing a 3D geological model of the shallow subsurface for lithostratigraphic labeling purposes. To enhance the accuracy of pseudo-lithostratigraphic modeling during the model training phase, we redesign the 3D CNN framework, fine-tuning its parameters using the Adam optimizer. The Adam optimizer ensures controlled parameter updates with minimal memory overhead, rendering it particularly well-suited for convolutional learning involving huge 3D datasets with multi-dimensional features. To validate our proposed 3D Adam-CNN model, we compare the performance of our approach with 1D and 2D CNN models in the Qingniandian area of Heilongjiang Province, Northeastern China. By cross-matching the model’s predictions with manually modeled shallow subsurface lithostratigraphic distributions, we substantiate its reliability and accuracy. The 3D Adam-CNN model emerges as a robust and effective solution for lithostratigraphic modeling in the deep subsurface, utilizing geophysical properties.

Hyperspectral image classification using Walsh Hadamard transform- based key band selection and deep convolutional neural networks

ABSTRACT Hyperspectral images (HSIs) have transformed the field of remote sensing by providing researchers with a wealth of information about the Earth’s surface. However, analyzing these images can be an overwhelming task due to the presence of overlapping areas, nested regions, and large intra-class variability. Hyperspectral image classification (HSIC) is a crucial part of identifying the various land cover classes present in hyperspectral images. In order to enhance the accuracy of HSIC, researchers utilize the potential of three-dimensional convolutional neural networks (3D-CNNs). With the ability to influence both the spectral and spatial data present in HSIs, 3D-CNNs provide a promising solution to overcome the challenges associated with HSIC. In this paper, a new method for key band selection is proposed to improve the performance of 3D-CNN model. The proposed method selects the most relevant key bands based on Walsh-Hadamard kernel strength features. These key bands are then used to extract overlapping 3D spatial patches, which serve as input to the proposed 3D-CNN model. To evaluate the performance of the 3D-CNN model six standard benchmark datasets are used. The effectiveness of the proposed method improves the performance of 3D-CNN for HSIC.

Machine learning-based vorticity evolution and super-resolution of homogeneous isotropic turbulence using wavelet projection

A wavelet-based machine learning method is proposed for predicting the time evolution of homogeneous isotropic turbulence where vortex tubes are preserved. Three-dimensional convolutional neural networks and long short-term memory are trained with a time series of direct numerical simulation (DNS) data of homogeneous isotropic turbulence at the Taylor microscale Reynolds number 92. The predicted results are assessed by using the flow visualization of vorticity and statistics, e.g., probability density functions of vorticity and enstrophy spectra. It is found that the predicted results are in good agreement with DNS results. The small-scale flow topology considering the second and the third invariants of the velocity gradient tensor likewise shows an approximate match. Furthermore, we apply the pre-trained neural networks to coarse-grained vorticity data using super-resolution. It is shown that the super-resolved flow field well agrees with the reference DNS field, and thus small-scale information and vortex tubes are well regenerated.

Using time-series imagery and 3DLSTM model to classify individual tree species

ABSTRACT Classification of individual tree species (ITS) is critical for fine-scale forest surveys. However, it is difficult to obtain the complete and high-precision data needed for ITS classification in large areas. Lower spatial resolution time-series imagery is more accessible than other types of imagery and contains rich phenological information. In this study, after delineating individual tree crowns using a 0.2-m unmanned aerial vehicle (UAV) image, we used 3-m time-series imagery and a new 3DLSTM model to identify ITS at the Gaofeng Forest Farm in Guangxi Province, China. The 3DLSTM ITS classification model combines three-dimensional convolutional neural network (3D CNN) and long short-term memory (LSTM) models; thus, spatial, multiband, and time-series information can be extracted simultaneously to identify ITS more accurately. In this study, when only 3-m Planet time-series imagery was used for classification, the 3DLSTM model offered an ITS classification accuracy of 92.68%, outperforming two ITS classifiers (DenseNet or AlexNet model) based on one individual image. Moreover, the 3DLSTM model was better at identifying broad-leaved tree species than other deep-learning models. The experimental results proved that ITS classification could be significantly improved using only 3DLSTM and time-series images, offering the possibility of classifying large-scale ITS at a low cost.

Deep Learning Approaches for Imaging-Based Automated Segmentation of Tuberous Sclerosis Complex

The present study presents a novel approach for identifying epileptogenic tubers in patients with tuberous sclerosis complex (TSC) and automating tuber segmentation using a three-dimensional convolutional neural network (3D CNN). The study retrospectively included 31 TSC patients whose lesions were manually annotated from multiparametric neuroimaging data. Epileptogenic tubers were determined via presurgical evaluation and stereoelectroencephalography recording. Neuroimaging metrics were extracted and compared between epileptogenic and non-epileptogenic tubers. Additionally, five datasets with different preprocessing strategies were used to construct and train 3D CNNs for automated tuber segmentation. The normalized positron emission tomography (PET) metabolic value was significantly lower in epileptogenic tubers defined via presurgical evaluation (p = 0.001). The CNNs showed high performance for localizing tubers, with an accuracy between 0.992 and 0.994 across the five datasets. The automated segmentations were highly correlated with clinician-based features. The neuroimaging characteristics for epileptogenic tubers were demonstrated, increasing surgical confidence in clinical practice. The validated deep learning detection algorithm yielded a high performance in determining tubers with an excellent agreement with reference clinician-based segmentation. Collectively, when coupled with our investigation of minimal input requirements, the approach outlined in this study represents a clinically invaluable tool for the management of TSC.

Attention-based 3D convolutional recurrent neural network model for multimodal emotion recognition.

Multimodal emotion recognition has become a hot topic in human-computer interaction and intelligent healthcare fields. However, combining information from different human different modalities for emotion computation is still challenging. In this paper, we propose a three-dimensional convolutional recurrent neural network model (referred to as 3FACRNN network) based on multimodal fusion and attention mechanism. The 3FACRNN network model consists of a visual network and an EEG network. The visual network is composed of a cascaded convolutional neural network-time convolutional network (CNN-TCN). In the EEG network, the 3D feature building module was added to integrate band information, spatial information and temporal information of the EEG signal, and the band attention and self-attention modules were added to the convolutional recurrent neural network (CRNN). The former explores the effect of different frequency bands on network recognition performance, while the latter is to obtain the intrinsic similarity of different EEG samples. To investigate the effect of different frequency bands on the experiment, we obtained the average attention mask for all subjects in different frequency bands. The distribution of the attention masks across the different frequency bands suggests that signals more relevant to human emotions may be active in the high frequency bands γ (31-50 Hz). Finally, we try to use the multi-task loss function Lc to force the approximation of the intermediate feature vectors of the visual and EEG modalities, with the aim of using the knowledge of the visual modalities to improve the performance of the EEG network model. The mean recognition accuracy and standard deviation of the proposed method on the two multimodal sentiment datasets DEAP and MAHNOB-HCI (arousal, valence) were 96.75 ± 1.75, 96.86 ± 1.33; 97.55 ± 1.51, 98.37 ± 1.07, better than those of the state-of-the-art multimodal recognition approaches. The experimental results show that starting from the multimodal information, the facial video frames and electroencephalogram (EEG) signals of the subjects are used as inputs to the emotion recognition network, which can enhance the stability of the emotion network and improve the recognition accuracy of the emotion network. In addition, in future work, we will try to utilize sparse matrix methods and deep convolutional networks to improve the performance of multimodal emotion networks.

Gestures recognition based on multimodal fusion by using 3D CNNs

Gestures have long been recognized as an interaction technique that can provide a more natural, creative, and intuitive way to communicate with computers. However, some existing difficulties include the high probability that the same type of movement done at different speeds will be recognized as a different category of movement; cluttered, occluded, and low-resolution backgrounds; and the near-impossibility of fusing different types of features. To this end, we propose a novel framework for integrating different scales of RGB and motion skeletons to obtain higher recognition accuracy using multiple features. Specifically, we provide a network architecture that combines a three-dimensional convolutional neural network (3DCNN) and post-fusion to better embed different features. Also, we combine RGB and motion skeleton information at different scales to mitigate speed and background issues. Experiments on several gesture recognition public datasets show desirable results, validating the superiority of the proposed gesture recognition method. Finally, we do a human-computer interaction experiment to prove its practicality.

Automating quality control in cardiac magnetic resonance: Artificial intelligence for discriminative assessment of planning and motion artifacts and real-time reacquisition guidance

BackgroundAccurate measurements from cardiovascular magnetic resonance (CMR) images require precise positioning of scan planes and elimination of motion artifacts from arrhythmia or breathing. Unidentified or incorrectly managed artifacts degrade image quality, invalidate clinical measurements, and decrease diagnostic confidence. Currently, radiographers must manually inspect each acquired image to confirm diagnostic quality and decide whether reacquisition or a change in sequences is warranted. We aimed to develop artificial intelligence (AI) to provide continuous quality scores across different quality domains, and from these, determine whether cines are clinically adequate, require replanning, or warrant a change in protocol. MethodsA three-dimensional convolutional neural network was trained to predict cine quality graded on a continuous scale by a level 3 CMR expert, focusing separately on planning and motion artifacts. It incorporated four distinct output heads for the assessment of image quality in terms of (a, b, c) 2-, 3- and 4-chamber misplanning, and (d) long- and short-axis arrhythmia/breathing artifact. Backpropagation was selectively performed across these heads based on the labels present for each cine. Each image in the testing set was reported by four level 3 CMR experts, providing a consensus on clinical adequacy. The AI's assessment of image quality and ability to identify images requiring replanning or sequence changes were evaluated with Spearman’s rho and the area under receiver operating characteristic curve (AUROC), respectively. ResultsA total of 1940 cines across 1387 studies were included. On the test set of 383 cines, AI-judged image quality correlated strongly with expert judgment, with Spearman’s rho of 0.84, 0.84, 0.81, and 0.81 for 2-, 3- and 4-chamber planning quality and the extent of arrhythmia or breathing artifacts, respectively. The AI also showed high efficacy in flagging clinically inadequate cines (AUROC 0.88, 0.93, and 0.93 for identifying misplanning of 2-, 3- and 4-chamber cines, and 0.90 for identifying movement artifacts). ConclusionAI can assess distinct domains of CMR cine quality and provide continuous quality scores that correlate closely with a consensus of experts. These ratings could be used to identify cases where reacquisition is warranted and guide corrective actions to optimize image quality, including replanning, prospective gating, or real-time imaging.

Stressed vegetation identification under natural gas micro-leakage from hyperspectral images using stacked autoencoder and multi-scale three-dimensional convolutional neural network

Stressed vegetation identification under natural gas micro-leakage from hyperspectral images using stacked autoencoder and multi-scale three-dimensional convolutional neural network

An acupuncture manipulation classification system based on three-axis attitude sensor and computer vision.

To explore the action characteristics of acupuncture manipulations by combining visual and sensor technique, so as to improve the identification and classification accuracy of acupuncture manipulations and to quantificate the classifiations. In this paper, the time domain features of acupuncture physical parameters and dynamic gesture features in the video of acupuncture manipulations are combined together to identify and classify acupuncture techniques. The acupuncture needle manipulation processes of 2 acupuncture experts and 3 young acupuncturists were selected as the study objects. The collected data included 4 basic manipulation techniques：lifting-thrusting reinforcing, lifting- thrusting reducing, twisting reinforcing and twisting reducing methods, all of which were performed by right-handed doctors. During acupuncture manipulation, a three-axis attitude sensor was used to acquire finger moving acceleration velocity and needle-rotating angle velocity, followed by analyzing the parameters of hand-moving velocity, amplitude, strength and angle. The mapping relationship among physical parameters and different manipulating methods was formed in time domain. The computer vision technology was employed to extract the spatio-temporal features of the acupuncture manipulation video images, and a hybrid model of three-dimensional convolutional neural network (3D CNN) and long- and short-term memory (LSTM) neural network were used for the recognition and classification of dynamic gestures of hand in acupuncture manipulation videos. Then the time-domain features of physical parameters were combined with the dynamic gestures in the classification process, with the manipulation classification realized. In performing the lift-thrusting reinforcing method, the needle insertion speed was faster and the force was larger, while the needle lifting speed was slower and the force was smaller. And in performing the lift-thrusting reducing method, the needle lifting speed was faster, the force was stronger, and the needle insertion speed was slower and the force was smaller. In the performance of twisting reinforcing, the leftward twisting force was bigger and the rotation amplitude was larger, while in performing the reducing method, the rightward twisting force was larger and the rotation amplitude was larger. When using the mean value of time of acceleration, speed, and amplitude as the basis of discrimination, the accuracy rates of lifting-thrusting reinforcing and reducing were 95.56% and 93.33%, while those of the two twisting manipulations were 95.56% and 91.11%, respectively. Compared with the classification method that only uses the sensor to obtain the manipulation information, the recognition accuracy was significantly improved. The acupuncture manipulation classification system can achieve quantitative analysis of physical parameters and dynamic recognition of acupuncture techniques, providing a certain foundation for the quantification and inheritance of acupuncture techniques.

Accelerating phase-field simulation of three-dimensional microstructure evolution in laser powder bed fusion with composable machine learning predictions

Phase-field (PF) modeling is a versatile physics-based computational method that has been used to simulate the evolution of microstructures. The PF method can produce accurate microstructures but suffers from a high computational cost, limiting its use in length scales relevant to additive manufacturing. Using small-scale PF simulations as training data, we trained a surrogate machine learning (ML) model as a computationally cheaper alternative. We use a three-dimensional (3D) U-Net convolutional neural network and learn microstructure evolution in a supervised fashion. With initial microstructure and thermal history as inputs, the ML model can predict the resulting grain orientations at a high accuracy compared to the PF model. Computationally, the ML model is orders of magnitudes faster than the direct PF simulation in a GPU implementation, and scales favorably with increasing number of cells. By spatio-temporally composing multiple ML model predictions at the small-scale, we demonstrate a large-scale 64-layer simulation of a 2 mm × 2 mm × 2 mm cube for a powder bed fusion additive manufacturing process. The ML results revealed a mixture of equiaxed, columnar, and curved grains, comparable to experimental observations. Microstructures resulting from different toolpath strategies, and lack of fusion defects are also demonstrated. This approach paves the way for microstructure-driven process design.

Development and validation of a three-dimensional deep learning-based system for assessing bowel preparation on colonoscopy video.

The performance of existing image-based training models in evaluating bowel preparation on colonoscopy videos was relatively low, and only a few models used external data to prove their generalization. Therefore, this study attempted to develop a more precise and stable AI system for assessing bowel preparation of colonoscopy video. We proposed a system named ViENDO to assess the bowel preparation quality, including two CNNs. First, Information-Net was used to identify and filter out colonoscopy video frames unsuitable for Boston bowel preparation scale (BBPS) scoring. Second, BBPS-Net was trained and tested with 5,566 suitable short video clips through three-dimensional (3D) convolutional neural network (CNN) technology to detect BBPS-based insufficient bowel preparation. Then, ViENDO was applied to complete withdrawal colonoscopy videos from multiple centers to predict BBPS segment scores in clinical settings. We also conducted a human-machine contest to compare its performance with endoscopists. In video clips, BBPS-Net for determining inadequate bowel preparation generated an area under the curve of up to 0.98 and accuracy of 95.2%. When applied to full-length withdrawal colonoscopy videos, ViENDO assessed bowel cleanliness with an accuracy of 93.8% in the internal test set and 91.7% in the external dataset. The human-machine contest demonstrated that the accuracy of ViENDO was slightly superior compared to most endoscopists, though no statistical significance was found. The 3D-CNN-based AI model showed good performance in evaluating full-length bowel preparation on colonoscopy video. It has the potential as a substitute for endoscopists to provide BBPS-based assessments during daily clinical practice.

Wireless capsule endoscopy multiclass classification using three-dimensional deep convolutional neural network model

BackgroundWireless capsule endoscopy (WCE) is a patient-friendly and non-invasive technology that scans the whole of the gastrointestinal tract, including difficult-to-access regions like the small bowel. Major drawback of this technology is that the visual inspection of a large number of video frames produced during each examination makes the physician diagnosis process tedious and prone to error. Several computer-aided diagnosis (CAD) systems, such as deep network models, have been developed for the automatic recognition of abnormalities in WCE frames. Nevertheless, most of these studies have only focused on spatial information within individual WCE frames, missing the crucial temporal data within consecutive frames.MethodsIn this article, an automatic multiclass classification system based on a three-dimensional deep convolutional neural network (3D-CNN) is proposed, which utilizes the spatiotemporal information to facilitate the WCE diagnosis process. The 3D-CNN model fed with a series of sequential WCE frames in contrast to the two-dimensional (2D) model, which exploits frames as independent ones. Moreover, the proposed 3D deep model is compared with some pre-trained networks. The proposed models are trained and evaluated with 29 subject WCE videos (14,691 frames before augmentation). The performance advantages of 3D-CNN over 2D-CNN and pre-trained networks are verified in terms of sensitivity, specificity, and accuracy.Results3D-CNN outperforms the 2D technique in all evaluation metrics (sensitivity: 98.92 vs. 98.05, specificity: 99.50 vs. 86.94, accuracy: 99.20 vs. 92.60). In conclusion, a novel 3D-CNN model for lesion detection in WCE frames is proposed in this study.ConclusionThe results indicate the performance of 3D-CNN over 2D-CNN and some well-known pre-trained classifier networks. The proposed 3D-CNN model uses the rich temporal information in adjacent frames as well as spatial data to develop an accurate and efficient model.

Evaluating the Performance of Mobile-Convolutional Neural Networks for Spatial and Temporal Human Action Recognition Analysis

Human action recognition is a computer vision task that identifies how a person or a group acts on a video sequence. Various methods that rely on deep-learning techniques, such as two- or three-dimensional convolutional neural networks (2D-CNNs, 3D-CNNs), recurrent neural networks (RNNs), and vision transformers (ViT), have been proposed to address this problem over the years. Motivated by the fact that most of the used CNNs in human action recognition present high complexity, and the necessity of implementations on mobile platforms that are characterized by restricted computational resources, in this article, we conduct an extensive evaluation protocol over the performance metrics of five lightweight architectures. In particular, we examine how these mobile-oriented CNNs (viz., ShuffleNet-v2, EfficientNet-b0, MobileNet-v3, and GhostNet) execute in spatial analysis compared to a recent tiny ViT, namely EVA-02-Ti, and a higher computational model, ResNet-50. Our models, previously trained on ImageNet and BU101, are measured for their classification accuracy on HMDB51, UCF101, and six classes of the NTU dataset. The average and max scores, as well as the voting approaches, are generated through three and fifteen RGB frames of each video, while two different rates for the dropout layers were assessed during the training. Last, a temporal analysis via multiple types of RNNs that employ features extracted by the trained networks is examined. Our results reveal that EfficientNet-b0 and EVA-02-Ti surpass the other mobile-CNNs, achieving comparable or superior performance to ResNet-50.