Spatio-temporal Convolutional Neural Network Research Articles

Short-term load forecasting of regional integrated energy system based on spatio-temporal convolutional graph neural network

Accurate short-term load forecasting has significant reference value for the preliminary planning and subsequent coordinated control of regional integrated energy systems. However, main limitations are: (1) Insufficient exploration of the complex correlations among diverse loads and between loads and meteorological factors; and (2) Feature information extraction only from the time dimension, neglecting the spatial dimension, resulting in incomplete feature information extraction. This paper proposes a new method for short-term load forecasting, namely the Attention Spatial–temporal Graph Convolutional Network (ATSTGCN) method. Firstly, the Grey Relational Analysis (GRA) method is used to analyze the correlation between diverse loads, meteorological features, and load features, constructing the input data for the graph neural network. Then, synchronous graph convolution is introduced to extract information between features. Next, the spatio-temporal attention mechanism is incorporated into the synchronous graph convolution to capture spatio-temporal features and reduce information loss using a spatio-temporal window. Finally, a short-term load forecasting method based on ATSTGCN for regional integrated energy systems is constructed. Extensive simulation experiments are conducted on the publicly available dataset from Arizona State University’s online system. The experimental results show that compared to other methods, the proposed method improves MAPE, MAE, and RMSE by 5.6%, 11.7%, and 11.6%.

A Spatiotemporal Deep Learning-Based Smart Discovery Approach for Marine Pollution Incidents from the Data-Driven Perspective

Marine pollution incidents (MPI) are often a dynamic process of time and space interaction. Currently, the monitoring of MPI is basically realized by manual analysis from expert experience. Such working mode has an obvious time lag, and is not useful for timely disposal. As a result, intelligent algorithms that can make quick discovery for MPI from massive monitoring data remain a practical demand in this field. Considering that monitoring elements generally have multi-dimensional characteristics and spatiotemporal causal relationships, this work develops a spatiotemporal deep learning-based smart discovery approach for MPI from the data-driven perspective. In particular, a systematic preprocessing workflow is developed for the spatiotemporal monitoring data, which facilitates following feature extraction. Then, a spatiotemporal convolution neural network structure is developed to extract features from original spatiotemporal monitoring data. On this basis, the discovery results of MPI can be output via neural computing structures. Taking the polluting marine oil spill incident in the Bohai Sea in eastern China as a case study, this work carries out a simulation application and its result analysis. The obtained simulation results can reveal the proper performance of the proposal.

Spatiotemporal Deep Learning-Based Cine Loop Quality Filter for Handheld Point-of-Care Echocardiography.

The reliability of automated image interpretation of point-of-care (POC) echocardiography scans depends on the quality of the acquired ultrasound data. This work reports on the development and validation of spatiotemporal deep learning models to assess the suitability of input ultrasound cine loops collected using a handheld echocardiography device for processing by an automated quantification algorithm (e.g. ejection fraction estimation). POC echocardiograms (n=885 DICOM cine loops from 175 patients) from two sites were collected using a handheld ultrasound device and annotated for image quality at the frame-level. Attributes of high-quality frames for left ventricular (LV) quantification included a temporally-stable LV, reasonable coverage of LV borders, and good contrast between the borders and chamber. Attributes of low-quality frames included temporal instability of the LV and/or imaging artifacts (e.g., lack of contrast, haze, reverberation, acoustic shadowing). Three different neural network architectures were investigated - (a) frame-level convolutional neural network (CNN) which operates on individual echo frames (VectorCNN), (b) single-stream sequence-level CNN which operates on a sequence of echo frames (VectorCNN+LSTM) and (c) two-stream sequence-level CNNs which operate on a sequence of echo and optical flow frames (VectorCNN+LSTM+Average, VectorCNN+LSTM+MinMax, and VectorCNN+LSTM+ConvPool). Evaluation on a sequestered test dataset containing 76 DICOM cine loops with 16,914 frames showed that VectorCNN+LSTM can effectively utilize both spatial and temporal information to regress the quality of an input frame (accuracy: 0.925, sensitivity = 0.860, specificity = 0.952), compared to the frame-level VectorCNN that only utilizes spatial information in that frame (accuracy: 0.903, sensitivity = 0.791, specificity = 0.949). Furthermore, an independent sample t-test indicated that the cine loops classified to be of adequate quality by the VectorCNN+LSTM model had a statistically significant lower bias in the automatically estimated EF (mean bias = - 3.73 ± 7.46 %, versus a clinically obtained reference EF) compared to the loops classified as inadequate (mean bias = -15.92 ± 12.17 %) (p = 0.007). Thus, cine loop stratification using the proposed spatiotemporal CNN model improves the reliability of automated point-of-care echocardiography image interpretation.

A multi-modal driver emotion dataset and study: Including facial expressions and synchronized physiological signals

To address the limitations of databases in the field of emotion recognition and to cater to the trend of integrating data from multiple sources, we have established a multi-modal emotional dataset based on spontaneous expression of drivers. By selecting emotional induction materials and inducing emotions before each driving task, facial expression videos and synchronous physiological signals of the drivers during driving were collected. The dataset includes records of 64 participants under five different emotions (neutral, happy, angry, sad, and fear), and the emotional valence, arousal, and peak time of all participants in each driving task were recorded. To analyze the dataset, spatio-temporal convolutional neural networks were designed to analyze the different modalities of data with varying durations in the dataset, aiming to investigate their performance in emotion recognition. The results demonstrate that the fusion of multi-modal data significantly improves the accuracy of driver's emotion recognition, with accuracy increases of 11.28% and 6.83% compared to using only facial video signals or physiological signals, respectively. Therefore, the publication and analysis of multi-modal emotional data for driving scenarios is crucial to support further research in the fields of multimodal perception and intelligent transportation engineering.

Spatiotemporal Pyramidal CNN with Depth-Wise Separable Convolution for Eye Blinking Detection in the Wild

Eye blinking detection in the wild plays an essential role in deception detection, driving fatigue detection, etc. Despite the fact that numerous attempts have already been made, the majority of them have encountered difficulties, such as the derived eye images having different resolutions as the distance between the face and the camera changes; or the requirement of a lightweight detection model to obtain a short inference time in order to perform in realtime. In this research, two problems are addressed: how the eye blinking detection model can learn efficiently from different resolutions of eye pictures in diverse conditions; and how to reduce the size of the detection model for faster inference time. We propose to utilize upsampling and downsampling the input eye images to the same resolution as one potential solution for the first problem, then find out which interpolation method can result in the highest performance of the detection model. For the second problem, although a recent spatiotemporal convolutional neural network used for eye blinking detection has a strong capacity to extract both spatial and temporal characteristics, it remains having a high number of network parameters, leading to high inference time. Therefore, using Depth-wise Separable Convolution rather than conventional convolution layers inside each branch is considered in this paper as a feasible solution.

A novel spatiotemporal 3D CNN framework with multi-task learning for efficient structural damage detection

In recent years, convolutional neural networks (CNNs) have demonstrated promising results in detecting structural damage. However, their architectures often overlook spatial and temporal effects simultaneously. This limitation can result in the loss of valuable information and an incapability to fully capture the complexity of the data, ultimately leading to reduced accuracy and suboptimal performance. This study proposes an intuitive three-dimensional CNN architecture that takes into account vibration history along with sensor spatial relations based on their relative positions. Furthermore, a multi-task learning (MTL) approach is suggested, which is a powerful approach for performing multiple tasks with a single network. The proposed 3D CNN method has been employed to detect single and double damage cases in an experimental steel frame through conventional classification alongside the transfer learning (TL). Moreover, MTL is used to detect single and double damage scenarios with a single unified network, which evaluates damage presence in separate tasks. The 3D CNN fulfilled state-of-the-art performance and 100% accuracy in detecting structural damage in almost all experiments. Additionally, the MTL model achieved promising results even in the presence of severe imbalanced classes of data. Furthermore, it was observed that the utilization of TL resulted in a notable reduction of computation time by 68% and the number of trainable parameters by 90% with the same level of accuracy in double-damage cases.

Real driving environment EEG-based detection of driving fatigue using the wavelet scattering network

BackgroundDriving fatigue is one of the main factors leading to traffic accidents. So, it is necessary to detect driver fatigue accurately and quickly. New methodTo precisely detect driving fatigue in a real driving environment, this paper adopts a classification method for driving fatigue based on the wavelet scattering network (WSN). Firstly, electroencephalogram (EEG) signals of 12 subjects in the real driving environment are collected and categorized into two states: fatigue and awake. Secondly, the WSN algorithm extracts wavelet scattering coefficients of EEG signals, and these coefficients are used as input in support vector machine (SVM) as feature vectors for classification. ResultsThe results showed that the average classification accuracy of 12 subjects reached 99.33%; the average precision rate reached 99.28%; the average recall rate reached 98.27%; the average F1 score reached 98.74%; and the average classification accuracy of the public data set SEED-VIG reached 99.39%. The average precision, recall rate and F1 score reached 99.27%, 98.41% and 98.83% respectively. Comparison with existing methodsIn addition, the WSN algorithm is compared with traditional convolutional neural network (CNN), Sparse-deep belief networks (SDBN), Spatio-temporal convolutional neural networks (STCNN), Long short-term memory (LSTM), and other methods, and it is found that WSN has higher classification accuracy. ConclusionFurthermore, this method has good versatility, providing excellent recognition effect on small sample data sets, and fast running time, making it convenient for real-time online monitoring of driver fatigue. Therefore, the WSN algorithm is promising in efficiently detecting driving fatigue state of drivers in real environments, contributing to improved traffic safety.

Satellite detection of canopy-scale tree mortality and survival from California wildfires with spatio-temporal deep learning

Mapping forest disturbances is paramount to carbon monitoring, estimating environmental drivers, and developing strategies to enhance forest resilience. Existing forest change products from Landsat and Sentinel-2 have improved our understanding of large-scale disturbance patterns; however, their relatively coarse spatial resolution (10 to 30-m) leads to the use of mixed pixels which constrains their application for detecting heterogeneous survival or mortality outcomes occurring at the level of individual trees or canopies. PlanetScope multispectral imagery at 3-m and near-daily frequency offers new capabilities to detect and monitor diverse tree mortality patterns following disturbance across landscapes. This research proposes a framework to detect canopy-scale (3 × 3-m) tree/shrub mortality and survival using PlanetScope monthly time series. A 3D Spatio-Temporal Convolutional Neural Network (ST-CNN) deep learning model was designed to fully utilize the spatial context and the temporal change unique to canopy survival and mortality from the PlanetScope time series. As a crucial component for training robust and scalable deep learning models, a large set of labels was collected via a semi-automatic workflow by combining pre-disturbance lidar crown segmentation and post-disturbance aerial imagery interpretation. We applied the framework to detect canopy-scale mortality and survival following 15 large wildfires in California from 2018–2021. We sampled 1,176 384 × 384-m scenes from burned areas with pre-fire aerial lidar, containing >1.8M tree and shrub canopy polygons labeled as dead or alive following wildfire. Evaluated with an independent testing dataset, the optimized ST-CNN model detects heterogeneous patterns representing survival and mortality outcomes at 3-m resolution which accurately align with observed/labeled data. Tree mortality detection accuracy was high and stable in the Sierra Nevada and North Coast Mountains ecoregions (user’s = 83%–86%; producer’s = 81%–82%), but decreased slightly within the sparser Central Foothills and South Coast and Mountains (user’s = 77%–81%; producer’s = 58%–61%) often due to confusion between shrub and tree mortality. Producer’s accuracy of tree mortality and survival increased with canopy height and remained stable (>75%) on canopies taller than 11-m. Further, a sensitivity analysis demonstrates the performance benefits of using spatial and/or temporal convolutions in the ST-CNN architecture for model prediction. Lastly, we demonstrate the scalability of the ST-CNN for regional-scale application on all large 2020 wildfires in California (∼1.6 Mha burn area). The wall-to-wall post-fire maps showed an overall 3-m tree mortality rate of 58.8%, ranging from 32% to 94% among individual fires. The trained ST-CNN provides an ecologically detailed estimation of the pre-disturbance forest composition (trees, shrubs, non-woody) and their outcomes (survival or mortality) post-disturbance. These data will improve higher resolution monitoring and assessment of forest disturbance impacts, allow for better understanding of forest vulnerability, and support forest management strategies and actions.

Transforming unmanned pineapple picking with spatio-temporal convolutional neural networks

Automated pineapple harvesting has emerged as a prominent prospective development within the agricultural domain. Nevertheless, the intricate growth conditions that pineapples encounter in the field, such as inadequate light, overexposure, obstructions caused by fruit leaves, or the overlapping of fruits, pose substantial challenges to the accuracy and robustness of traditional real-time detection algorithms. In recent times, Transformer models, when applied to computer vision, have exhibited commendable performance, underscoring their potential for target detection in smart agricultural applications. In this report, we propose a spatio-temporal convolutional neural network model that leverages the shifted window Transformer fusion region convolutional neural network model for the purpose of detecting pineapple fruits. Our study includes a comparative analysis of these results and those obtained through the utilization of conventional models. Additionally, we investigate the influence of various aspects of data preparation, including image resolution, object size, and object complexity, on the ultimate pineapple detection outcomes. Experimental findings elucidate that, in the case of detecting a single-category target like a pineapple, the employment of 2000 annotated supervised data points yields the optimal detection accuracy. Further augmenting the size of the training dataset does not yield any significant improvement in detection accuracy. Furthermore, images of pineapples captured from greater distances encompass smaller targets and an increased number of pineapple instances, rendering them more intricate and challenging to accurately detect. In summary, our study employs the spatio-temporal convolutional neural network model to attain pineapple detection with an impressive accuracy rate of 92.54% and an average inference time of 0.163 s, thus affirming the efficacy of our developed model in achieving superior detection results.

A novel transfer CNN with spatiotemporal input for accurate nuclear power fault diagnosis under different operating conditions

Deep Neural Network (DNN) models, recognized for their exceptional feature extraction and end-to-end diagnostic capabilities, have gained considerable attention from researchers in the realm of nuclear power fault diagnosis. However, existing deep-learning based diagnostic technologies present certain challenges when applied to practical engineering scenarios. A significant issue is the limited adaptability of data-based machine learning diagnostic methods to dynamic and evolving operational conditions, leading to inconsistent accuracy under different operating parameters. To solve this problem, a spatiotemporal Convolutional Neural Network (CNN) model is proposed. A novel input scheme, considering both spatial and temporal aspects, is specially designed for nuclear power plant fault diagnosis and MLP convolutional layer are used to improve the feature extraction capability of the model. Finally, the performance of the model is analyzed through a series of experiments based on simulation data and data visualization. The results show that the proposed method outperforms traditional CNNs in diagnosing nuclear power system faults with higher accuracy. Furthermore, by employing a transfer learning strategy, the model exhibits improved speed for cross-operating condition fault diagnosis.

Deep Learning for Metro Short-Term Origin-Destination Passenger Flow Forecasting Considering Section Capacity Utilization Ratio

Origin-destination (OD) short-term passenger flow forecasting (OD STPFF) in urban rail transit (URT) is essential for developing timely network measures. The capacity utilization ratios of critical sections are key factors in developing these measures. The OD pairs passing through critical sections require a higher prediction accuracy than others; however, most studies have raised equal concerns on the prediction accuracy of each OD pair, namely, prediction at the network level. To address this problem, we raise heterogeneous time-variant concerns on OD pairs and employ an operation-oriented deep-learning architecture called the spatiotemporal convolutional neural network (STCNN) for realizing short-term OD passenger flow prediction. The architecture contains OD pair importance calculation, lagged spatiotemporal relationship construction, lagged spatiotemporal learning, real-time information learning, and sequential-temporal learning blocks. To this end, critical OD pairs are ascertained first, and the topological lagged spatiotemporal relationship among critical OD pairs are constructed and then normalized into grid-shaped data. The third block utilizes a convolutional neural network (CNN) to learn the grid-shaped lagged spatiotemporal feature and real-time information in parallel. A temporal convolutional neural network (TCN) is utilized for learning the sequential-temporal feature at last. Further, we design a time-varying weighted masked loss function to jointly reinforce the concerns on critical OD pairs during model training. Finally, we test the proposed STCNN and its components on a field dataset from Chengdu Metro. Although the proposed STCNN performs only slightly better at the network level than the other models, it outperforms state-of-the-art methods with significant superiority on critical OD pairs.

3D-STCNN: Spatiotemporal Convolutional Neural Network Based on EEG 3D Features for Detecting Driving Fatigue

Fatigue driving has become one of the main causes of traffic accidents, and driving fatigue detection based on electroencephalogram (EEG) can effectively evaluate the driver’s mental state and avoid the occurrence of traffic accidents. This article evaluates a feature extraction method for extracting multiple features of EEG signals and establishes a spatiotemporal convolutional neural network (STCNN) to detect driver fatigue. Firstly, we constructed a three-dimensional feature of the EEG signal, which includes the frequency domain, time domain, and spatial features of the EEG signal. Then, we use STCNN for fatigue state classification. STCNN is composed of an attention time network based on attention mechanism and an attention convolutional neural network based on attention mechanism. In addition, we conducted fatigue driving experiments and collected EEG signals from 14 subjects in both awake and fatigued states, ultimately collecting EEG data under three different driving task loads. We conducted extensive experiments on this basis and compared the effectiveness of STCNN and six competitive methods. The results show that the classification accuracy of STCNN is 87.55%, which can effectively detect the fatigue status of drivers.

An advanced spatio-temporal convolutional recurrent neural network for storm surge predictions

An advanced spatio-temporal convolutional recurrent neural network for storm surge predictions

Simulation of video image detection in leisure sports tourism industry based on convolutional neural network

Due to the continuous development of information technology, emerging technologies such as 5G networks, Turing devices and deep learning are widely used in people's lives. At present, the sports and tourism industry has become a pillar industry supporting the national economy. Sports events have attracted many people's attention, and people are willing to devote themselves to sports activities. Participants and visitors are also increasing day by day. The article also conducts research on the rapid detection of human video behavior, obtains more representative features for this demand. This research is of great help to the development of the industry. This allows the network to manipulate the input video behavior at different scales and extract deeper video information. Therefore, this paper proposes another 3D convolutional neural network model that combines a multi-level pyramid network with an attention mechanism. The visual attributes in the video behavior data set are explicitly learned to refine the classification of similar features. This paper also proposes an integrated neural network based on visual attribute enhancement. It also shows good adaptability, and can complete video data processing based on time series. The extraction of time and space functions is realized by the two-stream neural network method, the two networks learn the corresponding pixel ratios, and it is not easy to lose the clues of the action recognition. This paper also proposes a spatiotemporal convolutional neural network video for human body recognition algorithm, which is applied to the leisure tourism industry according to the specific analysis of the above method, and promotes the development of the industry.

Deep Learning-Based Prediction of Right Ventricular Ejection Fraction Using 2D Echocardiograms

BackgroundEvidence has shown the independent prognostic value of right ventricular (RV) function, even in patients with left-sided heart disease. The most widely used imaging technique to measure RV function is echocardiography; however, conventional 2-dimensional (2D) echocardiographic assessment is unable to leverage the same clinical information that 3-dimensional (3D) echocardiography-derived right ventricular ejection fraction (RVEF) can provide. ObjectivesThe authors aimed to implement a deep learning (DL)–based tool to estimate RVEF from 2D echocardiographic videos. In addition, they benchmarked the tool's performance against human expert reading and evaluated the prognostic power of the predicted RVEF values. MethodsThe authors retrospectively identified 831 patients with RVEF measured by 3D echocardiography. All 2D apical 4-chamber view echocardiographic videos of these patients were retrieved (n = 3,583), and each subject was assigned to either the training or the internal validation set (80:20 ratio). Using the videos, several spatiotemporal convolutional neural networks were trained to predict RVEF. The 3 best-performing networks were combined into an ensemble model, which was further evaluated in an external data set containing 1,493 videos of 365 patients with a median follow-up time of 1.9 years. ResultsThe ensemble model predicted RVEF with a mean absolute error of 4.57 percentage points in the internal and 5.54 percentage points in the external validation set. In the latter, the model identified RV dysfunction (defined as RVEF <45%) with an accuracy of 78.4%, which was comparable to an expert reader’s visual assessment (77.0%; P = 0.678). The DL-predicted RVEF values were associated with major adverse cardiac events independent of age, sex, and left ventricular systolic function (HR: 0.924 [95% CI: 0.862-0.990]; P = 0.025). ConclusionsUsing 2D echocardiographic videos alone, the proposed DL-based tool can accurately assess RV function, with similar diagnostic and prognostic power as 3D imaging.

Automated segmentation of long and short axis DENSE cardiovascular magnetic resonance for myocardial strain analysis using spatio-temporal convolutional neural networks

BackgroundCine Displacement Encoding with Stimulated Echoes (DENSE) facilitates the quantification of myocardial deformation, by encoding tissue displacements in the cardiovascular magnetic resonance (CMR) image phase, from which myocardial strain can be estimated with high accuracy and reproducibility. Current methods for analyzing DENSE images still heavily rely on user input, making this process time-consuming and subject to inter-observer variability. The present study sought to develop a spatio-temporal deep learning model for segmentation of the left-ventricular (LV) myocardium, as spatial networks often fail due to contrast-related properties of DENSE images. Methods2D + time nnU-Net-based models have been trained to segment the LV myocardium from DENSE magnitude data in short- and long-axis images. A dataset of 360 short-axis and 124 long-axis slices was used to train the networks, from a combination of healthy subjects and patients with various conditions (hypertrophic and dilated cardiomyopathy, myocardial infarction, myocarditis). Segmentation performance was evaluated using ground-truth manual labels, and a strain analysis using conventional methods was performed to assess strain agreement with manual segmentation. Additional validation was performed using an externally acquired dataset to compare the inter- and intra-scanner reproducibility with respect to conventional methods. ResultsSpatio-temporal models gave consistent segmentation performance throughout the cine sequence, while 2D architectures often failed to segment end-diastolic frames due to the limited blood-to-myocardium contrast. Our models achieved a DICE score of 0.83 ± 0.05 and a Hausdorff distance of 4.0 ± 1.1 mm for short-axis segmentation, and 0.82 ± 0.03 and 7.9 ± 3.9 mm respectively for long-axis segmentations. Strain measurements obtained from automatically estimated myocardial contours showed good to excellent agreement with manual pipelines, and remained within the limits of inter-user variability estimated in previous studies. ConclusionSpatio-temporal deep learning shows increased robustness for the segmentation of cine DENSE images. It provides excellent agreement with manual segmentation for strain extraction. Deep learning will facilitate the analysis of DENSE data, bringing it one step closer to clinical routine.

Grid-Related Fine Action Segmentation Based on an STCNN-MCM Joint Algorithm during Smart Grid Training

Smart grid-training systems enable trainers to achieve the high safety standards required for power operation. Effective methods for the rational segmentation of continuous fine actions can improve smart grid-training systems, which is of great significance to sustainable power-grid operation and the personal safety of operators. In this paper, a joint algorithm of a spatio-temporal convolutional neural network and multidimensional cloud model (STCNN-MCM) is proposed to complete the segmentation of fine actions during power operation. Firstly, the spatio-temporal convolutional neural network (STCNN) is used to extract action features from the multi-sensor dataset of hand actions during power operation and to predict the next moment’s action to form a multi-outcome dataset; then, a multidimensional cloud model (MCM) is designed based on the motion features of the real power operation; finally, the corresponding probabilities are obtained from the distribution of the predicted data in the cloud model through the multi-outcome dataset for action-rsegmentation point determination. The results show that STCNN-MCM can choose the segmentation points of fine actions in power operation in a relatively efficient way, improve the accuracy of action division, and can be used to improve smart grid-training systems for the segmentation of continuous fine actions in power operation.

Abstract 14356: Machine Learning for Detection of Presence and Severity of Aortic Stenosis From B-mode Ultrasound Images: Results of a Blinded Clinical Trial

Introduction: Aortic stenosis (AS) is common, with &gt;= moderate severity ~10% above age 75. Despite new therapies, many with AS go undiagnosed due to lack of echo availability and untreated due to improper interpretation. To expand AS diagnosis, we developed an artificial intelligence (AI) algorithm to characterize AS from routine B-mode echo exams without Doppler (AutoAS). Methods: 80,000 de-identified clinical echoes were assembled spanning none, mild, moderate, and severe AS. About 30,000 of these were used to train a convolutional neural network (CNN) using labels derived from a cluster analysis of maximal jet velocity, mean AS gradient, and aortic valve area (AVA). Once labeled, we extracted all available B-mode clips from parasternal long-axis, short axis at aortic level, and apical 5-chamber views, irrespective of image quality. A spatiotemporal CNN was trained to return a probability distribution for none/mild/moderate/severe AS. To assess performance of the model, a hold-out test set of echoes uniformly distributed over patient sex and AS severity was used. Three Level 3 echo trained cardiologists blindly read the complete Doppler echoes and agreed to call AS severity by AVA (&lt; 1 cm2, severe; 1.0-1.5 cm2, moderate; &gt; 1.5 cm2 but restricted, mild; no restriction, none). Results were also dichotomized into none/mild and moderate/severe AS. Results: 40 cases comprised this initial test set. Consistency among the readers was outstanding, with κ = 0.95. The table shows the confusion matrix between AutoAS and panel reads, with excellent accuracy (κ = 0.90). Dichotomous results were also outstanding with sensitivity to detect moderate/severe AS of 91% and specificity of 94%. Conclusions: AI can identify AS from 3 B-mode views and characterize AS severity with accuracy highly congruent to an expert panel using full Doppler data. This may allow AS to be detected in low-resource and screening settings. Further assessment with a larger hold-out test set is planned.

Predicting treatment-specific lesion outcomes in acute ischemic stroke from 4D CT perfusion imaging using spatio-temporal convolutional neural networks.

For the diagnosis and precise treatment of acute ischemic stroke, predicting the final location and volume of lesions is of great clinical interest. Current deep learning-based prediction methods mainly use perfusion parameter maps, which can be calculated from spatio-temporal (4D) CT perfusion (CTP) imaging data, to estimate the tissue outcome of an acute ischemic stroke. However, this calculation relies on a deconvolution operation, an ill-posed problem requiring strong regularization and definition of an arterial input function. Thus, improved predictions might be achievable if the deep learning models were applied directly to acute 4D CTP data rather than perfusion maps. In this work, a novel deep spatio-temporal convolutional neural network is proposed for predicting treatment-dependent stroke lesion outcomes by making full use of raw 4D CTP data. By merging a U-Net-like architecture with temporal convolutional networks, we efficiently process the spatio-temporal information available in CTP datasets to make a tissue outcome prediction. The proposed method was evaluated on 147 patients using a 10-fold cross validation, which demonstrated that the proposed 3D+time model (mean Dice=0.45) significantly outperforms both a 2D+time variant of our approach (mean Dice=0.43) and a state-of-the-art method that uses perfusion maps (mean Dice=0.38). These results show that 4D CTP datasets include more predictive information than perfusion parameter maps, and that the proposed method is an efficient approach to make use of this complex data.

Deepfake attack prevention using steganography GANs.

Deepfakes are fake images or videos generated by deep learning algorithms. Ongoing progress in deep learning techniques like auto-encoders and generative adversarial networks (GANs) is approaching a level that makes deepfake detection ideally impossible. A deepfake is created by swapping videos, images, or audio with the target, consequently raising digital media threats over the internet. Much work has been done to detect deepfake videos through feature detection using a convolutional neural network (CNN), recurrent neural network (RNN), and spatiotemporal CNN. However, these techniques are not effective in the future due to continuous improvements in GANs. Style GANs can create fake videos with high accuracy that cannot be easily detected. Hence, deepfake prevention is the need of the hour rather than just mere detection. Recently, blockchain-based ownership methods, image tags, and watermarks in video frames have been used to prevent deepfake. However, this process is not fully functional. An image frame could be faked by copying watermarks and reusing them to create a deepfake. In this research, an enhanced modified version of the steganography technique RivaGAN is used to address the issue. The proposed approach encodes watermarks into features of the video frames by training an "attention model" with the ReLU activation function to achieve a fast learning rate. The proposed attention-generating approach has been validated with multiple activation functions and learning rates. It achieved 99.7% accuracy in embedding watermarks into the frames of the video. After generating the attention model, the generative adversarial network has trained using DeepFaceLab 2.0 and has tested the prevention of deepfake attacks using watermark embedded videos comprising 8,074 frames from different benchmark datasets. The proposed approach has acquired a 100% success rate in preventing deepfake attacks. Our code is available at https://github.com/shahidmuneer/deepfakes-watermarking-technique.