Spatiotemporal 3D Research Articles

Application of a deep learning algorithm for the diagnosis of HCC

Background & AimsHepatocellular carcinoma (HCC) is characterized by a high mortality rate. The Liver Imaging Reporting and Data System (LI-RADS) results in considerable proportions of indeterminate observations, rendering an accurate diagnosis difficult. MethodsWe developed four deep learning models for diagnosing HCC on computed tomography (CT) via a training-validation-testing approach. Thin-slice triphasic CT liver images and relevant clinical information were collected and processed for deep learning. HCC was diagnosed and verified via a 12-month clinical composite reference standard. CT observations among at-risk patients were annotated using LI-RADS. Diagnostic performance was assessed by internal validation and independent external testing. We conducted sensitivity analyses of different subgroups, deep learning explainability evaluation, and misclassification analysis. ResultsFrom 2,832 patients and 4,305 CT observations, the best-performing model was Spatio-Temporal 3D Convolution Network (ST3DCN), achieving area under curves (AUCs) of 0.919 (95%CI 0.903-0.935) and 0.901 (95%CI 0.879-0.924) at the observation (n=1077) and patient (n=685) levels respectively during internal validation, compared to 0.839 (95%CI 0.814-0.864) and 0.822 (95%CI 0.790-0.853) respectively for standard-of-care radiological interpretation. ST3DCN’s negative predictive values were 0.966 (95%CI 0.954-0.979) and 0.951 (95%CI 0.931-0.971) respectively. ST3DCN’s observation-level AUCs among at-risk patients, 2-5 cm observations and singular porto-venous phase analysis were 0.899 (95%CI 0.874-0.924), 0.872 (95%CI 0.838-0.909) and 0.912 (95%CI 0.895-0.929) respectively. In external testing (551/717 patients/observations), ST3DCN’s AUC was 0.901 (95%CI 0.877-0.924), non-inferior to radiological interpretation (AUC 0.900, 95%CI 0.877-923). ConclusionsST3DCN achieved strong, robust performance for accurate HCC diagnosis on CT. Deep learning can expedite and improve the diagnostic process of HCC. Impact and implicationsThe clinical applicability of deep learning in HCC diagnosis is potentially huge, especially considering the expected increase in the incidence and mortality of HCC in Eastern Asia and worldwide. Early diagnosis through deep learning can lead to earlier definitive management, particularly for at-risk patients. The model can be broadly deployed for patients undergoing a triphasic contrast CT scan of the liver to reduce the current high mortality rate of HCC.

Video anomaly detection based on hybrid attention mechanism

To improve the ability of video anomaly detection models to extract normal behavior features of samples and suppress abnormal behaviors, this paper proposes an unsupervised video anomaly detection model, which takes advantage of spatio-temporal feature fusion, storage module, attention mechanism, and 3D autoencoder model. The model utilizes autoencoder to capture scene feature maps to enhance anomaly feature extraction. These maps are merged with the original video frames, forming fundamental units constituting continuous sequences serving as the model's input. Moreover, the attention mechanism is integrated into the 3D convolutional neural network to strengthen the network's capability in extracting channel and spatial features from videos. Experimental validation is performed on a publicly accessible campus dataset, illustrating the model's superior accuracy in anomaly detection.

Annotation-free prediction of treatment-specific tissue outcome from 4D CT perfusion imaging in acute ischemic stroke

Acute ischemic stroke is a critical health condition that requires timely intervention. Following admission, clinicians typically use perfusion imaging to facilitate treatment decision-making. While deep learning models leveraging perfusion data have demonstrated the ability to predict post-treatment tissue infarction for individual patients, predictions are often represented as binary or probabilistic masks that are not straightforward to interpret or easy to obtain. Moreover, these models typically rely on large amounts of subjectively segmented data and non-standard perfusion analysis techniques. To address these challenges, we propose a novel deep learning approach that directly predicts follow-up computed tomography images from full spatio-temporal 4D perfusion scans through a temporal compression. The results show that this method leads to realistic follow-up image predictions containing the infarcted tissue outcomes. The proposed compression method achieves comparable prediction results to using perfusion maps as inputs but without the need for perfusion analysis or arterial input function selection. Additionally, separate models trained on 45 patients treated with thrombolysis and 102 treated with thrombectomy showed that each model correctly captured the different patient-specific treatment effects as shown by image difference maps. The findings of this work clearly highlight the potential of our method to provide interpretable stroke treatment decision support without requiring manual annotations.

Projections of future spatiotemporal urban 3D expansion in China under shared socioeconomic pathways

The 21st century is marked by urbanization, with global focus on horizontal city expansion. Yet, vertical growth lacks research due to data scarcity. This study investigates the fusion of urban building height predictions with conventional sprawl projections to offer a holistic 3D urban expansion foresight. We constructed a model for projecting urban building volume using socio-economic factors includes GDP, population, and urbanization indicators, enabling the anticipation of China's 2010–2100 building volume demand across five Shared Socioeconomic Pathways (SSPs). Furthermore, deep learning-based neural networks were harnessed for estimating land conversion probabilities and building heights. By aggregating the projected demand, we estimated spatial expansion and building heights for each scenario at ten-year intervals from 2010 to 2100. Compared to other products, our proposed method excels in spatial granularity and temporal continuity, which includes height properties in the 3D structure. According to the projections of this study, considering various SSPs, China's building volume is expected to reach approximately 4500 km3 to 6500 km3 by 2050, representing 1.3–1.9 times increase compared to 2010. Furthermore, forecasts for 2050–2100 anticipate a notable decline to 60 % of the 2050 demand. At a more local level, potential spatial imbalances may arise, as the projections indicate significant urban expansion in the eastern areas and a heightened density of urban land within existing city clusters. This approach surpasses the constraints of 2D analysis, pioneering the prediction of fine-resolution, long-term spatiotemporal urban expansion in China (2010–2100), contributing a model and data support for future sustainable urban development pursuits.

AST3DRNet: Attention-Based Spatio-Temporal 3D Residual Neural Networks for Traffic Congestion Prediction.

Traffic congestion prediction has become an indispensable component of an intelligent transport system. However, one limitation of the existing methods is that they treat the effects of spatio-temporal correlations on traffic prediction as invariable during modeling spatio-temporal features, which results in inadequate modeling. In this paper, we propose an attention-based spatio-temporal 3D residual neural network, named AST3DRNet, to directly forecast the congestion levels of road networks in a city. AST3DRNet combines a 3D residual network and a self-attention mechanism together to efficiently model the spatial and temporal information of traffic congestion data. Specifically, by stacking 3D residual units and 3D convolution, we proposed a 3D convolution module that can simultaneously capture various spatio-temporal correlations. Furthermore, a novel spatio-temporal attention module is proposed to explicitly model the different contributions of spatio-temporal correlations in both spatial and temporal dimensions through the self-attention mechanism. Extensive experiments are conducted on a real-world traffic congestion dataset in Kunming, and the results demonstrate that AST3DRNet outperforms the baselines in short-term (5/10/15 min) traffic congestion predictions with an average accuracy improvement of 59.05%, 64.69%, and 48.22%, respectively.

Self-encapsulated ionic fibers based on stress-induced adaptive phase transition for non-contact depth-of-field camouflage sensing

Ionically conductive fibers have promising applications; however, complex processing techniques and poor stability limit their practicality. To overcome these challenges, we proposed a stress-induced adaptive phase transition strategy to conveniently fabricate self-encapsulated hydrogel-based ionically conductive fibers (se-HICFs). se-HICFs can be produced simply by directly stretching ionic hydrogels with ultra-stretchable networks (us-IHs) or by dip-drawing from molten us-IHs. During this process, stress facilitated the directional migration and evaporation of water molecules in us-IHs, causing a phase transition in the surface layer of ionic fibers to achieve self-encapsulation. The resulting sheath-core structure of se-HICFs enhanced mechanical strength and stability while endowing se-HICFs with powerful non-contact electrostatic induction capabilities. Mimicking nature, se-HICFs were woven into spider web structures and camouflaged in wild environments to achieve high spatiotemporal resolution 3D depth-of-field sensing for different moving media. This work opens up a convenient route to fabricate stable functionalized ionic fibers.

Real-Time Intelligent Video Surveillance System using Recurrent Neural Network

Security is a significant concern at all locations where CCTV cameras are installed. Security is a top priority; you must invest considerable time and effort to keep track of everything. Shortly, developments in computer vision may substantially impact video surveillance systems. To measure the video feed in real-time and detect any abnormal behavior without human intervention., such as violence or theft. The real-time video was captured as data with regular and strange events, and it has been trained (21 videos) and tested (16 videos) using a deep learning neural network. The captured video has been converted as frames using a Spatio-temporal autoencoder and 3D convolution network with the help of the RNN algorithm with a threshold value of 0.006 to detect the events. The RNN algorithm analyzed the captured video to see the possibilities with higher accuracy, 96%, than others. A data processing model for event detection will be constructed using deep-learning neural network technologies. A sophisticated monitoring system also transmits real-time video and voice communications to the web.

A novel spatiotemporal 3D CNN framework with multi-task learning for efficient structural damage detection

In recent years, convolutional neural networks (CNNs) have demonstrated promising results in detecting structural damage. However, their architectures often overlook spatial and temporal effects simultaneously. This limitation can result in the loss of valuable information and an incapability to fully capture the complexity of the data, ultimately leading to reduced accuracy and suboptimal performance. This study proposes an intuitive three-dimensional CNN architecture that takes into account vibration history along with sensor spatial relations based on their relative positions. Furthermore, a multi-task learning (MTL) approach is suggested, which is a powerful approach for performing multiple tasks with a single network. The proposed 3D CNN method has been employed to detect single and double damage cases in an experimental steel frame through conventional classification alongside the transfer learning (TL). Moreover, MTL is used to detect single and double damage scenarios with a single unified network, which evaluates damage presence in separate tasks. The 3D CNN fulfilled state-of-the-art performance and 100% accuracy in detecting structural damage in almost all experiments. Additionally, the MTL model achieved promising results even in the presence of severe imbalanced classes of data. Furthermore, it was observed that the utilization of TL resulted in a notable reduction of computation time by 68% and the number of trainable parameters by 90% with the same level of accuracy in double-damage cases.

4DHumanOutfit: A multi-subject 4D dataset of human motion sequences in varying outfits exhibiting large displacements

We present a new dataset of densely sampled spatio-temporal 4D human motion data of different actors, outfits and motions. The dataset contains different actors wearing different outfits while performing different motions in each outfit. It samples the space of 4D human motion along 3 axes with identity, outfit and motion, therefore providing a cube of data where each identity is represented by a planar slice of the cube with all outfits and for all motions. The dataset has numerous potential applications for the processing and creation of digital humans including augmented reality, avatar creation and virtual try on. 4DHumanOutfit is released for research purposes at https://kinovis.inria.fr/4dhumanoutfit/. In addition to image data and 4D reconstructions, the dataset includes reference solutions for each axis. We present independent baselines along each axis that demonstrate the value of these reference solutions for evaluation tasks.

Deep spatio-temporal 3D dilated dense neural network for traffic flow prediction

Traffic flow prediction is increasingly vital for the administration of metropolitan areas. Many research on spatio-temporal networks have been explored but the impacts of both spatial and temporal flexibility, complex spatial correlation has not been considered simultaneously. We present the Spatio-Temporal 3D Multiscale Dilated Dense Network (ST-3DMDDN), a novel 3D Convolutional Neural Network (3DCNN) deep learning neural network for the road level and region level traffic flow prediction. It uses autocorrelation analysis’ early fusion method for importance sampling, a 3D multiscale dilated convolutional network to capture nearby and remote correlations simultaneously, and a densely connected network for deeper feature extraction. Considering traffic flow’s heterogeneity, a new block called the “Spatial and Channel Recalibrate” (SCR) is designed to accurately analyze the correlation contributions. The ST-3DMDDN model is evaluated using three real traffic flows, and the findings indicate that our approach surpasses the performance of the baseline approaches.

Spectrally selective bSSFP using off-resonant RF excitations permits high spatiotemporal resolution 3D metabolic imaging of hyperpolarized [1-13 C]Pyruvate-to-[1-13 C]lactate conversion.

To develop a high spatiotemporal resolution 3D dynamic pulse sequence for preclinical imaging of hyperpolarized [1-13 C]pyruvate-to-[1-13 C]lactate metabolism at 7T. A standard 3D balanced SSFP (bSSFP) sequence was modified to enable alternating-frequency excitations. RF pulses with 2.33 ms duration and 900 Hz FWHM were placed off-resonance of the target metabolites, [1-13 C]pyruvate (by approximately -245 Hz) and [1-13 C]lactate (by approximately 735 Hz), to selectively excite those resonances. Relatively broad bandwidth (compared to those metabolites' chemical shift offset) permits a short TR of 6.29 ms, enabling higher spatiotemporal resolution. Bloch equation simulations of the bSSFP response profile guided the sequence parameter selection to minimize spectral contamination between metabolites and preserve magnetization over time. Bloch equation simulations, phantom studies, and in vivo studies demonstrated that the two target resonances could be cleanly imaged without substantial bSSFP banding artifacts and with little spectral contamination between lactate and pyruvate and from pyruvate hydrate. High spatiotemporal resolution 3D images were acquired of in vivo pyruvate-lactate metabolism in healthy wild-type and endogenous pancreatic tumor-bearing mice, with 1.212 s acquisition time per single-metabolite image and (1.75 mm)3 isotropic voxels with full mouse abdomen 56 × 28 × 21 mm3 FOV and fully-sampled k-space. Kidney and tumor lactate/pyruvate ratios of two consecutive measurements in one animal, 1 h apart, were consistent. Spectrally selective bSSFP using off-resonant RF excitations can provide high spatio-temporal resolution 3D dynamic images of pyruvate-lactate metabolic conversion.

Workflow Comparison for Combined 4D MRI/CFD Patient-Specific Cardiovascular Flow Simulations of the Thoracic Aorta

AbstractComputational fluid dynamics (CFD) has been widely used to predict and understand cardiovascular flows. However, the accuracy of CFD predictions depends on faithful reconstruction of patient vascular anatomy and accurate patient-specific inlet and outlet boundary conditions. 4-Dimensional magnetic resonance imaging (4D MRI) can provide patient-specific data to obtain the required geometry and time-dependent flow boundary conditions for CFD simulations, and can further be used to validate CFD predictions. This work presents a framework to combine both spatiotemporal 4D MRI data and patient monitoring data with CFD simulation workflows. To assist practitioners, all aspects of the modeling workflow, from geometry reconstruction to results postprocessing, are illustrated and compared using three software packages (ansys, comsol, SimVascular) to predict hemodynamics in the thoracic aorta. A sensitivity analysis with respect to inlet boundary condition is presented. Results highlight the importance of 4D MRI data for improving the accuracy of flow predictions on the ascending aorta and the aortic arch. In contrast, simulation results for the descending aorta are less sensitive to the patient-specific inlet boundary conditions.

Mammalian embryo development: Towards a single-molecule perspective.

Revealing how pluripotent cells generate distinct cell lineages in the early mammalian embryo is central to uncovering the mechanisms that drive mammalian development and for unlocking the potential of pluripotent cells for regenerative medicine. The objective of my lab is to provide single-molecule insights into nuclear protein complexes that influence the ability of pluripotent cells to differentiate. Although recent studies have characterised the enzymatic activities of some of these protein complexes, little is known about their intra-nuclear dynamics or how they influence spatiotemporal 3D enhancer-promoter relationships. To investigate these dynamics, we have established tools for live-cell 3D tracking of single HaloTag-tagged nuclear proteins and inactive dCas9-labelled genes. We have also developed machine-learning algorithms to infer properties about the resulting trajectories, allowing us to determine the chromatin binding kinetics of key nuclear protein complexes and also how they control chromatin movement at specific enhancers/promoters. Using these approaches, we reveal the chromatin binding kinetics of two major protein complexes that regulate differentiation - the chromatin remodeller NuRD and the histone methyltransferase KMT2B. Furthermore, we show that both NuRD and KMT2B influence the range genes explore within the nucleus. We then use single-cell transcriptomics (scRNAseq) and chromosome conformation capture (Hi-C) to show that these changes in chromatin movement are linked to the length-scale over which enhancers activate transcription at nearby genes. We propose a model linking the movement of enhancers to gene activation as pluripotent cells differentiate. Our results highlight the importance of making dynamic measurements at single-cell and single-molecule resolution to provide insight into transcriptional control during differentiation. We are currently establishing these technologies within live mouse embryos and we believe this will help the field understand the underlying molecular mechanisms of cell-fate decisions in vivo.

Volumetric imaging of optically cleared and fluorescently labeled animal tissue (VIOLA) for quantifying the 3D biodistribution of nanoparticles at cellular resolution in tumor tissue.

Nanoparticle (NP) technology holds significant promise to mediate targeted drug delivery to specific organs in the body. Understanding the 3D biodistribution of NPs in heterogeneous environments such as the tumor tissue can provide crucial information on efficacy, safety and potential clinical outcomes. Here we present a novel end-to-end workflow, VIOLA, which makes use of tissue clearing methodology in conjunction with high resolution imaging and advanced 3D image processing to quantify the spatiotemporal 3D biodistribution of fluorescently labeled ACCURIN® NPs. Specifically, we investigate the spatiotemporal biodistribution of NPs in three different murine tumor models (CT26, EMT6, and KPC-GEM) of increasing complexity and translational relevance. We have developed new endpoints to characterize NP biodistribution at multiple length scales. Our observations reveal that the macroscale NP biodistribution is spatially heterogeneous and exhibits a gradient with relatively high accumulation at the tumor periphery that progressively decreases towards the tumor core in all the tumor models. Microscale analysis revealed that NP extravasation from blood vessels increases in a time dependent manner and plateaus at 72h post injection. Volumetric analysis and pharmacokinetic modeling of NP biodistribution in the vicinity of the blood vessels revealed that the local NP density exhibits a distance dependent spatiotemporal biodistribution which provide insights into the dynamics of NP extravasation in the tumor tissue. Our data represents a comprehensive analysis of NP biodistribution at multiple length scales in different tumor models providing unique insights into their spatiotemporal dynamics. Specifically, our results show that NPs exhibit a dynamic equilibrium with macroscale heterogeneity combined with microscale homogeneity.

Reconstructing in-depth activity for chaotic 3D spatiotemporal excitable media models based on surface data.

Motivated by potential applications in cardiac research, we consider the task of reconstructing the dynamics within a spatiotemporal chaotic 3D excitable medium from partial observations at the surface. Three artificial neural network methods (a spatiotemporal convolutional long-short-term-memory, an autoencoder, and a diffusion model based on the U-Net architecture) are trained to predict the dynamics in deeper layers of a cube from observational data at the surface using data generated by the Barkley model on a 3D domain. The results show that despite the high-dimensional chaotic dynamics of this system, such cross-prediction is possible, but non-trivial and as expected, its quality decreases with increasing prediction depth.

Spatio-Temporal Self-Attention Network for Video Saliency Prediction

3D convolutional neural networks have achieved promising results for video tasks in computer vision, including video saliency prediction that is explored in this paper. However, 3D convolution encodes visual representation merely on fixed local spacetime according to its kernel size, while human attention is always attracted by relational visual features at different time. To overcome this limitation, we propose a novel Spatio-Temporal Self-Attention 3D Network (STSANet) for video saliency prediction, in which multiple Spatio-Temporal Self-Attention (STSA) modules are employed at different levels of 3D convolutional backbone to directly capture long-range relations between spatio-temporal features of different time steps. Besides, we propose an Attentional Multi-Scale Fusion (AMSF) module to integrate multi-level features with the perception of context in semantic and spatio-temporal subspaces. Extensive experiments demonstrate the contributions of key components of our method, and the results on DHF1K, Hollywood-2, UCF, and DIEM benchmark datasets clearly prove the superiority of the proposed model compared with all state-of-the-art models.

Digital twin demonstrates significance of biomechanical growth control in liver regeneration after partial hepatectomy

Partial liver removal is an important therapy option for liver cancer. In most patients within a few weeks, the liver is able to fully regenerate. In some patients, however, regeneration fails with often severe consequences. To better understand the control mechanisms of liver regeneration, experiments in mice were performed, guiding the creation of a spatiotemporal 3D model of the regenerating liver. The model represents cells and blood vessels within an entire liver lobe, a macroscopic liver subunit. The model could reproduce the experimental data only if a biomechanical growth control (BGC)-mechanism, inhibiting cell cycle entrance at high compression, was taken into account and predicted that BGC may act as a short-range growth inhibitor minimizing the number of proliferating neighbor cells of a proliferating cell, generating a checkerboard-like proliferation pattern. Model-predicted cell proliferation patterns in pigs and mice were found experimentally. The results underpin the importance of biomechanical aspects in liver growth control.

ST3DNetCrime: Improved ST-3DNet Model for Crime Prediction at Fine Spatial Temporal Scales

Crime prediction is crucial for sustainable urban development and protecting citizens’ quality of life. However, there exist some challenges in this regard. First, the spatio-temporal correlations in crime data are relatively complex and are heterogenous in time and space, hence it is difficult to model the spatio-temporal correlation in crime data adequately. Second, crime prediction at fine spatial temporal scales can be applied to micro patrol command; however, crime data are sparse in both time and space, making crime prediction very challenging. To overcome these challenges, based on the deep spatio-temporal 3D convolutional neural networks (ST-3DNet), we devise an improved ST-3DNet framework for crime prediction at fine spatial temporal scales (ST3DNetCrime). The framework utilizes diurnal periodic integral mapping to solve the problem of sparse and irregular crime data at fine spatial temporal scales. ST3DNetCrime can, respectively, capture the spatio-temporal correlations of recent crime data, near historical crime data and distant historical crime data as well as describe the difference in the correlations’ contributions in space. Extensive experiments on real-world datasets from Los Angeles demonstrated that the proposed ST3DNetCrime framework has better prediction performance and enhanced robustness compared with baseline methods. In additon, we verify that each component of ST3DNetCrime is helpful in improving prediction performance.

SASTA-Net: self-attention spatiotemporal adversarial network for typhoon prediction

To solve the problems of poor authenticity and lack of clarity for short-time typhoon prediction, we propose a self-attentional spatiotemporal adversarial network (SASTA-Net). First, we introduce a multispatiotemporal feature fusion method to fully extract and fuse the multichannel spatiotemporal feature information to effectively enhance feature expression. Second, we propose an SATA-LSTM prediction model that incorporates spatial memory cell and attention mechanisms in order to capture spatial features and important details in sequences. Finally, a spatiotemporal 3D discriminator is designed to correctly distinguish the generated predicted cloud image from the real cloud image and generate a more accurate and real typhoon cloud image by adversarial training. The evaluation results on the typhoon cloud image data set show that the proposed SASTA-Net achieves 67.3, 0.878, 31.27, and 56.48 in mean square error, structural similarity, peak signal to noise ratio, and sharpness, respectively, which is superior to the most advanced prediction algorithm.

Predicting Appearance of Vehicles From Blind Spots Based on Pedestrian Behaviors at Crossroads

Conventional prediction approaches for traffic scenes primarily predict the future states of visible objects ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e</i> ., not in blind spots) based on their current observations. This study focused on the prediction of future states of objects in blind spots ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e.g</i> ., those outside the filed-of-view or occluded regions) based on the current observations of other visible objects. We proposed a method that predicts the appearance of vehicles from a blind spot based on the behaviors of visible pedestrians who observe vehicles in the blind spot. Our proposed method utilizes a spatiotemporal 3D convolutional neural network and learns pedestrian behaviors for predictions. The method explicitly represents subtle motions and the surrounding environments of pedestrians using pose estimation and semantic segmentation. To conduct evaluation experiments, we built two datasets of videos capturing real traffic scenes. The datasets are collected by cameras with and without ego-motions. Using the datasets, we conducted experiments not only on simpler configurations but also on realistic traffic environments. Based on the experimental results, the following conclusions could be obtained: (i) our proposed method achieved a high performance at a level similar to that of humans in our prediction task, and predicted the appearance of vehicles from blind spots more than 1.5 s before they actually appeared. (ii) Explicit representations of pose and semantic masks captured information complementary to RGB videos, and ensembling the representations improved the prediction performance. (iii) Fine-tuning the models using videos with ego-motions is important to achieve good prediction in the videos captured by driving cars.