3D Convolution Operations Research Articles

Two-stage multi-dimensional convolutional stacked autoencoder network model for hyperspectral images classification

Deep learning models have been widely used in hyperspectral images classification. However, the classification results are not satisfactory when the number of training samples is small. Focused on above-mentioned problem, a novel Two-stage Multi-dimensional Convolutional Stacked Autoencoder (TMC-SAE) model is proposed for hyperspectral images classification. The proposed model is composed of two sub-models SAE-1 and SAE-2. The SAE-1 is a 1D autoencoder with asymmetric structre based on full connection layers and 1D convolution layers to reduce spectral dimensionality. The SAE-2 is a hybrid autoencoder composed of 2D and 3D convolution operations to extract spectral-spatial features from the reduced dimensionality data by SAE-1. The SAE-1 is trained with raw data by unsupervised learning and the encoder of SAE-1 is employed to reduce spectral dimensionality of raw data. The data after dimension reduction is used to train the SAE-2 by unsupervised learning. The fine-tuning of SAE-2 encoder and the training of classifier are implemented simultaneously with small number of samples by supervised learning. Comparative experiments are performed on three widely used hyperspectral remote sensing data. The extensive comparative experiments demonstrate that the proposed architecture can effectively extract deep features and maintain high classification accuracy with small number of training samples.

OctFormer: Efficient Octree-Based Transformer for Point Cloud Compression with Local Enhancement

Point cloud compression with a higher compression ratio and tiny loss is essential for efficient data transportation. However, previous methods that depend on 3D convolution or frequent multi-head self-attention operations bring huge computations. To address this problem, we propose an octree-based Transformer compression method called OctFormer, which does not rely on the occupancy information of sibling nodes. Our method uses non-overlapped context windows to construct octree node sequences and share the result of a multi-head self-attention operation among a sequence of nodes. Besides, we introduce a locally-enhance module for exploiting the sibling features and a positional encoding generator for enhancing the translation invariance of the octree node sequence. Compared to the previous state-of-the-art works, our method obtains up to 17% Bpp savings compared to the voxel-context-based baseline and saves an overall 99% coding time compared to the attention-based baseline.

Spatiotemporal Feature Hierarchy-Based Blind Prediction of Natural Video Quality via Transfer Learning

In this paper, we propose a pyramidal spatiotemporal feature hierarchy (PSFH)-based no-reference (NR) video quality assessment (VQA) method using transfer learning. First, we generate simulated videos by a generative adversarial network (GAN)-based image restoration model. The residual maps between the distorted frames and simulated frames, which can capture rich information, are utilized as one input of the quality regression network. Second, we use 3D convolution operations to construct a PSFH network with five stages. The spatiotemporal features incorporating the shared features transferred from the pretrained image restoration model are fused stage by stage. Third, with the guidance of the transferred knowledge, each stage generates multiple feature mapping layers that encode different semantics and degradation information using 3D convolution layers and gated recurrent units (GRUs). Finally, five approximate perceptual quality scores and a precise prediction score are obtained by fully connected (FC) networks. The whole model is trained under a finely designed loss function that combines pseudo-Huber loss and Pearson linear correlation coefficient (PLCC) loss to improve the robustness and prediction accuracy. According to the extensive experiments, outstanding results can be obtained compared with other state-of-the-art methods. Both the source code and models are available online. <xref ref-type="fn" rid="fn1" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</xref>

Scene point cloud understanding and reconstruction technologies in 3D space

Scene point cloud understanding and reconstruction technologies in 3D space

FPGA-Based Design for Accelerating 3D Convolutional Neural Networks

Three-dimensional convolutional neural networks (3D CNNs) have been shown to be excellent for classification in human action recognition, hyperspectral image classification and many other aspects. However, 3D CNNs often contain convolution kernels of different sizes and a large number of operations, and the numerous operations make 3D CNNs consume a lot of time when executing inference on resource-constrained terminals. Aiming at these two points, this paper proposes a 3D CNN accelerator design based on loop optimization and weight reuse. The design adopts loop tiling, loop unrolling and loop fusion to optimize the direct convolution form of the 3D convolution operation; and guided by the optimized form of the 3D convolution operation, a module that can accelerate different 3D convolution operations is designed; and a memory access method based on weight reuse is also designed to reduce the memory access time of features and weights. Experimental results show that the proposed accelerator not only supports convolution kernels of different sizes, but also has a performance density of 1.59, which outperforms most existing accelerators.

A 3D Cross-Modality Feature Interaction Network with Volumetric Feature Alignment for Brain Tumor and Tissue Segmentation.

Accurate volumetric segmentation of brain tumors and tissues is beneficial for quantitative brain analysis and brain disease identification in multi-modal Magnetic Resonance (MR) images. Nevertheless, due to the complex relationship between modalities, 3D Fully Convolutional Networks (3D FCNs) using simple multi-modal fusion strategies hardly learn the complex and nonlinear complementary information between modalities. Meanwhile, the indiscriminative feature aggregation between low-level and high-level features easily causes volumetric feature misalignment in 3D FCNs. On the other hand, the 3D convolution operations of 3D FCNs are excellent at modeling local relations but typically inefficient at capturing global relations between distant regions in volumetric images. To tackle these issues, we propose an Aligned Cross-Modality Interaction Network (ACMINet) for segmenting the regions of brain tumors and tissues from MR images. In this network, the cross-modality feature interaction module is first designed to adaptively and efficiently fuse and refine multi-modal features. Secondly, the volumetric feature alignment module is developed for dynamically aligning low-level and high-level features by the learnable volumetric feature deformation field. Thirdly, we propose the volumetric dual interaction graph reasoning module for graph-based global context modeling in spatial and channel dimensions. Our proposed method is applied to brain glioma, vestibular schwannoma, and brain tissue segmentation tasks, and we performed extensive experiments on BraTS2018, BraTS2020, Vestibular Schwannoma, and iSeg-2017 datasets. Experimental results show that ACMINet achieves state-of-the-art segmentation performance on all four benchmark datasets and obtains the highest DSC score of hard-segmented enhanced tumor region on the validation leaderboard of the BraTS2020 challenge.

Spectral–Spatial Depth-Based Framework for Hyperspectral Underwater Target Detection

Ocean-related research is of critical significance to the national marine military force. Hyperspectral underwater target detection (HUTD) has attracted widespread attention in recent years. However, most of the previous methods only relied on the spectral features of underwater targets and did not fully exploit their spatial characteristics. To address the issue, a spectral-spatial depth-based framework is proposed, which utilizes 3D convolution operation to capture spectral-spatial features and gains finer detection based on predicted depth. Especially, the proposed framework adopts the data transferring network to remove the noise interference by transferring the real hyperspectral data into corresponding synthetic data, which are exploited to train models. Then, considering that underwater target spectra highly depend on its depth in water, the depth estimation network is utilized to predict an accurate depth of a target, which can contribute to selecting a suitable detection network and gaining a general contour of the target. Since the underwater target spectrum is jointly determined by the target and the surrounding water column, the spectral-spatial detection network extracts the spectral-spatial features for underwater target detection. Using pool dataset, sea dataset and a synthetic HSI, we evaluate the performance of the proposed framework in terms of ROC curve and AUC value, both qualitatively and quantitatively. Meanwhile, extensive detection experiments demonstrate the robustness and effectiveness of the TDSS-UTD over several state-of-the-art methods.

Abnormal behavior recognition based on feature fusion C3D network

The real-time detection and recognition ability of human action recognition in a video surveillance system is a key problem in an intelligent surveillance system. Because the behavior recognition for video surveillance systems is affected by the complexity of the scene, the classification performance of the behavior recognition models is not satisfactory. To increase the processing efficiency of the network and solve the problem of low classification accuracy of human action recognition, we designed a deep learning model based on three-dimensional (3D) convolutional network multiscale feature fusion to reduce the impact of constant appearance changes, background clutter, and pedestrian occlusion. The model alternately uses 3D convolution and 3D pooling operations to extract temporal and spatial features between consecutive frames after data preprocessing, and then uses a feature pyramid structure to select three sets of feature layers with different scales. The model performs deconvolution operations in a bottom to up order and fuses with the features of the previous layer, then downsampling and high-level feature layer fusion are performed sequentially from top to bottom. Using the newly generated the highest-level feature layer to realize abnormal behavior recognition. The C3D network algorithm based on feature fusion proposed in this paper is compared with the three most advanced methods of C3D, R3D, and R ( 2 + 1 ) D on the pedestrian abnormal action recognition (PAAR) dataset and the same parameters, and the accuracy is significantly improved.

A lightweight 3D-2D convolutional neural network for spectral-spatial classification of hyperspectral images

Hyperspectral Image (HSI) is usually composed of hundreds of capturing wavelength bands, which not only increase the size of the HSI rapidly but also impose various obstacles in classifying the objects accurately. Moreover, the traditional machine learning schemes utilize only the spectral features for HSI classification, which, therefore, neglect the spatial features that have a significant impact on the classification improvement. To address the aforementioned issues, in this paper, we propose to employ the principal component analysis (PCA), the baseline feature extraction method, and a thoughtfully designed stacked autoencoder, a deep learning-based feature extraction approach, for reducing the high dimensionality of the HSI and then propose a novel lightweight 3D-2D convolutional neural network (CNN) framework to concurrently exploit both spatial and spectral features from the dimensionality-reduced HSI for classification. In particular, PCA and stacked autoencoder are applied to reduce the high dimensionality of the original HSI and then the proposed 3D-2D CNN provides a combination of 3D and 2D convolution operations to extract the subtle spatial and spectral features for efficient classification. We well-adjust the proposed 3D-2D CNN architecture, and perform extensive experiments on three benchmark HSI datasets and compare our approach with the state-of-the-art classical and deep learning methods. Experimental results illustrate that we have achieved an overall accuracy of 99.73%, 99.90%, and 99.32% on Indian Pines, Pavia University, and Kennedy Space Center datasets, respectively, which outperform the classical machine learning and independent 2D and 3D CNN-based state-of-the-art methods.

Mixed Structure with 3D Multi-Shortcut-Link Networks for Hyperspectral Image Classification

A hyperspectral image classification method based on a mixed structure with a 3D multi-shortcut-link network (MSLN) was proposed for the features of few labeled samples, excess noise, and heterogeneous homogeneity of features in hyperspectral images. First, the spatial–spectral joint features of hyperspectral cube data were extracted through 3D convolution operation; then, the deep network was constructed and the 3D MSLN mixed structure was used to fuse shallow representational features and deep abstract features, while the hybrid activation function was utilized to ensure the integrity of nonlinear data. Finally, the global self-adaptive average pooling and L-softmax classifier were introduced to implement the terrain classification of hyperspectral images. The mixed structure proposed in this study could extract multi-channel features with a vast receptive field and reduce the continuous decay of shallow features while improving the utilization of representational features and enhancing the expressiveness of the deep network. The use of the dropout mechanism and L-softmax classifier endowed the learned features with a better generalization property and intraclass cohesion and interclass separation properties. Through experimental comparative analysis of six groups of datasets, the results showed that this method, compared with the existing deep-learning-based hyperspectral image classification methods, could satisfactorily address the issues of degeneration of the deep network and “the same object with distinct spectra, and distinct objects with the same spectrum.” It could also effectively improve the terrain classification accuracy of hyperspectral images, as evinced by the overall classification accuracies of all classes of terrain objects in the six groups of datasets: 97.698%, 98.851%, 99.54%, 97.961%, 97.698%, and 99.138%.

Hyperspectral Image Segmentation using End-to-End CNN Architecture with built-in Feature Compressor for UAV Systems

Hyperspectral image segmentation is an important task for geographical surveying. Real-time processing of this operation is especially important for sensors mounted on-board Unmanned Aerial Vehicles in the context of visual servoing, landmarks recognition and data compression for efficient storage and transmission. To this end, this paper proposes a machine learning approach for segmentation using an efficient Convolutional Neural Network (CNN) which incorporates a feature compressor and a subsequent segmentation module based on 3D convolution operations. The experimental results demonstrate that the proposed approach gives segmentation accuracy at par with conventional approaches based on Principal Component Analysis (PCA) to reduce the feature dimensionality. Moreover, the proposed network is at least 35% faster than the conventional CNN-based approaches using 3D convolutions.

Revisiting Deep Hyperspectral Feature Extraction Networks via Gradient Centralized Convolution

The hyperspectral images are composed of a variety of textures across the different bands which increase the spectral similarity and make it difficult to predict the pixel-wise labels without inducing additional complexity at the feature level. To extract robust and discriminative features from the different regions of land cover, the hyperspectral research community is still seeking such type of convolutions which can efficiently deal with fine-grained texture information during the feature extraction phase, which often overlook this aspect by vanilla convolution. To overcome the above shortcoming, this article proposes a generalized gradient centralized 3D convolution ( <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">G2C-Conv3D</monospace> ) operation, which is a weighted combination between the vanilla and gradient centralized 3D convolutions ( <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">GC-Conv3D</monospace> ) to extract both the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">intensity-level</i> semantic information and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">gradient-level</i> information. This can be easily plugged into the existing HSI feature extraction networks to boost the performance of accurate prediction for land-cover types. To validate the feasibility of the proposed <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">G2C-Conv3D</monospace> , we have considered the existing CNN3D, MS3DNet, ContextNet, and SSRN feature extraction models and as well as CAE3D, VAE3D, and SAE3D autoencoder (AE) networks, respectively. All these networks are embedded with <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">G2C-Conv3D</monospace> convolution to implement both generalized gradient centralized feature extraction networks (G2C-FE) and generalized gradient centralized AE networks (G2C-AE) for fine-grained spectral–spatial feature learning. In addition, <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">G2C-Conv2D</monospace> is also considered with few networks. The extensive experiments are conducted on four most widely used hyperspectral datasets i.e., IP, KSC, UH, and UP, respectively, and compared with the nine methods. The results demonstrate that the proposed <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">G2C-Conv3D</monospace> can effectively enhance the feature learning ability of the existing networks and both the qualitative and quantitative results show the superiority and effectiveness of the proposed <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">G2C-Conv3D</monospace> . The source codes will be publicly available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/danfenghong/G2C-Conv3D-HSI</uri> .

A Novel Method for Precipitation Nowcasting Based on ST-LSTM

Precipitation nowcasting is of great significance for severe convective weather warnings. Radar echo extrapolation is a commonly used precipitation nowcasting method. However, the traditional radar echo extrapolation methods are encountered with the dilemma of low prediction accuracy and extrapolation ambiguity. The reason is that those methods cannot retain important long-term information and fail to capture short-term motion information from the long-range data stream. In order to solve the above problems, we select the spatiotemporal long short-term memory (ST-LSTM) as the recurrent unit of the model and integrate the 3D convolution operation in it to strengthen the model's ability to capture short-term motion information which plays a vital role in the prediction of radar echo motion trends. For the purpose of enhancing the model's ability to retain long-term important information, we also introduce the channel attention mechanism to achieve this goal. In the experiment, the training and testing datasets are constructed using radar data of Shanghai, we compare our model with three benchmark models under the reflectance thresholds of 15 and 25. Experimental results demonstrate that the proposed model outperforms the three benchmark models in radar echo extrapolation task, which obtains a higher accuracy rate and improves the clarity of the extrapolated image.

Multiple CNN Variants and Ensemble Learning for Sunspot Group Classification by Magnetic Type

Abstract A solar active region is a source of disturbance for the Sun–terrestrial space environment and usually causes extreme space weather, such as geomagnetic storms. The main indicator of an active region is sunspots. Certain types of sunspots are related to extreme space weather caused by eruptive events such as coronal mass ejections or solar flares. Thus, the automatic classification of sunspot groups is helpful to predict solar activity quickly and accurately. This paper completed the automatic classification of a sunspot group data set based on the Mount Wilson classification scheme, which contains continuum and magnetogram images provided by the Solar Dynamics Observatory’s Helioseismic and Magnetic Imager SHARP data from 2010 May 1 to 2017 December 12. After applying some data preprocessing steps such as image cropping and data standardization, the features of magnetic type in the data are more obvious, and the amount of data is increased. The processed data are spliced into two frames of single-channel data for the neural network to perform 3D convolution operations. This paper constructs a variety of convolutional neural networks with different structures and numbers of layers, selects 10 models as representatives, and chooses XGBoost, which is commonly used in ensemble-learning algorithms, to fuse the results of independent classification models. We found that XGBoost is an effective way to fuse models, which is proved by the relatively balanced high scores in the three magnetic types. The accuracy of the ensemble model is above 92%. The F1 scores of the magnetic types of Alpha, Beta, and Beta-x reached 0.95, 0.91, and 0.82 respectively.

The Use of 3D Convolutional Autoencoder in Fault and Fracture Network Characterization

Conventional pattern recognition methods directly use 1D poststack data or 2D prestack data for the statistical pattern recognition of fault and fracture network, thereby ignoring the spatial structure information in 3D seismic data. As a result, the generated fault and fracture network is not distinguishable and has poor continuity. In this paper, a fault and fracture network characterization method based on 3D convolutional autoencoder is proposed. First, in the autoencoder training frame, 3D prestack data are used as input, and the 3D convolution operation is used to mine the spatial structure information to the maximum and gradually reduce the spatial dimension of the input. Then, the residual network is used to recover the input’s details and the corresponding spatial dimension. Lastly, the hidden features extracted by the encoders are recognized via k -means, SOM, and two-step clustering analysis. The validity of the method is verified by testing the seismic simulation data and applying real seismic data. The 3D convolution can directly process the seismic data and maximize the prestack texture attributes and spatial structure information provided by 3D seismic data without dimensionality reduction and other preprocessing operations. The interleaving convolution layer and residual block overcome low learning and accuracy rates due to the deepening of networks.

Anchor-Based Spatio-Temporal Attention 3-D Convolutional Networks for Dynamic 3-D Point Cloud Sequences

With the rapid development of measurement technology, LiDAR and depth cameras are widely used in the perception of the 3D environment. Recent learning based methods for robot perception most focus on the image or video, but deep learning methods for dynamic 3D point cloud sequences are underexplored. Therefore, developing efficient and accurate perception method compatible with these advanced instruments is pivotal to autonomous driving and service robots. An Anchor-based Spatio-Temporal Attention 3D Convolution operation (ASTA3DConv) is proposed in this paper to process dynamic 3D point cloud sequences. The proposed convolution operation builds a regular receptive field around each point by setting several virtual anchors around each point. The features of neighborhood points are firstly aggregated to each anchor based on the spatio-temporal attention mechanism. Then, anchor-based 3D convolution is adopted to aggregate these anchors' features to the core points. The proposed method makes better use of the structured information within the local region and learns spatio-temporal embedding features from dynamic 3D point cloud sequences. Anchor-based Spatio-Temporal Attention 3D Convolutional Neural Networks (ASTA3DCNNs) are built for classification and segmentation tasks based on the proposed ASTA3DConv and evaluated on action recognition and semantic segmentation tasks. The experiments and ablation studies on MSRAction3D and Synthia datasets demonstrate the superior performance and effectiveness of our method for dynamic 3D point cloud sequences. Our method achieves the state-of-the-art performance among the methods with dynamic 3D point cloud sequences as input on MSRAction3D and Synthia datasets.

Multi-View Separable Pyramid Network for AD Prediction at MCI Stage by 18F-FDG Brain PET Imaging.

Alzheimer's Disease (AD), one of the main causes of death in elderly people, is characterized by Mild Cognitive Impairment (MCI) at prodromal stage. Nevertheless, only part of MCI subjects could progress to AD. The main objective of this paper is thus to identify those who will develop a dementia of AD type among MCI patients. 18F-FluoroDeoxyGlucose Positron Emission Tomography (18F-FDG PET) serves as a neuroimaging modality for early diagnosis as it can reflect neural activity via measuring glucose uptake at resting-state. In this paper, we design a deep network on 18F-FDG PET modality to address the problem of AD identification at early MCI stage. To this end, a Multi-view Separable Pyramid Network (MiSePyNet) is proposed, in which representations are learned from axial, coronal and sagittal views of PET scans so as to offer complementary information and then combined to make a decision jointly. Different from the widely and naturally used 3D convolution operations for 3D images, the proposed architecture is deployed with separable convolution from slice-wise to spatial-wise successively, which can retain the spatial information and reduce training parameters compared to 2D and 3D networks, respectively. Experiments on ADNI dataset show that the proposed method can yield better performance than both traditional and deep learning-based algorithms for predicting the progression of Mild Cognitive Impairment, with a classification accuracy of 83.05%.

A Spectral-Aware Convolutional Neural Network for Pansharpening

Pansharpening aims at fusing a low-resolution multiband optical (MBO) image, such as a multispectral or a hyperspectral image, with the associated high-resolution panchromatic (PAN) image to yield a high spatial resolution MBO image. Though having achieved superior performances to traditional methods, existing convolutional neural network (CNN)-based pansharpening approaches are still faced with two challenges: alleviating the phenomenon of spectral distortion and improving the interpretation abilities of pansharpening CNNs. In this work, we develop a novel spectral-aware pansharpening neural network (SA-PNN). On the one hand, SA-PNN employs a network structure composed of a detail branch and an approximation branch, which is consistent with the detail injection framework; on the other hand, SA-PNN strengthens processing along the spectral dimension by using a spectral-aware strategy, which involves spatial feature transforms (SFTs) coupling the approximation branch with the detail branch as well as 3D convolution operations in the approximation branch. Our method is evaluated with experiments on real-world multispectral and hyperspectral datasets, verifying its excellent pansharpening performance.

A Multi-Task Collaborative Network for Light Field Salient Object Detection

Being able to predict the salient object is of fundamental importance in image processing and computer vision. With numerous approaches proposed for automatic image and video salient object detection, much less work has been dedicated to detecting and segmenting salient objects from light fields. In this article, based on the intrinsic characteristics of light fields, we carefully explore the complementary coherence among multiple cues including spatial, edge and depth information, and elaborately design a multi-task collaborative network for light field salient object detection. More specifically, the correlation mechanisms among edge detection, depth inference and salient object detection are carefully investigated to facilitate the representative saliency features. We first model the coherence among low-level features and heuristic semantic priors, as well as the edge information. Subsequently, the depth-oriented saliency features are derived from the geometry of light fields, in which the 3D convolution operation is leveraged with powerful representation capability to model the disparity correlations among multiple viewpoint images. Finally, a feature-enhanced salient object generator is developed to integrate these complementary saliency features, leading to the final salient object predictions for light fields. Quantitative and qualitative experiments demonstrate the superiority of our proposed model against the state-of-the-art methods over the public light field salient object detection datasets.

Multiple local 3D CNNs for region-based prediction in smart cities

In smart cities, region-based prediction (e.g. traffic flow and electricity flow) is of great importance to city management and public safety, and it remains a daunting challenge that involves complicated spatial-temporal-related factors such as weather, holidays, events, etc. Region-based forecasting aims to predict the future situation for regions in a city based on historical data. In the existing literature, the state-of-the-art method solve region-based problems with long short-term memory (LSTM) algorithms that extract the temporal view and local convolutional neural network (CNN) algorithms that extract the spatial view (local spatial correlation via local CNN). In this paper, we propose a deep learning-based method for region-based prediction for smart cities. First, we divide the cities into regions based on the space dimension and model the situation of the cities in 3D volumes. Based on the constructed 3D volumes, we design a model called multiple local 3D CNN spatial-temporal residual networks (LMST3D-ResNet) for region-based prediction in smart cities. LMST3D-ResNet can extract multiple temporal dependencies (including trend, period and closeness) for local regions and then predict the future citywide activities according to the learned multiple spatial-temporal features. LMST3D-ResNet can also combine the spatial-temporal features with external factors. LMST3D-ResNet includes 3D CNNs and ResNet mechanisms for processing spatial-temporal information. In particular, 3D CNNs have the ability to model 3-dimensional information due to 3D convolution and 3D pooling operations, while ResNet enables the connection of the convolutional neural network across layers to obtain a deeper network structure. Specifically, in our proposed model, a novel region-based information extraction mechanism and an end-to-end multiple spatial-temporal dependency learning structure are designed for local regions. Extensive experimental results on two datasets, i.e., MLElectricity and BJTaxi demonstrate the superior performance of our proposed method over the exisiting state-of-the-art methods.