Spatial Encoder Research Articles

Spatial–Temporal-Correlation-Constrained Dynamic Graph Convolutional Network for Traffic Flow Forecasting

Accurate traffic flow prediction in road networks is essential for intelligent transportation systems (ITS). Since traffic data are collected from the road network with spatial topological and time series sequences, the traffic flow prediction is regarded as a spatial–temporal prediction task. With the powerful ability to model the non-Euclidean data, the graph convolutional network (GCN)-based models have become the mainstream framework for traffic forecasting. However, existing GCN-based models either use the manually predefined graph structure to capture the spatial features, ignoring the heterogeneity of road networks, or simply perform 1-D convolution with fixed kernel to capture the temporal dependencies of traffic data, resulting in insufficient long-term temporal feature extraction. To solve those issues, a spatial–temporal correlation constrained dynamic graph convolutional network (STC-DGCN) is proposed for traffic flow forecasting. In STC-DGCN, a spatial–temporal embedding encoder module (STEM) is first constructed to encode the dynamic spatial relationships for road networks at different time steps. Then, a temporal feature encoder module with heterogeneous time series correlation modeling (TFE-HCM) and a spatial feature encoder module with dynamic multi-graph modeling (SFE-DCM) are designed to generate dynamic graph structures for effectively capturing the dynamic spatial and temporal correlations. Finally, a spatial–temporal feature fusion module based on a gating fusion mechanism (STM-GM) is proposed to effectively learn and leverage the inherent spatial–temporal relationships for traffic flow forecasting. Experimental results from three real-world traffic flow datasets demonstrate the superior performance of the proposed STC-DGCN compared with state-of-the-art traffic flow forecasting models.

SCANet: A lightweight deep learning network for massive MIMO CSI feedback based on spatial and channel attention mechanism

Multi-user massive multiple-input multiple-output (MIMO) communication systems consume too much downlink bandwidth due to the huge channel state information (CSI) feedback, deep learning-based CSI feedback approaches fortunately can alleviate the feedback overhead while obtaining an accurate CSI. However, there is a trade-off between the high feedback performance and low computational complexity. In this paper, a low-complexity CSI feedback approach is proposed based on spatial and channel attention mechanism, namely the Spatial and Channel Attention Network (SCANet). Specifically, the spatial and channel attention mechanism makes the network’s attention mainly focus on the specific spatial regions and key feature channels. We devise a serial architecture in the encoder that composes of Spatial and Channel Attention Block (SCAB) and Encoder Transformer Block. Moreover, we design a hybrid architecture in the decoder that composes of the CNNs Block and Decoder Transformer Block. These designs enable the network to effectively extract both global and local CSI features. Computer simulations in both the indoor and outdoor scenarios show that under the same system configurations, the proposed low-complexity SCANet achieves almost the same performance as the state-of-the-art network while reducing the computational complexity by 85.76% fewer floating-point operations per second (FLOPS) on average.

TDT-MIL: a framework with a dual-channel spatial positional encoder for weakly-supervised whole slide image classification.

The classic multiple instance learning (MIL) paradigm is harnessed for weakly-supervised whole slide image (WSI) classification. The spatial position relationship located between positive tissues is crucial for this task due to the small percentage of these tissues in billions of pixels, which has been overlooked by most studies. Therefore, we propose a framework called TDT-MIL. We first serially connect a convolutional neural network and transformer for basic feature extraction. Then, a novel dual-channel spatial positional encoder (DCSPE) module is designed to simultaneously capture the complementary local and global positional information between instances. To further supplement the spatial position relationship, we construct a convolutional triple-attention (CTA) module to attend to the inter-channel information. Thus, the spatial positional and inter-channel information is fully mined by our model to characterize the key pathological semantics in WSI. We evaluated TDT-MIL on two publicly available datasets, including CAMELYON16 and TCGA-NSCLC, with the corresponding classification accuracy and AUC up to 91.54%, 94.96%, and 90.21%, 94.36%, respectively, outperforming state-of-the-art baselines. More importantly, our model possesses a satisfactory capability in solving the imbalanced WSI classification task using an ingenious but interpretable structure.

FINet: Handwriting trajectory reconstruction of Chinese characters based on the font imitate network

Recovering online handwriting trajectory from Chinese handwritten images is still a difficult problem because of abundant strokes, complex structures and different writing styles of Chinese characters. To address this problem, this paper proposes a handwriting trajectory reconstruction method for Chinese characters called the Font Imitate Network. The Font Imitate Network integrates two networks, the spatial encoder uses HRNet to encode the spatial association information of each position on the offline image; the temporal decoder consists of a multilayer perceptron and a recurrent neural network. The stroke feature network is designed to help predict the next handwriting point by focusing on the stroke path in the form of heatmap. Experiments are conducted on the CASIA-OLHWDB1.1 with DTW, LDTW, RMSE and character recognition accuracy as the evaluation metrics. The experimental results show our FINet outperforms the state-of-the-art methods and the recognition accuracy of offline characters is improved by the handwriting reconstruction.

GCN-Enhanced Spatial-Spectral Dual-Encoder Network for Simultaneous Segmentation of Retinal Layers and Fluid in OCT Images

Changes in retinal thickness and the morphology of fluid provide valuable diagnostic information for Macular Edema (ME). Optical Coherence Tomography (OCT) is a pivotal diagnostic tool for ME, and the accurate segmentation of OCT retinal images aids ophthalmologists in monitoring disease progression. However, the unpredictable nature of ME’s occurrence poses significant challenges for existing methods in achieving precise segmentation of both retinal layers and fluids simultaneously. In this study, a novel U-shaped encoder-decoder architecture named GD-Net, is introduced for simultaneously segmenting retinal layers of diverse thicknesses and fluids. Specifically, a novel Fast Fourier Encoder is integrated within the encoder branch to capture spectral domain features overlooked by the spatial encoder. Additionally, Multi-scale Graph Convolution module is inserted between the encoder and decoder branches to leverage retinal layer topology for spatial reasoning. GD-Net is further optimized using a joint loss function, which includes a weighted version of region segmentation loss and contour loss, effectively mitigating the issues of blurred fluid edges and internal holes. The method was validated on three publicly available datasets: DUKE DME, Peripapillary OCT, and RETOUCH, achieving overall mean Dice score of 0.839, 0.826, and 0.872, respectively, outperforming the other state-of-the-art techniques. GD-Net demonstrated consistent high-precision segmentation on B-scans from different devices, proving its robustness and surpassing other methods in terms of visual quality, thus offering substantial support for clinical ophthalmologists in diagnosing ME. The code and weights are publicly available at: https://github.com/DBook111/GD-Net.

Spatio-temporal learning and explaining for dynamic functional connectivity analysis: Application to depression

BackgroundFunctional connectivity has been shown to fluctuate over time. The present study aimed to identifying major depressive disorders (MDD) with dynamic functional connectivity (dFC) from resting-state fMRI data, which would be helpful to produce tools of early depression diagnosis and enhance our understanding of depressive etiology. MethodsThe resting-state fMRI data of 178 subjects were collected, including 89 MDD and 89 healthy controls. We propose a spatio-temporal learning and explaining framework for dFC analysis. A yet effective spatio-temporal model is developed to classifying MDD from healthy controls with dFCs. The model is a stacking neural network model, which learns network structure information by a multi-layer perceptron based spatial encoder, and learns time-varying patterns by a Transformer based temporal encoder. We propose to explain the spatio-temporal model with a two-stage explanation method of importance feature extracting and disorder-relevant pattern exploring. The layer-wise relevance propagation (LRP) method is introduced to extract the most relevant input features in the model, and the attention mechanism with LRP is applied to extract the important time steps of dFCs. The disorder-relevant functional connections, brain regions, and brain states in the model are further explored and identified. ResultsWe achieved the best classification performance in identifying MDD from healthy controls with dFC data. The top important functional connectivity, brain regions, and dynamic states closely related to MDD have been identified. LimitationsThe data preprocessing may affect the classification performance of the model, and this study needs further validation in a larger patient population. ConclusionsThe experimental results demonstrate that the proposed spatio-temporal model could effectively classify MDD, and uncover structural and temporal patterns of dFCs in depression.

An MS-TCN based spatiotemporal model with three-axis tactile for enhancing flexible printed circuit assembly

PurposeCurrent flexible printed circuit (FPC) assembly relies heavily on manual labor, limiting capacity and increasing costs. Small FPC size makes automation challenging as terminals can be visually occluded. The purpose of this study is to use 3D tactile sensing to mimic human manual mating skills for enabling sensing offset between FPC terminals (FPC-t) and FPC mating slots (FPC-s) under visual occlusion.Design/methodology/approachThe proposed model has three stages: spatial encoding, offset estimation and action strategy. The spatial encoder maps sparse 3D tactile data into a compact 1D feature capturing valid spatial assembly information to enable temporal processing. To compensate for low sensor resolution, consecutive spatial features are input to a multistage temporal convolutional network which estimates alignment offsets. The robot then performs alignment or mating actions based on the estimated offsets.FindingsExperiments are conducted on a Redmi Note 4 smartphone assembly platform. Compared to other models, the proposed approach achieves superior offset estimation. Within limited trials, it successfully assembles FPCs under visual occlusion using three-axis tactile sensing.Originality/valueA spatial encoder is designed to encode three-axis tactile data into feature maps, overcoming multistage temporal convolution network’s (MS-TCN) inability to directly process such input. Modifying the output to estimate assembly offsets with related motion semantics overcame MS-TCN’s segmentation points output, unable to meet assembly monitoring needs. Training and testing the improved MS-TCN on an FPC data set demonstrated accurate monitoring of the full process. An assembly platform verified performance on automated FPC assembly.

Multi-Encoder Spatio-Temporal Feature Fusion Network for Electric Vehicle Charging Load Prediction

Electric vehicles (EVs) have been initiated as a preference for decarbonizing road transport. Accurate charging load prediction is essential for the construction of EV charging facilities systematically and for the coordination of EV energy demand with the requisite peak power supply. It is noted that the charging load of EVs exhibits high complexity and randomness due to temporal and spatial uncertainties. Therefore, this paper proposes a SEDformer-based charging road prediction method to capture the spatio-temporal characteristics of charging load data. As a deep learning model, SEDformer comprises multiple encoders and a single decoder. In particular, the proposed model includes a Temporal Encoder Block based on the self-attention mechanism and a Spatial Encoder Block based on the channel attention mechanism with sequence decomposition, followed by an aggregated decoder for information fusion. It is shown that the proposed method outperforms various baseline models on a real-world dataset from Palo Alto, U.S., demonstrating its superiority in addressing spatio-temporal data-driven load forecasting problems.

MDTGAN: Multi domain generative adversarial transfer learning network for traffic data imputation

Accurate, real-time, and efficient traffic data, crucial for intelligent transportation systems, which is often disrupted in the real world due to the influence of weather, interference, and equipment failures. Generative Adversarial Networks (GANs), have achieved significant results in image restoration, providing insights for traffic data imputation. Recent research has focused on developing more effective and reasonable model by learning transferable common knowledge from different cities or different scenarios, which is also an excellent idea for improving the data imputation performance. In light of, we contrive to design a GANs based transferable traffic data imputation model, namely Multi Domain Generative Adversarial Transfer Learning Network (MDTGAN). Firstly, the model consists of two stages. In pre-training stage, the model utilizes the source domain datasets (Similar cities road network datasets) to update parameters and transferred them. Then, the model parameters are optimized using the target domain dataset (Target city road network dataset) in fine-tuning stage to avoid the issue of insufficient samples caused by data missing. Secondly, to adapt the model to different dataset topologies, MDTGAN models the node-level data by taking node time series as input, enabling the model to autonomously learn the prevalent spatiotemporal correlations inherent in nodes. Additionally, we introduce a domain discriminator module that guides the spatial encoder in learning domain-invariant features of network node spatial information, enhancing the model generalization ability. The experiments conducted on three publicly available datasets, in which one dataset is regarded as the target dataset, while the others serve as source datasets. The experimental results demonstrate that the MDTGAN model consistently outperforms the baseline models. Specifically, the MDTGAN model can transfer valuable knowledge from the source domain datasets to improve data imputation performance on the target domain dataset, providing practical significance for cities lacking historical traffic data.

Robust image hiding network with Frequency and Spatial Attentions

Convert Image Communication (CIC) is a promising technology to protect the privacy of images. Recently, the emergence of robust CIC resistant to JPEG compression has gained due to the widespread use of JPEG compression in image communication. This paper introduces a R̲obust image hiding network with F̲requency and S̲patial A̲ttentions (RFSA) to implement robust CIC. RFSA can hide an image within another image with high robust. It incorporates multiple image attentions corresponding to imperceptibility, recovered image quality, and resistance to JPEG compression, which ensure that secret images are hidden within regions that cause little distortion and can well withstand JPEG compression. Additionally, two encoders, that is, a frequency encoder and a spatial encoder, are mixed to adaptively embed secret images across both frequency and spatial domains. Experimental results demonstrate that the proposed scheme not only maintains high image quality and capacity but also exhibits exceptional resistance to JPEG compression compared to other state-of-the-art image hiding methods. The average Peak Signal-to-Noise Ratio (PSNR) of the recovered image remains at 24.96 dB even under JPEG compression with a quality factor of 55.

GEUKE: A geographic entities uniformly explicit knowledge embedding model

AbstractKnowledge embedding for geographic knowledge graphs can effectively improve computational efficiency and provide support for knowledge reasoning, knowledge answering and other applications of knowledge graphs. To maintain a more comprehensive understanding of spatial features through knowledge embedding, it is crucial to integrate the representation and computation of various entity types, encompassing points, lines, and polygons. This article proposes a geographic entities uniformly explicit knowledge embedding model (GEUKE). In GEUKE, spatial data of point, line, and polygon‐type geographic entities are expressed in the form of subgraphs, and space embedding is generated using a SubGNN‐based uniform spatial feature encoder. GEUKE improves the energy function in TransE to train spatial feature‐based embedding and structural‐based embedding of geographic entities into a unified vector space. Experimental results show that GEUKE has higher performance than TransE, TransH, TransD, and TransE‐GDR on link prediction and triple classification task. Within the spatial feature embedding process, GEUKE effectively preserves the inherent features of entities, encompassing location, neighborhood, and structural attributes, while simultaneously ensuring a coherent spatial data representation across all three entity types: points, lines, and polygons. By maintaining the spatial features of geographic entities and their interrelations, this capability unleashes the full potential of applications such as knowledge reasoning and geospatial question answering in a manner that is conducive to diverse geospatial scenarios.

Deep learning system for malignancy risk prediction in cystic renal lesions: a multicenter study.

To develop an interactive, non-invasive artificial intelligence (AI) system for malignancy risk prediction in cystic renal lesions (CRLs). In this retrospective, multicenter diagnostic study, we evaluated 715 patients. An interactive geodesic-based 3D segmentation model was created for CRLs segmentation. A CRLs classification model was developed using spatial encoder temporal decoder (SETD) architecture. The classification model combines a 3D-ResNet50 network for extracting spatial features and a gated recurrent unit (GRU) network for decoding temporal features from multi-phase CT images. We assessed the segmentation model using sensitivity (SEN), specificity (SPE), intersection over union (IOU), and dice similarity (Dice) metrics. The classification model's performance was evaluated using the area under the receiver operator characteristic curve (AUC), accuracy score (ACC), and decision curve analysis (DCA). From 2012 to 2023, we included 477 CRLs (median age, 57 [IQR: 48-65]; 173 men) in the training cohort, 226 CRLs (median age, 60 [IQR: 52-69]; 77 men) in the validation cohort, and 239 CRLs (median age, 59 [IQR: 53-69]; 95 men) in the testing cohort (external validation cohort 1, cohort 2, and cohort 3). The segmentation model and SETD classifier exhibited excellent performance in both validation (AUC = 0.973, ACC = 0.916, Dice = 0.847, IOU = 0.743, SEN = 0.840, SPE = 1.000) and testing datasets (AUC = 0.998, ACC = 0.988, Dice = 0.861, IOU = 0.762, SEN = 0.876, SPE = 1.000). The AI system demonstrated excellent benign-malignant discriminatory ability across both validation and testing datasets and illustrated improved clinical decision-making utility. In this era when incidental CRLs are prevalent, this interactive, non-invasive AI system will facilitate accurate diagnosis of CRLs, reducing excessive follow-up and overtreatment. The rising prevalence of CRLs necessitates better malignancy prediction strategies. The AI system demonstrated excellent diagnostic performance in identifying malignant CRL. The AI system illustrated improved clinical decision-making utility.

DKNN: deep kriging neural network for interpretable geospatial interpolation

Geospatial interpolation plays a pivotal role in spatial analysis because it provides high-quality data support for various spatiotemporal data mining (STDM) tasks. However, statistical methods, such as kriging, face challenges in dealing with complex geo-big data. Additionally, deep-learning-based methods, despite their exceptional performance, suffer from limitations, such as poor interpretability. To harness the complementary advantages of these statistical methods and deep learning approaches, this study proposes a novel geospatial artificial intelligence (GeoAI) framework called deep kriging neural network (DKNN). The primary contribution lies in the development of an asymmetric encoder-decoder structure, which includes a deep-learning-based spatial encoder and a geostatistics-based kriging decoder. The spatial encoder consists of three specialized neural networks, whereas the kriging decoder relies on the proposed unified kriging system. During forward propagation, the kriging decoder leverage messages from the spatial encoder to generate interpolation weights for prediction. Conversely, during backward propagation, the kriging decoder guides the spatial encoder in learning interpretable knowledge. Experiments were conducted using both synthetic and practical datasets. The results demonstrate an average improvement of 20.18% in MAE, 25.04% in RMSE and 24.06% in MAPE when compared to the best-performing baseline method. Furthermore, these results confirm the superior interpretability of our DKNN framework.

Recurrent Partial Kernel Network for Efficient Optical Flow Estimation

Optical flow estimation is a challenging task consisting of predicting per-pixel motion vectors between images. Recent methods have employed larger and more complex models to improve the estimation accuracy. However, this impacts the widespread adoption of optical flow methods and makes it harder to train more general models since the optical flow data is hard to obtain. This paper proposes a small and efficient model for optical flow estimation. We design a new spatial recurrent encoder that extracts discriminative features at a significantly reduced size. Unlike standard recurrent units, we utilize Partial Kernel Convolution (PKConv) layers to produce variable multi-scale features with a single shared block. We also design efficient Separable Large Kernels (SLK) to capture large context information with low computational cost. Experiments on public benchmarks show that we achieve state-of-the-art generalization performance while requiring significantly fewer parameters and memory than competing methods. Our model ranks first in the Spring benchmark without finetuning, improving the results by over 10% while requiring an order of magnitude fewer FLOPs and over four times less memory than the following published method without finetuning. The code is available at github.com/hmorimitsu/ptlflow/tree/main/ptlflow/models/rpknet.

Two-Level Spatio-Temporal Feature Fused Two-Stream Network for Micro-Expression Recognition.

Micro-expressions, which are spontaneous and difficult to suppress, reveal a person's true emotions. They are characterized by short duration and low intensity, making the task of micro-expression recognition challenging in the field of emotion computing. In recent years, deep learning-based feature extraction and fusion techniques have been widely used for micro-expression recognition, particularly methods based on Vision Transformer that have gained popularity. However, the Vision Transformer-based architecture used in micro-expression recognition involves a significant amount of invalid computation. Additionally, in the traditional two-stream architecture, although separate streams are combined through late fusion, only the output features from the deepest level of the network are utilized for classification, thus limiting the network's ability to capture subtle details due to the lack of fine-grained information. To address these issues, we propose a new two-level spatio-temporal feature fused with a two-stream architecture. This architecture includes a spatial encoder (modified ResNet) for learning texture features of the face, a temporal encoder (Swin Transformer) for learning facial muscle motor features, a feature fusion algorithm for integrating multi-level spatio-temporal features, a classification head, and a weighted average operator for temporal aggregation. The two-stream architecture has the advantage of extracting richer features compared to the single-stream architecture, leading to improved performance. The shifted window scheme of Swin Transformer restricts self-attention computation to non-overlapping local windows and allows cross-window connections, significantly improving the performance and reducing the computation compared to Vision Transformer. Moreover, the modified ResNet is computationally less intensive. Our proposed feature fusion algorithm leverages the similarity in output feature shapes at each stage of the two streams, enabling the effective fusion of multi-level spatio-temporal features. This algorithm results in an improvement of approximately 4% in both the F1 score and the UAR. Comprehensive evaluations conducted on three widely used spontaneous micro-expression datasets (SMIC-HS, CASME II, and SAMM) consistently demonstrate the superiority of our approach over comparative methods. Notably, our approach achieves a UAR exceeding 0.905 on CASME II, making it one of the few frameworks in the published micro-expression recognition literature to achieve such high performance.

ERNetCL: A novel emotion recognition network in textual conversation based on curriculum learning strategy

Emotion recognition in conversation (ERC) has emerged as a research hotspot in domains such as conversational robots and question-answer systems. How to efficiently and adequately retrieve contextual emotional cues has been one of the key challenges in the ERC task. Existing efforts do not fully model the context and employ complex network structures, resulting in limited performance gains. In this paper, we propose a novel emotion recognition network based on curriculum learning strategy (ERNetCL). The proposed ERNetCL primarily consists of temporal encoder (TE), spatial encoder (SE), and curriculum learning (CL) loss. We utilize TE and SE to combine the strengths of previous methods in a simplistic manner to efficiently capture temporal and spatial contextual information in the conversation. To ease the harmful influence resulting from emotion shift and simulate the way humans learn curriculum from easy to hard, we apply the idea of CL to the ERC task to progressively optimize the network parameters. At the beginning of training, we assign lower learning weights to difficult samples. As the epoch increases, the learning weights for these samples are gradually raised. Extensive experiments on four datasets exhibit that our proposed method is effective and dramatically beats other baseline models.

TinyPredNet: A Lightweight Framework for Satellite Image Sequence Prediction

Satellite image sequence prediction aims to precisely infer future satellite image frames with historical observations, which is a significant and challenging dense prediction task. Though existing deep learning models deliver promising performance for satellite image sequence prediction, the methods suffer from quite expensive training costs, especially in training time and GPU memory demand, due to the inefficiently modeling for temporal variations. This issue seriously limits the lightweight application in satellites such as space-borne forecast models. In this article, we propose a lightweight prediction framework TinyPredNet for satellite image sequence prediction, in which a spatial encoder and decoder model the intra-frame appearance features and a temporal translator captures inter-frame motion patterns. To efficiently model the temporal evolution of satellite image sequences, we carefully design a multi-scale temporal-cascaded structure and a channel attention-gated structure in the temporal translator. Comprehensive experiments are conducted on FengYun-4A (FY-4A) satellite dataset, which show that the proposed framework achieves very competitive performance with much lower computation cost compared to state-of-the-art methods. In addition, corresponding interpretability experiments are conducted to show how our designed structures work. We believe the proposed method can serve as a solid lightweight baseline for satellite image sequence prediction.

MM-SFENet: multi-scale multi-task localization and classification of bladder cancer in MRI with spatial feature encoder network.

Objective. Bladder cancer is a common malignant urinary carcinoma, with muscle-invasive and non-muscle-invasive as its two major subtypes. This paper aims to achieve automated bladder cancer invasiveness localization and classification based on MRI.Approach. Different from previous efforts that segment bladder wall and tumor, we propose a novel end-to-end multi-scale multi-task spatial feature encoder network (MM-SFENet) for locating and classifying bladder cancer, according to the classification criteria of the spatial relationship between the tumor and bladder wall. First, we built a backbone with residual blocks to distinguish bladder wall and tumor; then, a spatial feature encoder is designed to encode the multi-level features of the backbone to learn the criteria.Main Results. We substitute Smooth-L1 Loss with IoU Loss for multi-task learning, to improve the accuracy of the classification task. By learning two datasets collected from bladder cancer patients at the hospital, the mAP, IoU, Acc, Sen and Spec are used as the evaluation metrics. The experimental result could reach 93.34%, 83.16%, 85.65%, 81.51%, 89.23% on test set1 and 80.21%, 75.43%, 79.52%, 71.87%, 77.86% on test set2.Significance. The experimental result demonstrates the effectiveness of the proposed MM-SFENet on the localization and classification of bladder cancer. It may provide an effective supplementary diagnosis method for bladder cancer staging.

Two-pathway spatiotemporal representation learning for extreme water temperature prediction

Accurate predictions of extreme water temperatures are criticalto understanding the variability of the marine environment and reducing marine disasters maximized by global warming. In this study, we propose a two-pathway framework with separated spatial and temporal encoders for accurate prediction of water temperature, especially extremely high water temperature, through effective spatiotemporal representation learning. The spatial and temporal encoder networks based on the Transformer’s self-attention mechanism performs the task of predicting the water temperature time series at the 16 coastal locations around the Korean Peninsula for the seven consecutive days ahead at daily intervals with various combinations of patch embedding methods, positional embedding for spatial features. Comparative experiments with conventional deep convolutional and recurrent networks are also conducted for comparison. By comparing and assessing these results, the proposed two-pathway framework can improve the predictability of extremely high coastal water temperature by better capturing spatiotemporal interrelationships and long-range dependencies from open ocean and regional sea, and further determines the optimal architectural details of self-attention-based spatial and temporal encoders. Furthermore, to examine the explainability of the proposed model and its consistency with domain knowledge, spatial and temporal attention maps are visualized and analyzed that represents weights for spatiotemporal input sequences that are more relevant to predict for future predictions.

TASA: Temporal Attention With Spatial Autoencoder Network for Odor-Induced Emotion Classification Using EEG.

The olfactory system enables humans to smell different odors, which are closely related to emotions. The high temporal resolution and non-invasiveness of Electroencephalogram (EEG) make it suitable to objectively study human preferences for odors. Effectively learning the temporal dynamics and spatial information from EEG is crucial for detecting odor-induced emotional valence. In this paper, we propose a deep learning architecture called Temporal Attention with Spatial Autoencoder Network (TASA) for predicting odor-induced emotions using EEG. TASA consists of a filter-bank layer, a spatial encoder, a time segmentation layer, a Long Short-Term Memory (LSTM) module, a multi-head self-attention (MSA) layer, and a fully connected layer. We improve upon the previous work by utilizing a two-phase learning framework, using the autoencoder module to learn the spatial information among electrodes by reconstructing the given input with a latent representation in the spatial dimension, which aims to minimize information loss compared to spatial filtering with CNN. The second improvement is inspired by the continuous nature of the olfactory process; we propose to use LSTM-MSA in TASA to capture its temporal dynamics by learning the intercorrelation among the time segments of the EEG. TASA is evaluated on an existing olfactory EEG dataset and compared with several existing deep learning architectures to demonstrate its effectiveness in predicting olfactory-triggered emotional responses. Interpretability analyses with DeepLIFT also suggest that TASA learns spatial-spectral features that are relevant to olfactory-induced emotion recognition.