Local Window Research Articles

ST-SHAP: A hierarchical and explainable attention network for emotional EEG representation learning and decoding

Background:Emotion recognition using electroencephalogram (EEG) has become a research hotspot in the field of human–computer interaction, how to sufficiently learn complex spatial–temporal representations of emotional EEG data and obtain explainable model prediction results are still great challenges. New methodIn this study, a novel hierarchical and explainable attention network ST-SHAP which combines the Swin Transformer (ST) and SHapley Additive exPlanations (SHAP) technique is proposed for automatic emotional EEG classification. Firstly, a 3D spatial–temporal feature of emotional EEG data is generated via frequency band filtering, temporal segmentation, spatial mapping, and interpolation to fully preserve important spatial–temporal-frequency characteristics. Secondly, a hierarchical attention network is devised to sufficiently learn an abstract spatial–temporal representation of emotional EEG and perform classification. Concretely, in this decoding model, the W-MSA module is used for modeling correlations within local windows, the SW-MSA module allows for information interactions between different local windows, and the patch merging module further facilitates local-to-global multiscale modeling. Finally, the SHAP method is utilized to discover important brain regions for emotion processing and improve the explainability of the Swin Transformer model. Results:Two benchmark datasets, namely SEED and DREAMER, are used for classification performance evaluation. In the subject-dependent experiments, for SEED dataset, ST-SHAP achieves an average accuracy of 97.18%, while for DREAMER dataset, the average accuracy is 96.06% and 95.98% on arousal and valance dimension respectively. In addition, import brain regions that conform to prior knowledge of neurophysiology are discovered via a data-driven approach for both datasets. Comparison with existing methods:In terms of subject-dependent and subject-independent emotional EEG decoding accuracies, our method outperforms several closely related existing methods. Conclusion:These experimental results fully prove the effectiveness and superiority of our proposed algorithm.

Protein language models learn evolutionary statistics of interacting sequence motifs

Protein language models (pLMs) have emerged as potent tools for predicting and designing protein structure and function, and the degree to which these models fundamentally understand the inherent biophysics of protein structure stands as an open question. Motivated by a finding that pLM-based structure predictors erroneously predict nonphysical structures for protein isoforms, we investigated the nature of sequence context needed for contact predictions in the pLM Evolutionary Scale Modeling (ESM-2). We demonstrate by use of a "categorical Jacobian" calculation that ESM-2 stores statistics of coevolving residues, analogously to simpler modeling approaches like Markov Random Fields and Multivariate Gaussian models. We further investigated how ESM-2 "stores" information needed to predict contacts by comparing sequence masking strategies, and found that providing local windows of sequence information allowed ESM-2 to best recover predicted contacts. This suggests that pLMs predict contacts by storing motifs of pairwise contacts. Our investigation highlights the limitations of current pLMs and underscores the importance of understanding the underlying mechanisms of these models.

Reinforcement learning-based method for type B aortic dissection localization

In the segmentation of aortic dissection, there are issues such as low contrast between the aortic dissection and surrounding organs and vessels, significant differences in dissection morphology, and high background noise. To address these issues, this paper proposed a reinforcement learning-based method for type B aortic dissection localization. With the assistance of a two-stage segmentation model, the deep reinforcement learning was utilized to perform the first-stage aortic dissection localization task, ensuring the integrity of the localization target. In the second stage, the coarse segmentation results from the first stage were used as input to obtain refined segmentation results. To improve the recall rate of the first-stage segmentation results and include the segmentation target more completely in the localization results, this paper designed a reinforcement learning reward function based on the direction of recall changes. Additionally, the localization window was separated from the field of view window to reduce the occurrence of segmentation target loss. Unet, TransUnet, SwinUnet, and MT-Unet were selected as benchmark segmentation models. Through experiments, it was verified that the majority of the metrics in the two-stage segmentation process of this paper performed better than the benchmark results. Specifically, the Dice index improved by 1.34%, 0.89%, 27.66%, and 7.37% for each respective model. In conclusion, by incorporating the type B aortic dissection localization method proposed in this paper into the segmentation process, the overall segmentation accuracy is improved compared to the benchmark models. The improvement is particularly significant for models with poorer segmentation performance.

Stereo and LiDAR Loosely Coupled SLAM Constrained Ground Detection.

In many robotic applications, creating a map is crucial, and 3D maps provide a method for estimating the positions of other objects or obstacles. Most of the previous research processes 3D point clouds through projection-based or voxel-based models, but both approaches have certain limitations. This paper proposes a hybrid localization and mapping method using stereo vision and LiDAR. Unlike the traditional single-sensor systems, we construct a pose optimization model by matching ground information between LiDAR maps and visual images. We use stereo vision to extract ground information and fuse it with LiDAR tensor voting data to establish coplanarity constraints. Pose optimization is achieved through a graph-based optimization algorithm and a local window optimization method. The proposed method is evaluated using the KITTI dataset and compared against the ORB-SLAM3, F-LOAM, LOAM, and LeGO-LOAM methods. Additionally, we generate 3D point cloud maps for the corresponding sequences and high-definition point cloud maps of the streets in sequence 00. The experimental results demonstrate significant improvements in trajectory accuracy and robustness, enabling the construction of clear, dense 3D maps.

An Entropy-based Method to Evaluate the Appropriate Spatial Resolution

Abstract. Geospatial data is available at increasingly finer spatial resolutions. However, such data can also be problematic because it may result in increased financial, resource and time cost. It might also be unnecessary if the phenomenon of interest does not vary at this scale or if the scientific application does not require it. This paper explores the scale effects associated with increased spatial resolution using the entropy-based local indicator of spatial association (LISA-ELSA). Results are illustrated for two datasets a digital elevation model (30 m ASTER GDEM) and for a raster PM2.5 air pollution map (1000 m). It is shown that increasing the size of the local window for the LISA can be used to explore the effect of reducing the spatial resolution. It is possible to identify areas where the spatial association is invariant with increasing window size and which could be represented with coarser pixels.

SPDepth: Enhancing Self-Supervised Indoor Monocular Depth Estimation via Self-Propagation

Due to the existence of low-textured areas in indoor scenes, some self-supervised depth estimation methods have specifically designed sparse photometric consistency losses and geometry-based losses. However, some of the loss terms cannot supervise all the pixels, which limits the performance of these methods. Some approaches introduce an additional optical flow network to provide dense correspondences supervision, but overload the loss function. In this paper, we propose to perform depth self-propagation based on feature self-similarities, where high-accuracy depths are propagated from supervised pixels to unsupervised ones. The enhanced self-supervised indoor monocular depth estimation network is called SPDepth. Since depth self-similarities are significant in a local range, a local window self-attention module is embedded at the end of the network to propagate depths in a window. The depth of a pixel is weighted using the feature correlation scores with other pixels in the same window. The effectiveness of self-propagation mechanism is demonstrated in the experiments on the NYU Depth V2 dataset. The root-mean-squared error of SPDepth is 0.585 and the δ1 accuracy is 77.6%. Zero-shot generalization studies are also conducted on the 7-Scenes dataset and provide a more comprehensive analysis about the application characteristics of SPDepth.

Experimental evaluation of subjective thermal perceptions for different local window screenings

In tropical climate, restricting convective gain while allowing cross ventilation creates paradoxical difficulty regarding indoor thermal comfort. Changing global climate gives new dimensions to the scenario results in subsequent pressure on increasing energy demand towards thermal comfort. Passive techniques can be a good way to dealt with the problem. However, occupants’ thermal perception may vary from person to person for a definite indoor environmental setup. Perception of subjective thermal comfort is crucial for human health and wellbeing which in turns effects performance. In the current study evaluation of subjective thermal performance regarding improved indoor thermal comfort has been studied for three different local window screening: i) plastic bottle with its wider face towards wind direction, ii) plastic bottle with its narrower face towards wind direction and iii) perforated bamboo screen. Six subjects: four female and two male (ages between 22-24), have been investigated for 30mins. in an experimental simulation chamber for three consecutive days. Questionnaire regarding thermal sensation and perception has been prepared to collect subjective response at 10mins. interval. From the experiment it is seen that subjective responses varies for different window screenings as well as for a specific indoor thermal condition created with definite screening. Subjects find the indoor environment comfortable for screening with plastic bottle with its wider face towards wind direction compared to other two. Bamboo screening has comparatively better performance than the screening plastic bottle with its narrower face towards wind direction. The current analysis only considers the thermal sensation of the subject and further extension of study considering factors like the subject's site-specific thermal sensation and other psychological effects can contribute towards improving indoor thermal comfort in tropics towards human health and wellbeing.

Application of a novel local and automatic PCA algorithm for diffraction pattern denoising in TEM-ASTAR analysis in microelectronics

This paper introduces a novel denoising method for TEM-ASTAR™ Diffraction Pattern (DP) datasets, termed LAT–PCA (Local Automatic Thresholding – Principal Component Analysis). This approach enhances the established PCA algorithm by partitioning the 4D dataset (a 2D map of 2D DPs) into localized windows. Within these windows, PCA identifies a basis where the physical signal predominantly resides in the higher-order principal components. By thresholding lower-order components, the method effectively reduces noise while retaining the essential features of the DPs. The locality of the approach, focusing on small windows, enhances computational efficiency and aligns with the localized nature of the crystallographic grain signals in ASTAR. The automatic aspect of the method employs a theoretical pure noise distribution, i.e. a Marchenko-Pastur Distribution, to set a threshold, beyond which the components are mostly noise.The LAT–PCA method offers significant reductions in acquisition and post-processing times. With denoised data, selecting the correct parameters for accurate phase maps and grain orientations becomes more straightforward, facilitating robust quantitative grain analysis. Experiments performed on a silicon-germanium-carbon sample validate the method's efficacy. The sample was analyzed with varying acquisition times to produce a high signal-to-noise ratio reference dataset and a lower ratio test dataset. The LAT–PCA algorithm's denoising results on the test dataset were benchmarked against the reference, demonstrating considerable improvements and reliability.In summary, LAT–PCA is an effective, automatic solution for denoising TEM DP datasets. Its adaptability to different noise levels and local processing capability makes it a valuable tool for enhancing dataset quality and reducing the time required for data acquisition and analysis. This method can minimize acquisition time, conserve microscope usage, and reduce sample drift and deterioration, leading to more accurate characterizations with fewer deformation artifacts.

AMTrack:Transformer tracking via action information and mix-frequency features

Nowadays, Transformer-based visual tracking algorithms have been developing quickly because of the self-attention mechanism of Transformer, which has the capability to model global information. Although the self-attention mechanism in Transformer can effectively capture long-range dependencies in feature space, they only use flattened two-dimensional features and are unable to capture long-range temporal dependencies. Furthermore, since the self-attention in Transformer functions as a low-pass filter, it picks up on low-frequency features of the target while ignoring high-frequency features. This research suggests a Transformer tracker based on action information and mix-frequency features (AMTrack) to address these problems. Specifically, to address the lack of temporal remote dependencies, we introduce the target action aware module and the target action offset module. The target action aware module sets up several pathways to extract spatio-temporal, channel, and motion feature independently. In contrast, the target action offset module computes the target’s offset information by computing relative feature maps. Furthermore, in order to address the imbalance between high and low frequency features, we propose a mix-frequency attention and multi-frequency self-attention convolutional block. The mix-frequency attention uses high-frequency features within partitioned local windows as input for the high-frequency branch and average-pooled low-frequency features as the input for the low-frequency branch, calculating attention scores in both branches respectively. The multi-frequency self-attention convolutional block uses self-attention to capture low-frequency features and convolution to capture high-frequency features. Extensive experiments are carried out on eight challenging tracking datasets (e.g., OTB100 (Object Tracking Benchmark 100), NFS (Need For Speed), UAV123 (Unmanned Aerial Vehicles 123), TC128 (Temple Color 128), VOT2018 (Visual Object Tracking 2018), LaSOT (Large-scale Single Object Tracking), TrackingNet (Tracking Network), GOT-10k (Generic Object Tracking-10k)), and the experimental results show that our tracker achieves excellent tracking performance when compared with several state-of-the-art tracking algorithms. The experimental results show that on LaSOT, the success rates AUC (Area Under Curve), PNorm, and P reach 65.8%, 69.2%, and 68.0%, respectively, where the AUC value is 2.1% higher than the baseline algorithm TrDiMP (Transformer Discriminative Model Prediction). On other datasets, our tracker also achieves excellent tracking performance.

Feature matching based on local windows aggregation

The core goal of feature matching is to establish correspondences between two images. Current methods without detectors achieve impressive results but often focus on global features, neglecting regions with subtle textures and resulting in fewer matches in areas with weak textures. This paper proposes a feature-matching method based on local window aggregation, which balances global features and local texture variations for more accurate matches, especially in weak-texture regions. Our method first applies a local window aggregation module to minimize irrelevant interference using window attention, followed by global attention, generating coarse and fine-grained feature maps. These maps are processed by a matching module, initially obtaining coarse matches via the nearest neighbor principle. The coarse matches are then refined on fine-grained maps through local window refinement. Experimental results show our method surpasses state-of-the-art techniques in pose estimation, homography estimation, and visual localization under the same training conditions.

DRRGlobal: Uncovering the weak phases from global seismograms using the damped rank-reduction method

Some target seismic signals in the earthquake data can be very weak compared with interfering phases, and are thus difficult to detect, which further hinders the effective usage of these weak phases for subsequent high-resolution imaging of earth interiors. The strong ambient noise makes this situation even more troublesome since the weak signals can be mostly buried in the noise. Here, we present an open-source package for uncovering the weak phases from global seismograms. We adopt a two-step scheme to reconstruct and denoise array data. The first step is weighted average interpolation which puts the data into irregular grids. The second step adopts the weighted projection-onto-convex sets based on damped rank-reduction to further interpolate and denoise for the binned data. Taking the complexity of the weak signal into consideration, we adopt the automatic strategy to select an appropriate rank in different localized windows. We conduct several synthetic tests to carefully investigate the performance regarding effectiveness, robustness, and efficiency, and compare the algorithm with the frequency–wavenumber-domain projection onto convex sets method that is already used in the global seismology literature. Finally, the proposed framework is validated via a recorded array data set of the 1995 May 5 Philippines earthquake.

Deep learning models for regional phase detection on seismic stations in Northern Europe and the European Arctic

SUMMARY Seismic phase detection and classification using deep learning is so far poorly investigated for regional events since most studies focus on local events and short time windows as the input to the detection models. To evaluate deep learning on regional seismic records, we create a data set of events in Northern Europe and the European Arctic. This data set consists of about 151 000 three component event waveforms and corresponding phase arrival picks at stations in mainland Norway, Finland and Svalbard. We train several state-of-the-art and one newly developed deep learning model on this data set to pick P- and S-wave arrivals. The new method modifies the popular PhaseNet model with new convolutional blocks including transformers. This yields more accurate predictions on the long input time windows associated with regional events. Evaluated on event records not used for training, our new method improves the performance of the current state-of-the-art methods when it comes to recall, precision and pick time residuals. Finally, we test our new model for continuous mode processing on 4 d of single-station data from the ARCES array. Results show that our new method outperforms the existing array detector at ARCES. This opens up new opportunities to improve automatic array processing with deep learning detectors.

Depth Image Rectification Based on an Effective RGB–Depth Boundary Inconsistency Model

Depth image has been widely involved in various tasks of 3D systems with the advancement of depth acquisition sensors in recent years. Depth images suffer from serious distortions near object boundaries due to the limitations of depth sensors or estimation methods. In this paper, a simple method is proposed to rectify the erroneous object boundaries of depth images with the guidance of reference RGB images. First, an RGB–Depth boundary inconsistency model is developed to measure whether collocated pixels in depth and RGB images belong to the same object. The model extracts the structures of RGB and depth images, respectively, by Gaussian functions. The inconsistency of two collocated pixels is then statistically determined inside large-sized local windows. In this way, pixels near object boundaries of depth images are identified to be erroneous when they are inconsistent with collocated ones in RGB images. Second, a depth image rectification method is proposed by embedding the model into a simple weighted mean filter (WMF). Experiment results on two datasets verify that the proposed method well improves the RMSE and SSIM of depth images by 2.556 and 0.028, respectively, compared with recent optimization-based and learning-based methods.

Stereo Matching Method with Cost Volume Collaborative Filtering

Aiming at the problem of matching ambiguity and low disparity accuracy at the object boundary in stereo matching, a novel stereo matching algorithm with cost volume collaborative filtering is proposed. Firstly, for each pixel, two support windows are built, namely a local cross- support window as well as a global support window for the whole image. Secondly, a new adaptive weighted guide filter with a cross-support window as a kernel window is derived, and it is used to locally filter the cost volume. In addition, a minimum spanning tree is constructed in the whole image window, and then the minimum spanning tree filter is used to globally filter the cost volume. The collaborative filtering of cost volume is realized by fusing the filtering results of the local filter and global filter, so that each pixel can not only receive the support of the neighboring pixels in the local adaptive window, but can also receive the effective support of other pixels in the whole image, thus effectively eliminating the matching ambiguity in different texture regions while maintaining the disparity edges. The experimental results show that the average matching error rate of our method on the Middlebury stereo images is 3.17%. Compared with the other state-of-the-art methods, our method has higher robustness and matching accuracy, the generated disparity maps are smoother, and the disparity edges are better preserved.

Learning Complementary Spatial-Temporal Transformer for Video Salient Object Detection.

Besides combining appearance and motion information, another crucial factor for video salient object detection (VSOD) is to mine spatial-temporal (ST) knowledge, including complementary long-short temporal cues and global-local spatial context from neighboring frames. However, the existing methods only explored part of them and ignored their complementarity. In this article, we propose a novel complementary ST transformer (CoSTFormer) for VSOD, which has a short-global branch and a long-local branch to aggregate complementary ST contexts. The former integrates the global context from the neighboring two frames using dense pairwise attention, while the latter is designed to fuse long-term temporal information from more consecutive frames with local attention windows. In this way, we decompose the ST context into a short-global part and a long-local part and leverage the powerful transformer to model the context relationship and learn their complementarity. To solve the contradiction between local window attention and object motion, we propose a novel flow-guided window attention (FGWA) mechanism to align the attention windows with object and camera movements. Furthermore, we deploy CoSTFormer on fused appearance and motion features, thus enabling the effective combination of all three VSOD factors. Besides, we present a pseudo video generation method to synthesize sufficient video clips from static images for training ST saliency models. Extensive experiments have verified the effectiveness of our method and illustrated that we achieve new state-of-the-art results on several benchmark datasets.

Erosion and impurity transport for the edge localized mode suppression window in KSTAR

Erosion and impurity transport for the edge localized mode suppression window in KSTAR

LDConv: Linear deformable convolution for improving convolutional neural networks

Neural networks based on convolutional operations have achieved remarkable results in the field of deep learning, but there are two inherent flaws in standard convolutional operations. On the one hand, the convolution operation is confined to a local window, so it cannot capture information from other locations, and its sampled shapes is fixed. On the other hand, the size of the convolutional kernel is fixed to k × k, which is a fixed square shape, and the number of parameters tends to grow squarely with size. Although Deformable Convolution (Deformable Conv) address the problem of fixed sampling of standard convolutions, the number of parameters also tends to grow in a squared manner, and Deformable Conv do not explore the effect of different initial sample shapes on network performance. In response to the above questions, the Linear Deformable Convolution (LDConv) is explored in this work, which gives the convolution kernel an arbitrary number of parameters and arbitrary sampled shapes to provide richer options for the trade-off between network overhead and performance. In LDConv, a novel coordinate generation algorithm is defined to generate different initial sampled positions for convolutional kernels of arbitrary size. To adapt to changing targets, offsets are introduced to adjust the shape of the samples at each position. LDConv corrects the growth trend of the number of parameters for standard convolution and Deformable Conv to a linear growth. Compared to Deformable Conv, LDConv provides richer choices and can be equivalent to deformable convolution when the number of parameters of LDConv is set to the square of K. Differently, this paper also explores the effect of neural networks by using LDConv with the same size and different initial sampling shapes. LDConv completes the process of efficient feature extraction by irregular convolutional operations and brings more exploration options for convolutional sampled shapes. Object detection experiments on representative datasets COCO2017, VOC 7 + 12, and VisDrone-DET2021 fully demonstrate the advantages of LDConv. LDConv is a plug-and-play convolutional operation that can replace the convolutional operation to improve network performance. The code for the relevant tasks can be found at https://github.com/CV-ZhangXin/LDConv.

Reconstruction and denoising of high-dimensional seismic data via Frobenius-nuclear mixed norm constraints

Abstract The seismic data acquisition design with ‘two-wide and one-high’ geometry effectively improves the imaging quality of seismic records. However, when data are acquired in the field, complex near surface conditions and environmental factors can introduce a variety of noises and gaps in seismic data, impacting the accuracy of seismic imaging. Currently, the method of low-rank matrix/tensor completion is commonly employed for data reconstruction after normal moveout (NMO). In a complex subsurface medium, common midpoint data processed with NMO may not satisfy the linear or quasi-linear assumptions within local data windows. Therefore, this paper exploits the inherent low-rank structure of high-dimensional data to propose a high-dimensional tensor completion method under the Frobenius-nuclear mixed norm constraint (FN-TC). This method unfolds the 4D data tensor into the frequency-space domain along its modes (m, n) and subsequently imposes a non-convex Frobenius-nuclear mixed norm constraint on the unfolded approximate matrices. This approach closely approximates the rank function of the factor matrices, thereby enhancing the accuracy of data modeling. Theoretical and practical studies demonstrate that the novel FN-TC approach can effectively reconstruct high-dimensional seismic data and suppress noise, thereby providing data support for subsequent high-precision seismic imaging.

Sparse transformer with local and seasonal adaptation for multivariate time series forecasting

Transformers have achieved remarkable performance in multivariate time series(MTS) forecasting due to their capability to capture long-term dependencies. However, the canonical attention mechanism has two key limitations: (1) its quadratic time complexity limits the sequence length, and (2) it generates future values from the entire historical sequence. To address this, we propose a Dozer Attention mechanism consisting of three sparse components: (1) Local, each query exclusively attends to keys within a localized window of neighboring time steps. (2) Stride, enables each query to attend to keys at predefined intervals. (3) Vary, allows queries to selectively attend to keys from a subset of the historical sequence. Notably, the size of this subset dynamically expands as forecasting horizons extend. Those three components are designed to capture essential attributes of MTS data, including locality, seasonality, and global temporal dependencies. Additionally, we present the Dozerformer Framework, incorporating the Dozer Attention mechanism for the MTS forecasting task. We evaluated the proposed Dozerformer framework with recent state-of-the-art methods on nine benchmark datasets and confirmed its superior performance. The experimental results indicate that excluding a subset of historical time steps from the time series forecasting process does not compromise accuracy while significantly improving efficiency. Code is available at https://github.com/GRYGY1215/Dozerformer.

CURE: A Hierarchical Framework for Multi-Robot Autonomous Exploration Inspired by Centroids of Unknown Regions

In this paper, a novel multi-robot autonomous exploration approach CURE is proposed based on dynamic Voronoi diagrams and centroids of unknown connected regions. Compared with existing approaches, the novelty of this work is twofold: 1) Dynamic Voronoi diagram is used for partition of the space being explored to improve the efficiency of multi-robot exploration, and then a new parameter-insensitive utility function is elaborately designed to evaluate the information of centroids, which helps guide the robot to explore unknown regions. 2) A hierarchical framework consisting of global and local exploration windows for detecting centroids is designed, wherein the global exploration window is activated to find centroids to guide the robot exploration when there are no centroids in any one local exploration window. We validate the feasibility and exploration efficiency of the proposed approach in various complex simulation scenarios and challenging real-world tasks. All test results show that the exploration time consumption and path cost are reduced by up to 50.7% and 34.4%, respectively, compared with an advanced RRT-based multi-robot exploration approach. (Supplementary video link: https://youtu.be/P5jXKlGQOec) <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This paper is motivated by the efficient multi-robot autonomous exploration problem. In some applications such as target search and disaster rescue, the information about the environment is totally unknown to the robots, and thus they are required to explore unknown environments autonomously. In this case, it is necessary to improve the efficiency of multi-robot exploration due to the time limitation of the task and the battery capacity. In this paper, a hierarchical framework is proposed to improve the efficiency of multi-robot autonomous exploration. Each robot only needs to explore the Voronoi partition it is responsible for and is guided to the unknown region by the centroid detected in the global and local exploration windows. Overall, the proposed approach can dramatically reduce the exploration time and path cost.