Self-attention Module Research Articles

Combining external-latent attention for medical image segmentation

The attention mechanism comes as a new entry point for improving the performance of medical image segmentation. How to reasonably assign weights is a key element of the attention mechanism, and the current popular schemes include the global squeezing and the non-local information interactions using self-attention (SA) operation. However, these approaches over-focus on external features and lack the exploitation of latent features. The global squeezing approach crudely represents the richness of contextual information by the global mean or maximum value, while non-local information interactions focus on the similarity of external features between different regions. Both ignore the fact that the contextual information is presented more in terms of the latent features like the frequency change within the data. To tackle above problems and make proper use of attention mechanisms in medical image segmentation, we propose an external-latent attention collaborative guided image segmentation network, named TransGuider. This network consists of three key components: 1) a latent attention module that uses an improved entropy quantification method to accurately explore and locate the distribution of latent contextual information. 2) an external self-attention module using sparse representation, which can preserve external global contextual information while reducing computational overhead by selecting representative feature description map for SA operation. 3) a multi-attention collaborative module to guide the network to continuously focus on the region of interest, refining the segmentation mask. Our experimental results on several benchmark medical image segmentation datasets show that TransGuider outperforms the state-of-the-art methods, and extensive ablation experiments demonstrate the effectiveness of the proposed components. Our code will be available at https://github.com/chasingone/TransGuider.

A multi-task deep learning model for EGFR genotyping prediction and GTV segmentation of brain metastasis

BackgroundThe precise prediction of epidermal growth factor receptor (EGFR) mutation status and gross tumor volume (GTV) segmentation are crucial goals in computer-aided lung adenocarcinoma brain metastasis diagnosis. However, these two tasks present continuous difficulties due to the nonuniform intensity distributions, ambiguous boundaries, and variable shapes of brain metastasis (BM) in MR images.The existing approaches for tackling these challenges mainly rely on single-task algorithms, which overlook the interdependence between these two tasks.MethodsTo comprehensively address these challenges, we propose a multi-task deep learning model that simultaneously enables GTV segmentation and EGFR subtype classification. Specifically, a multi-scale self-attention encoder that consists of a convolutional self-attention module is designed to extract the shared spatial and global information for a GTV segmentation decoder and an EGFR genotype classifier. Then, a hybrid CNN-Transformer classifier consisting of a convolutional block and a Transformer block is designed to combine the global and local information. Furthermore, the task correlation and heterogeneity issues are solved with a multi-task loss function, aiming to balance the above two tasks by incorporating segmentation and classification loss functions with learnable weights.ResultsThe experimental results demonstrate that our proposed model achieves excellent performance, surpassing that of single-task learning approaches. Our proposed model achieves a mean Dice score of 0.89 for GTV segmentation and an EGFR genotyping accuracy of 0.88 on an internal testing set, and attains an accuracy of 0.81 in the EGFR genotype prediction task and an average Dice score of 0.85 in the GTV segmentation task on the external testing set. This shows that our proposed method has outstanding performance and generalization.ConclusionWith the introduction of an efficient feature extraction module, a hybrid CNN-Transformer classifier, and a multi-task loss function, the proposed multi-task deep learning network significantly enhances the performance achieved in both GTV segmentation and EGFR genotyping tasks. Thus, the model can serve as a noninvasive tool for facilitating clinical treatment.

Semantic Segmentation of Urban Airborne LiDAR Point Clouds Based on Fusion Attention Mechanism and Multi-Scale Features

Semantic segmentation of point clouds provided by airborne LiDAR survey in urban scenes is a great challenge. This is due to the fact that point clouds at boundaries of different types of objects are easy to be mixed and have geometric spatial similarity. In addition, the 3D descriptions of the same type of objects have different scales. To address above problems, a fusion attention convolutional network (SMAnet) was proposed in this study. The fusion attention module includes a self-attention module (SAM) and multi-head attention module (MAM). The SAM can capture feature information according to correlation of adjacent point cloud and it can distinguish the mixed point clouds with similar geometric features effectively. The MAM strengthens connections among point clouds according to different subspace features, which is beneficial for distinguishing point clouds at different scales. In feature extraction, lightweight multi-scale feature extraction layers are used to effectively utilize local information of different neighbor fields. Additionally, in order to solve the feature externalization problem and expand the network receptive field, the SoftMax-stochastic pooling (SSP) algorithm is proposed to extract global features. The ISPRS 3D Semantic Labeling Contest dataset was chosen in this study for point cloud segmentation experimentation. Results showed that the overall accuracy and average F1-score of SMAnet reach 85.7% and 75.1%, respectively. It is therefore superior to common algorithms at present. The proposed model also achieved good results on the GML(B) dataset, which proves that the model has good generalization ability.

Dual-Stream Transformer With Distribution Alignment for Visible-Infrared Person Re-Identification

Visible-infrared person re-identification(VI-ReID) aims to match the person images captured by visible and infrared cameras and suffers from severe cross-modality discrepancy and intra-modality variations. Existing approaches mainly use convolution neural network (CNN)-based architectures to extract pedestrian features, which fail to capture the long-range dependencies within an image. In addition, previous works usually attempt to bridge the modality gap by using adversarial learning to generate style-consistent images or designing different feature-level metric learning constraints. However, few works consider the cross-modality disparity from the perspective of assessing overall distance distribution discrepancy. To address these problems, we design a pure Transformer-based Visible-Infrared (TransVI) network with a conventional two-stream structure, which can explicitly capture modality-specific representations and learn multi-modality sharable knowledge. TransVI can efficiently address the lack of global dependency in CNN-based architectures due to the multi-head self-attention modules in the transformer, which allows us to capture the long-range dependencies of pedestrian images. Furthermore, we introduce the Cross-Modality Dissimilarity-based Maximum Mean Discrepancy (CMD-MMD) constraint to handle the cross-modality discrepancy at the distance distribution level. Specifically, CMD-MMD leverages intra-modality distribution separability to guide inter-modality distribution separability learning, aligning pair-wise distance distributions of intra- and inter-modality for within-class and between-class, respectively. In this way, the distance distributions of intra- and inter-modality become more similar, significantly mitigating the cross-modality discrepancy and learning more modality invariant representations. Extensive experimental results on two public VI-ReID datasets confirm that our proposed framework can achieve state-of-the-art performance.

Explainable predictions of multi-component oxides enabled by attention-based neural networks

Multicomponent oxides (MCOs) have attracted considerable attention due to their wide range of applications. However, the extensive search space of MCO components and the scarcity of MCO crystal structures in existing literature have promoted the use of deep machine learning methods for predicting MCO properties. Despite these advances, accurately predicting the thermal expansion of MCOs across wide temperature and composition ranges remains a complex task. An innovative attention-based deep learning model was introduced in this study. The two proposed self-attention modules greatly improved the model's performance, achieving an 86.88% reduction in root mean square error for predicting thermal expansion coefficients of multicomponent oxides. Additionally, the model demonstrates impressive adaptability and interpretability. Its training results can further aid in comprehending the thermal expansion coefficient variations of multicomponent oxide materials. In summary, judiciously crafted self-attention models overcome tradeoffs between performance and interpretability for materials discovery.

MMV-Net: A Multiple Measurement Vector Network for Multifrequency Electrical Impedance Tomography.

Multifrequency electrical impedance tomography (mfEIT) is an emerging biomedical imaging modality to reveal frequency-dependent conductivity distributions in biomedical applications. Conventional model-based image reconstruction methods suffer from low spatial resolution, unconstrained frequency correlation, and high computational cost. Deep learning has been extensively applied in solving the EIT inverse problem in biomedical and industrial process imaging. However, most existing learning-based approaches deal with the single-frequency setup, which is inefficient and ineffective when extended to the multifrequency setup. This article presents a multiple measurement vector (MMV) model-based learning algorithm named MMV-Net to solve the mfEIT image reconstruction problem. MMV-Net considers the correlations between mfEIT images and unfolds the update steps of the Alternating Direction Method of Multipliers for the MMV problem (MMV-ADMM). The nonlinear shrinkage operator associated with the weighted l2,1 regularization term of MMV-ADMM is generalized in MMV-Net with a cascade of a Spatial Self-Attention module and a Convolutional Long Short-Term Memory (ConvLSTM) module to better capture intrafrequency and interfrequency dependencies. The proposed MMV-Net was validated on our Edinburgh mfEIT Dataset and a series of comprehensive experiments. The results show superior image quality, convergence performance, noise robustness, and computational efficiency against the conventional MMV-ADMM and the state-of-the-art deep learning methods.

Efficient Person Reidentification for IoT-Assisted Cyber–Physical Systems

The main objective of this study is to propose a cyber-physical system (CPS) based person re-identification framework for smart surveillance. The Internet of Things (IoT) based interconnected vision sensors in smart cities are considered essential elements of a CPS, and contribute significantly to urban security. However, the re-identification of targeted persons using emerging edge AI techniques still faces certain challenges. To improve efficiency at the edge and overcome the traditional sensing of video cameras, we employed an AI-based person re-identification framework for CPS that is functional in IoT environments. In addition, we present dual attention dilated network (DADNet), which integrates an energy-efficient convolutional neural network (CNN) with a self-attention module to substantially improve the person matching probability. Further, we applied dual feature fusion to intelligently integrate discriminative and robust features using early and late fusion strategies that allow DADNet to significantly consider the foreground and marginally utilize the background information. Furthermore, we impose diversity orthogonality regularization over several CNN layers, which boosts the performance of DADNet, resulting in an appropriate usage over IoT networks. A comprehensive set of ablation studies, comparison with other state-of-the-art approaches, and a time complexity analysis confirm the strength of our DADNet for re-identification tasks in AI-enabled IoT settings that are well-suited for a CPS.

PMIndoor: Pose Rectified Network and Multiple Loss Functions for Self-Supervised Monocular Indoor Depth Estimation.

Self-supervised monocular depth estimation, which has attained remarkable progress for outdoor scenes in recent years, often faces greater challenges for indoor scenes. These challenges comprise: (i) non-textured regions: indoor scenes often contain large areas of non-textured regions, such as ceilings, walls, floors, etc., which render the widely adopted photometric loss as ambiguous for self-supervised learning; (ii) camera pose: the sensor is mounted on a moving vehicle in outdoor scenes, whereas it is handheld and moves freely in indoor scenes, which results in complex motions that pose challenges for indoor depth estimation. In this paper, we propose a novel self-supervised indoor depth estimation framework-PMIndoor that addresses these two challenges. We use multiple loss functions to constrain the depth estimation for non-textured regions. We introduce a pose rectified network that only estimates the rotation transformation between two adjacent frames of images for the camera pose problem, and improves the pose estimation results with the pose rectified network loss. We also incorporate a multi-head self-attention module in the depth estimation network to enhance the model's accuracy. Extensive experiments are conducted on the benchmark indoor dataset NYU Depth V2, demonstrating that our method achieves excellent performance and is better than previous state-of-the-art methods.

Win-Former: Window-Based Transformer for Maize Plant Point Cloud Semantic Segmentation

Semantic segmentation of plant point clouds is essential for high-throughput phenotyping systems, while existing methods still struggle to balance efficiency and performance. Recently, the Transformer architecture has revolutionized the area of computer vision, and has potential for processing 3D point clouds. Applying the Transformer for semantic segmentation of 3D plant point clouds remains a challenge. To this end, we propose a novel window-based Transformer (Win-Former) network for maize 3D organic segmentation. First, we pre-processed the Pheno4D maize point cloud dataset for training. The maize points were then projected onto a sphere surface, and a window partition mechanism was proposed to construct windows into which points were distributed evenly. After that, we employed local self-attention within windows for computing the relationship of points. To strengthen the windows’ connection, we introduced a Cross-Window self-attention (C-SA) module to gather the cross-window features by moving entire windows along the sphere. The results demonstrate that Win-Former outperforms the famous networks and obtains 83.45% mIoU with the lowest latency of 31 s on maize organ segmentation. We perform extensive experiments on ShapeNet to evaluate stability and robustness, and our proposed model achieves competitive results on part segmentation tasks. Thus, our Win-Former model effectively and efficiently segments the maize point cloud and provides technical support for automated plant phenotyping analysis.

Improving Computation and Memory Efficiency for Real-world Transformer Inference on GPUs

Transformer models have emerged as a leading approach in the field of natural language processing (NLP) and are increasingly being deployed in production environments. Graphic processing units (GPUs) have become a popular choice for the transformer deployment and often rely on the batch processing technique to ensure high hardware performance. Nonetheless, the current practice for transformer inference encounters computational and memory redundancy due to the heavy-tailed distribution of sequence lengths in NLP scenarios, resulting in low practical performance. In this article, we propose a unified solution for improving both computation and memory efficiency of the real-world transformer inference on GPUs. The solution eliminates the redundant computation and memory footprint across a transformer model. At first, a GPU-oriented computation approach is proposed to process the self-attention module in a fine-grained manner, eliminating its redundant computation. Next, the multi-layer perceptron module continues to use the word-accumulation approach to eliminate its redundant computation. Then, to better unify the fine-grained approach and the word-accumulation approach, it organizes the data layout of the self-attention module in block granularity. Since aforementioned approaches make the required memory size largely reduce and constantly fluctuate, we propose the chunk-based approach to enable a better balance between memory footprint and allocation/free efficiency. Our experimental results show that our unified solution achieves a decrease of average latency by 28% on the entire transformer model, 63.8% on the self-attention module, and reduces memory footprint of intermediate results by 7.8×, compared with prevailing frameworks.

A ship-radiated noise classification method based on domain knowledge embedding and attention mechanism

Ship classification based on machine learning (ML) has proven to be a significant underwater acoustic research direction. One of the critical challenges rests with how to embed domain signal knowledge into ML models to obtain suitable features that highly correlate with the classification and create better predictors. In this paper, a novel ML-based ship classification model, Hierarchical Underwater Acoustic Transformer (HUAT), is proposed to improve the classification performance. Firstly, the Detection of Envelope Modulation on Noise (DEMON) spectra of ship-radiated noise signals are estimated by cyclostationary analysis. The motivation for using a DEMON-based preprocessing scheme is that valuable propeller information can be revealed by exploiting the second-order cyclostationarity of ship-radiated noise signals. Secondly, the useful features of DEMON spectra are enhanced using a multi-head self-attention module, and the potential features of the Mel spectrograms are extracted employing a Convolutional Neural Network (CNN) module. The two kinds of features are fused to provide ship classification patterns. The challenge of feature learning in the deep classification model is reduced by leveraging domain-related classification knowledge. Finally, the Swin Transformer, based on shifted window self-attention mechanism, is used to learn high-level feature representations and conduct ship classification. Experimental results show that the HUAT model achieves excellent classification performance on ship-radiated noise datasets, ShipsEar and DeepShip. And its classification efficiency is better than the model based on traditional Transformer architecture. In addition, the proposed method provides technical support for the underwater intelligent system capable of automatically sensing sailing vessels and recognizing vessel types.

Multi-Dimension Compression of Feed-Forward Network in Vision Transformers

Vision Transformers (ViTs) have recently made a splash in computer vision domain and achieved state-of-the-art in many vision tasks. Nevertheless, due to their vast model size and high computational costs, rare transformer-based models are adopted in real-world applications. Since the computational costs of attention operation is the square of the input size, some compression methods for the Multi-Head Self-Attention (MHSA) module have been proposed, reducing its FLOPs successfully but almost without parameters reduction. Meanwhile, the number of parameters and computational costs in the Feed-Forward Network (FFN) module exceeds the MHSA larger, while its compression technologies have not been delved deeper. Consequently, we focus our insight on the compression of FFN layer and present a pruning method named Multi-Dimension Compression of Feed-Forward Network in Vision Transformers(MCF), which greatly reduces the model’s parameters and computational costs. Firstly, we identify the critical elements in the output of the FFN module and then employ them to guide the irregular sparsity of this layer, recognizing insignificant elements of FFN layer that have less impact on the output. Successively, to discard the insignificant elements, we transform the irregular sparsity into regular sparsity and prune them, thus reducing the models’ parameters and getting a substantial speed-up during inference. Extensive results on ImageNet-1K validate the effectiveness of our proposed method, which obtains significant parameters and computational costs reduction with almost unimpaired generalization. For example, we compress DeiT-Tiny with 42% reduction in FLOPs and 33% reduction in parameters, almost without losing accuracy on the ImageNet dataset. Further, we verify the effectiveness of our method in the downstream task, using the pruned DeiT-Small as the backbone for the object detection task on the COCO dataset, gaining revenue without compromising its performance.

Multiple Distilling-based spatial-temporal attention networks for unsupervised human action recognition

Unsupervised action recognition based on spatiotemporal fusion feature extraction has attracted much attention in recent years. However, existing methods still have several limitations: (1) The long-term dependence relationship is not effectively extracted at the time level. (2) The high-order motion relationship between non-adjacent nodes is not effectively captured at the spatial level. (3) The model complexity is too high when the cascade layer input sequence is long, or there are many key points. To solve these problems, a Multiple Distilling-based spatial-temporal attention (MD-STA) networks is proposed in this paper. This model can extract temporal and spatial features respectively and fuse them. Specifically, we first propose a Screening Self-attention (SSA) module; this module can find long-term dependencies in distant frames and high-order motion patterns between non-adjacent nodes in a single frame through a sparse metric on dot product pairs. Then, we propose the Frames and Keypoint-Distilling (FKD) module, which uses extraction operations to halve the input of the cascade layer to eliminate invalid key points and time frame features, thus reducing time and memory complexity. Finally, the Dim-reduction Fusion (DRF) module is proposed to reduce the dimension of existing features to further eliminate redundancy. Numerous experiments were conducted on three distinct datasets: NTU-60, NTU-120, and UWA3D, showing that MD-STA achieves state-of-the-art standards in skeleton-based unsupervised action recognition.

TeaDiseaseNet: multi-scale self-attentive tea disease detection.

Accurate detection of tea diseases is essential for optimizing tea yield and quality, improving production, and minimizing economic losses. In this paper, we introduce TeaDiseaseNet, a novel disease detection method designed to address the challenges in tea disease detection, such as variability in disease scales and dense, obscuring disease patterns. TeaDiseaseNet utilizes a multi-scale self-attention mechanism to enhance disease detection performance. Specifically, it incorporates a CNN-based module for extracting features at multiple scales, effectively capturing localized information such as texture and edges. This approach enables a comprehensive representation of tea images. Additionally, a self-attention module captures global dependencies among pixels, facilitating effective interaction between global information and local features. Furthermore, we integrate a channel attention mechanism, which selectively weighs and combines the multi-scale features, eliminating redundant information and enabling precise localization and recognition of tea disease information across diverse scales and complex backgrounds. Extensive comparative experiments and ablation studies validate the effectiveness of the proposed method, demonstrating superior detection results in scenarios characterized by complex backgrounds and varying disease scales. The presented method provides valuable insights for intelligent tea disease diagnosis, with significant potential for improving tea disease management and production.

MSFANet: Multi-Scale Strip Feature Attention Network for Cloud and Cloud Shadow Segmentation

Cloud and cloud shadow segmentation is one of the most critical challenges in remote sensing image processing. Because of susceptibility to factors such as disturbance from terrain features and noise, as well as a poor capacity to generalize, conventional deep learning networks, when directly used to cloud and cloud shade detection and division, have a tendency to lose fine features and spatial data, leading to coarse segmentation of cloud and cloud shadow borders, false detections, and omissions of targets. To address the aforementioned issues, a multi-scale strip feature attention network (MSFANet) is proposed. This approach uses Resnet18 as the backbone for obtaining semantic data at multiple levels. It incorporates a particular attention module that we name the deep-layer multi-scale pooling attention module (DMPA), aimed at extracting multi-scale contextual semantic data, deep channel feature information, and deep spatial feature information. Furthermore, a skip connection module named the boundary detail feature perception module (BDFP) is introduced to promote information interaction and fusion between adjacent layers of the backbone network. This module performs feature exploration on both the height and width dimensions of the characteristic pattern to enhance the recovery of boundary detail intelligence of the detection targets. Finally, during the decoding phase, a self-attention module named the cross-layer self-attention feature fusion module (CSFF) is employed to direct the aggregation of deeplayer semantic feature and shallow detail feature. This approach facilitates the extraction of feature information to the maximum extent while conducting image restoration. The experimental outcomes unequivocally prove the efficacy of our network in effectively addressing complex cloud-covered scenes, showcasing good performance across the cloud and cloud shadow datasets, the HRC_WHU dataset, and the SPARCS dataset. Our model outperforms existing methods in terms of segmentation accuracy, underscoring its paramount importance in the field of cloud recognition research.

Intermittent fault diagnosis for electronics-rich analog circuit systems based on multi-scale enhanced convolution transformer network with novel token fusion strategy

Intermittent faults of analog circuits in the core area of electronic systems may induce false alarms of built-in test (BIT) system and even cause catastrophic accidents. Due to notable non-repeatability and high randomness, intermittent faults are arduous to be detected. In order to enhance the safety and reliability of electronic systems, an end-to-end diagnostic framework based on multi-scale enhanced convolution Transformer network (MSECTN) with novel token mixing and fusion strategy has been developed. Intermittent fault is a kind of random fault from global perspective, and there is also abundant local information inherent in the fault interval. Wholly composed of self-attention mechanisms, Transformer network possesses prominent capability of representing global features and modelling distant temporal associations, thus can be applied to identify intermittent faults. Meanwhile, introduce the local perception of convolution operation into Transformer network. One convolution embedding and convolutional multi-head self-attention module are designed to capture more abundant local details. Since the information at single scale cannot sufficiently reflect the circuit status, multi-scale convolution embedding with kernels in different sizes is performed to enrich the feature representation. Moreover, one token mixing strategy incorporating Fourier transformation, pooling operation, and depth-wise convolution is proposed to simultaneously model the frequency correlation and enhance the information interaction among spatially adjacent tokens. In order to excavate more inherent and distinguishing features, a novel token fusion strategy is designed to eliminate the effects of redundant feature information. Experimental results on two typical analog circuits demonstrated that the proposed approach can effectively identify different intermittent faults.

NIEG-PNet: noise intensity estimation guided progressive network for hyperspectral image denoising

ABSTRACT How to remove various kinds of noises of hyperspectral images (HSIs) simultaneously is an important issue that we face, which will directly affect the subsequent application of HSIs. Many HSI denoising techniques have been proposed for this purpose and deep convolutional neural network (CNN) is the most effective approach in recent years. However, existing HSI denoising methods are usually unsatisfactory in removing mixed noises and preserving more details. In this paper, we propose a novel noise intensity estimation guided progressive network (NIEG-PNet) to address these problems. To be specific, we design three core modules to improve the HSI denoising accuracy. First, we propose a noise intensity estimation module (NIEM) to capture multiscale noises and structural characteristics of HSI, which can also be a noise prior to guide the network learning. Second, we propose a recurrent feedback grouped denoising module (RFGDM) to fully capture the spectral correlation. Moreover, we design a dual self-attention module including a position self-attention module and a channel self-attention module to exploit the local and global spatial and spectral correlations and alleviate the problem of small available training HSIs. Last, we propose a global spatial-spectral consistency module (GSSCM), which designs a new parallel structure to combine the two-dimensional and the three-dimensional convolutions more effectively. Moreover, it can explore the relationship between the spectral and the horizontal or vertical direction of the spaces. The experimental results on both synthetic and real-data experiments show the superiority of the proposed NIEG-PNet compared to other traditional and advanced HSI denoising methods.

Two-Stream Spatial–Temporal Transformer Networks For Driver Drowsiness Detection

<p>For driver drowsiness detection in the real world, the existing methods have good performance in general. However, when the face is blocked, the light is dim, and the driver’s head posture changes, the performance will deteriorate significantly. In this paper, a two-stream Spatial-Temporal transformer network intended to perform diver drowsiness detection task is proposed to solve the above problems. The spatial-temporal graph is extracted from the video and then the results are obtained from 2s-STTN. The model is a two-stream transformer network model. In our model, the Spatial Self-Attention module is used to learn the embedding of different facial landmarks, and the Temporal Self-Attention module is used to learn the correlation between the frames of facial feature points. Different activated facial landmarks are separated and recognized by class activation mapping technology. Each flow recognizes different activated facial features, extracts spatial or temporal features, and integrates the information about facial features, so as to improve the performance of the system. 2s-STTN can not only mine the long-term dependence of driver behavior from video, but also mine the driver drowsiness information provided by the unobstructed facial signs when the face is blocked. By conducting experiments and comparing our model with other models, it is demonstrated that the proposed model has good performance in driver drowsiness detection.</p> <p> </p>

Complex Transformer Network for Single-Angle Plane-Wave Imaging.

Plane-wave imaging (PWI) is a high-frame-rate imaging technique that sacrifices image quality. Deep learning can potentially enhance plane-wave image quality, but processing complex in-phase and quadrature (IQ) data and suppressing incoherent signals pose challenges. To address these challenges, we present a complex transformer network (CTN) that integrates complex convolution and complex self-attention (CSA) modules. The CTN operates in a four-step process: delaying complex IQ data from a 0° single-angle plane wave for each pixel as CTN input data; extracting reconstruction features with a complex convolution layer; suppressing irrelevant features derived from incoherent signals with two CSA modules; and forming output images with another complex convolution layer. The training labels are generated by minimum variance (MV). Simulation, phantom and in vivo experiments revealed that CTN produced comparable- or even higher-quality images than MV, but with much shorter computation time. Evaluation metrics included contrast ratio, contrast-to-noise ratio, generalized contrast-to-noise ratio and lateral and axial full width at half-maximum and were -11.59 dB, 1.16, 0.68, 278 μm and 329 μm for simulation, respectively, and 9.87 dB, 0.96, 0.62, 357 μm and 305 μm for the phantom experiment, respectively. In vivo experiments further indicated that CTN could significantly improve details that were previously vague or even invisible in DAS and MV images. And after being accelerated by GPU, the CTN runtime (76.03 ms) was comparable to that of delay-and-sum (DAS, 61.24 ms). The proposed CTN significantly improved the image contrast, resolution and some unclear details by the MV beamformer, making it an efficient tool for high-frame-rate imaging.

Real-World Non-Homogeneous Haze Removal by Sliding Self-Attention Wavelet Network

In complex natural haze scenes, image haze removal still faces significant challenges in removing non-homogeneous and dense haze. The double complexity of haze distribution, on the one hand, is reflected in the interference of haze to the global image information, and on the other hand, it is reflected in the imbalance of image brightness and color caused by random haze distribution. In natural scenes with prominent edge and texture features, the above problems may cause severe degradation of image quality and performance of various tasks. Numerous studies on network learning show that the effect of haze removal is closely related to haze feature expression. Therefore, to improve the performance of dehazing, this paper proposes a sliding self-attention wavelet network. Specifically, we first design a sliding self-attention module to identify haze regions in images and capture rich haze-related feature information. Then, considering the uneven distribution of haze in images, discrete wavelet transform (DWT) and inverse transform (IDWT) are used for constructing a hierarchical encoder-decoder structure, which can fully use the multi-resolution characteristics of DWT, locally decompose feature maps of different scales, extract low and high-frequency information, and then gradually recover sharp edges and precise texture details from hazy images. Finally, to enable the proposed network to generate more realistic haze-free images on different complex haze scenes, we develop a DWT-based adversarial loss function to constrain the low and high-frequency components of generated images closer to the corresponding clear images. Experimental results on the relevant public benchmark datasets show that the proposed algorithm achieves favorable dehazing performance.