Long-range Contextual Information Research Articles

STC-UNet: renal tumor segmentation based on enhanced feature extraction at different network levels

Renal tumors are one of the common diseases of urology, and precise segmentation of these tumors plays a crucial role in aiding physicians to improve diagnostic accuracy and treatment effectiveness. Nevertheless, inherent challenges associated with renal tumors, such as indistinct boundaries, morphological variations, and uncertainties in size and location, segmenting renal tumors accurately remains a significant challenge in the field of medical image segmentation. With the development of deep learning, substantial achievements have been made in the domain of medical image segmentation. However, existing models lack specificity in extracting features of renal tumors across different network hierarchies, which results in insufficient extraction of renal tumor features and subsequently affects the accuracy of renal tumor segmentation. To address this issue, we propose the Selective Kernel, Vision Transformer, and Coordinate Attention Enhanced U-Net (STC-UNet). This model aims to enhance feature extraction, adapting to the distinctive characteristics of renal tumors across various network levels. Specifically, the Selective Kernel modules are introduced in the shallow layers of the U-Net, where detailed features are more abundant. By selectively employing convolutional kernels of different scales, the model enhances its capability to extract detailed features of renal tumors across multiple scales. Subsequently, in the deeper layers of the network, where feature maps are smaller yet contain rich semantic information, the Vision Transformer modules are integrated in a non-patch manner. These assist the model in capturing long-range contextual information globally. Their non-patch implementation facilitates the capture of fine-grained features, thereby achieving collaborative enhancement of global–local information and ultimately strengthening the model’s extraction of semantic features of renal tumors. Finally, in the decoder segment, the Coordinate Attention modules embedding positional information are proposed aiming to enhance the model’s feature recovery and tumor region localization capabilities. Our model is validated on the KiTS19 dataset, and experimental results indicate that compared to the baseline model, STC-UNet shows improvements of 1.60%, 2.02%, 2.27%, 1.18%, 1.52%, and 1.35% in IoU, Dice, Accuracy, Precision, Recall, and F1-score, respectively. Furthermore, the experimental results demonstrate that the proposed STC-UNet method surpasses other advanced algorithms in both visual effectiveness and objective evaluation metrics.

A Novel Mamba Architecture with a Semantic Transformer for Efficient Real-Time Remote Sensing Semantic Segmentation

Real-time remote sensing segmentation technology is crucial for unmanned aerial vehicles (UAVs) in battlefield surveillance, land characterization observation, earthquake disaster assessment, etc., and can significantly enhance the application value of UAVs in military and civilian fields. To realize this potential, it is essential to develop real-time semantic segmentation methods that can be applied to resource-limited platforms, such as edge devices. The majority of mainstream real-time semantic segmentation methods rely on convolutional neural networks (CNNs) and transformers. However, CNNs cannot effectively capture long-range dependencies, while transformers have high computational complexity. This paper proposes a novel remote sensing Mamba architecture for real-time segmentation tasks in remote sensing, named RTMamba. Specifically, the backbone utilizes a Visual State-Space (VSS) block to extract deep features and maintains linear computational complexity, thereby capturing long-range contextual information. Additionally, a novel Inverted Triangle Pyramid Pooling (ITP) module is incorporated into the decoder. The ITP module can effectively filter redundant feature information and enhance the perception of objects and their boundaries in remote sensing images. Extensive experiments were conducted on three challenging aerial remote sensing segmentation benchmarks, including Vaihingen, Potsdam, and LoveDA. The results show that RTMamba achieves competitive performance advantages in terms of segmentation accuracy and inference speed compared to state-of-the-art CNN and transformer methods. To further validate the deployment potential of the model on embedded devices with limited resources, such as UAVs, we conducted tests on the Jetson AGX Orin edge device. The experimental results demonstrate that RTMamba achieves impressive real-time segmentation performance.

Learning Motion and Temporal Cues for Unsupervised Video Object Segmentation.

In this article, we address the challenges in unsupervised video object segmentation (UVOS) by proposing an efficient algorithm, termed MTNet, which concurrently exploits motion and temporal cues. Unlike previous methods that focus solely on integrating appearance with motion or on modeling temporal relations, our method combines both aspects by integrating them within a unified framework. MTNet is devised by effectively merging appearance and motion features during the feature extraction process within encoders, promoting a more complementary representation. To capture the intricate long-range contextual dynamics and information embedded within videos, a temporal transformer module is introduced, facilitating efficacious interframe interactions throughout a video clip. Furthermore, we employ a cascade of decoders all feature levels across all feature levels to optimally exploit the derived features, aiming to generate increasingly precise segmentation masks. As a result, MTNet provides a strong and compact framework that explores both temporal and cross-modality knowledge to robustly localize and track the primary object accurately in various challenging scenarios efficiently. Extensive experiments across diverse benchmarks conclusively show that our method not only attains state-of-the-art performance in UVOS but also delivers competitive results in video salient object detection (VSOD). These findings highlight the method's robust versatility and its adeptness in adapting to a range of segmentation tasks. The source code is available at https://github.com/hy0523/MTNet.

3D hand pose and mesh estimation via a generic Topology-aware Transformer model.

In Human-Robot Interaction (HRI), accurate 3D hand pose and mesh estimation hold critical importance. However, inferring reasonable and accurate poses in severe self-occlusion and high self-similarity remains an inherent challenge. In order to alleviate the ambiguity caused by invisible and similar joints during HRI, we propose a new Topology-aware Transformer network named HandGCNFormer with depth image as input, incorporating prior knowledge of hand kinematic topology into the network while modeling long-range contextual information. Specifically, we propose a novel Graphformer decoder with an additional Node-offset Graph Convolutional layer (NoffGConv). The Graphformer decoder optimizes the synergy between the Transformer and GCN, capturing long-range dependencies and local topological connections between joints. On top of that, we replace the standard MLP prediction head with a novel Topology-aware head to better exploit local topological constraints for more reasonable and accurate poses. Our method achieves state-of-the-art 3D hand pose estimation performance on four challenging datasets, including Hands2017, NYU, ICVL, and MSRA. To further demonstrate the effectiveness and scalability of our proposed Graphformer Decoder and Topology aware head, we extend our framework to HandGCNFormer-Mesh for the 3D hand mesh estimation task. The extended framework efficiently integrates a shape regressor with the original Graphformer Decoder and Topology aware head, producing Mano parameters. The results on the HO-3D dataset, which contains various and challenging occlusions, show that our HandGCNFormer-Mesh achieves competitive results compared to previous state-of-the-art 3D hand mesh estimation methods.

A novel transformer autoencoder for multi-modal emotion recognition with incomplete data

Multi-modal signals have become essential data for emotion recognition since they can represent emotions more comprehensively. However, in real-world environments, it is often impossible to acquire complete data on multi-modal signals, and the problem of missing modalities causes severe performance degradation in emotion recognition. Therefore, this paper represents the first attempt to use a transformer-based architecture, aiming to fill the modality-incomplete data from partially observed data for multi-modal emotion recognition (MER). Concretely, this paper proposes a novel unified model called transformer autoencoder (TAE), comprising a modality-specific hybrid transformer encoder, an inter-modality transformer encoder, and a convolutional decoder. The modality-specific hybrid transformer encoder bridges a convolutional encoder and a transformer encoder, allowing the encoder to learn local and global context information within each particular modality. The inter-modality transformer encoder builds and aligns global cross-modal correlations and models long-range contextual information with different modalities. The convolutional decoder decodes the encoding features to produce more precise recognition. Besides, a regularization term is introduced into the convolutional decoder to force the decoder to fully leverage the complete and incomplete data for emotional recognition of missing data. 96.33%, 95.64%, and 92.69% accuracies are attained on the available data of the DEAP and SEED-IV datasets, and 93.25%, 92.23%, and 81.76% accuracies are obtained on the missing data. Particularly, the model acquires a 5.61% advantage with 70% missing data, demonstrating that the model outperforms some state-of-the-art approaches in incomplete multi-modal learning.

A Novel Local-Global Graph Convolutional Method for Point Cloud Semantic Segmentation.

Although convolutional neural networks (CNNs) have shown good performance on grid data, they are limited in the semantic segmentation of irregular point clouds. This article proposes a novel and effective graph CNN framework, referred to as the local-global graph convolutional method (LGGCM), which can achieve short- and long-range dependencies on point clouds. The key to this framework is the design of local spatial attention convolution (LSA-Conv). The design includes two parts: generating a weighted adjacency matrix of the local graph composed of neighborhood points, and updating and aggregating the features of nodes to obtain the spatial geometric features of the local point cloud. In addition, a smooth module for central points is incorporated into the process of LSA-Conv to enhance the robustness of the convolution against noise interference by adjusting the position coordinates of the points adaptively. The learned robust LSA-Conv features are then fed into a global spatial attention module with the gated unit to extract long-range contextual information and dynamically adjust the weights of features from different stages. The proposed framework, consisting of both encoding and decoding branches, is an end-to-end trainable network for semantic segmentation of 3-D point clouds. The theoretical analysis of the approximation capabilities of LSA-Conv is discussed to determine whether the features of the point cloud can be accurately represented. Experimental results on challenging benchmarks of the 3-D point cloud demonstrate that the proposed framework achieves excellent performance.

Spatial Prior-Guided Bi-Directional Cross-Attention Transformers for Tooth Instance Segmentation.

Tooth instance segmentation of dental panoramic X-ray images represents a task of significant clinical importance. Teeth demonstrate symmetry within the upper and lower jawbones and are arranged in a specific order. However, previous studies frequently overlook this crucial spatial prior information, resulting in misidentifications of tooth categories for adjacent or similarly shaped teeth. In this paper, we propose SPGTNet, a spatial prior-guided transformer method, designed to both the extracted tooth positional features from CNNs and the long-range contextual information from vision transformers for dental panoramic X-ray image segmentation. Initially, a center-based spatial prior perception module is employed to identify each tooth's centroid, thereby enhancing the spatial prior information for the CNN sequence features. Subsequently, a bi-directional cross-attention module is designed to facilitate the interaction between the spatial prior information of the CNN sequence features and the long-distance contextual features of the vision transformer sequence features. Finally, an instance identification head is employed to derive the tooth segmentation results. Extensive experiments on three public benchmark datasets have demonstrated the effectiveness and superiority of our proposed method in comparison with other state-of-the-art approaches. The proposed method demonstrates the capability to accurately identify and analyze tooth structures, thereby providing crucial information for dental diagnosis, treatment planning, and research.

GA2MIF: Graph and Attention Based Two-Stage Multi-Source Information Fusion for Conversational Emotion Detection

Multimodal Emotion Recognition in Conversation (ERC) plays an influential role in the field of human-computer interaction and conversational robotics since it can motivate machines to provide empathetic services. Multimodal data modeling is an up-and-coming research area in recent years, which is inspired by human capability to integrate multiple senses. Several graph-based approaches claim to capture interactive information between modalities, but the heterogeneity of multimodal data makes these methods prohibit optimal solutions. In this work, we introduce a multimodal fusion approach named Graph and Attention based Two-stage Multi-source Information Fusion (GA2MIF) for emotion detection in conversation. Our proposed method circumvents the problem of taking heterogeneous graph as input to the model while eliminating complex redundant connections in the construction of graph. GA2MIF focuses on contextual modeling and cross-modal modeling through leveraging Multi-head Directed Graph ATtention networks (MDGATs) and Multi-head Pairwise Cross-modal ATtention networks (MPCATs), respectively. Extensive experiments on two public datasets (i.e., IEMOCAP and MELD) demonstrate that the proposed GA2MIF has the capacity to validly capture intra-modal long-range contextual information and inter-modal complementary information, as well as outperforms the prevalent State-Of-The-Art (SOTA) models by a remarkable margin.

Attention-Aware Non-Rigid Image Registration for Accelerated MR Imaging.

Accurate motion estimation at high acceleration factors enables rapid motion-compensated reconstruction in Magnetic Resonance Imaging (MRI) without compromising the diagnostic image quality. In this work, we introduce an attention-aware deep learning-based framework that can perform non-rigid pairwise registration for fully sampled and accelerated MRI. We extract local visual representations to build similarity maps between the registered image pairs at multiple resolution levels and additionally leverage long-range contextual information using a transformer-based module to alleviate ambiguities in the presence of artifacts caused by undersampling. We combine local and global dependencies to perform simultaneous coarse and fine motion estimation. The proposed method was evaluated on in-house acquired fully sampled and accelerated data of 101 patients and 62 healthy subjects undergoing cardiac and thoracic MRI. The impact of motion estimation accuracy on the downstream task of motion-compensated reconstruction was analyzed. We demonstrate that our model derives reliable and consistent motion fields across different sampling trajectories (Cartesian and radial) and acceleration factors of up to 16x for cardiac motion and 30x for respiratory motion and achieves superior image quality in motion-compensated reconstruction qualitatively and quantitatively compared to conventional and recent deep learning-based approaches. The code is publicly available at https://github.com/lab-midas/GMARAFT.

SiamMAN: Siamese Multi-Phase Aware Network for Real-Time Unmanned Aerial Vehicle Tracking

In this paper, we address aerial tracking tasks by designing multi-phase aware networks to obtain rich long-range dependencies. For aerial tracking tasks, the existing methods are prone to tracking drift in scenarios with high demand for multi-layer long-range feature dependencies such as viewpoint change caused by the characteristics of the UAV shooting perspective, low resolution, etc. In contrast to the previous works that only used multi-scale feature fusion to obtain contextual information, we designed a new architecture to adapt the characteristics of different levels of features in challenging scenarios to adaptively integrate regional features and the corresponding global dependencies information. Specifically, for the proposed tracker (SiamMAN), we first propose a two-stage aware neck (TAN), where first a cascaded splitting encoder (CSE) is used to obtain the distributed long-range relevance among the sub-branches by the splitting of feature channels, and then a multi-level contextual decoder (MCD) is used to achieve further global dependency fusion. Finally, we design the response map context encoder (RCE) utilizing long-range contextual information in backpropagation to accomplish pixel-level updating for the deeper features and better balance the semantic and spatial information. Several experiments on well-known tracking benchmarks illustrate that the proposed method outperforms SOTA trackers, which results from the effective utilization of the proposed multi-phase aware network for different levels of features.

Multi-task Contexture Learning Network for automated vertebrae segmentation and tumor diagnosis from MRI

Automated vertebrae segmentation and tumor diagnosis from magnetic resonance imaging (MRI) scans play a crucial role in early detection of spinal tumors. However, the task is challenging due to the variability in fields of view across MRI images and the diverse appearances of tumors. To address these challenges, we propose a novel approach, the Multi-task Contexture Learning Network (MCLN), which concurrently performs accurate vertebrae segmentation and precise identification of tumors. Specifically, MCLN first constructs a spatial pyramid graph convolutional network and a label attention network to effectively capture long-range contextual information and generate a precise segmentation mask. Then, MCLN incorporates the input MRI image and the segmentation mask to recognize vertebral tumors utilizing a mixed kernel multi-loss network. We evaluate our proposed MCLN on a challenging dataset consisting of 600 MRI images, achieving impressive results with a mean Dice coefficient of 90.03% for vertebrae segmentation and a mean accuracy of 95.11% for tumor diagnosis. These results demonstrate the remarkable performance of MCLN in addressing automated vertebrae segmentation and tumor diagnosis from MRI.

A Novel Deep Learning Model for Medical Image Segmentation with Convolutional Neural Network and Transformer.

Accurate segmentation of medical images is essential for clinical decision-making, and deep learning techniques have shown remarkable results in this area. However, existing segmentation models that combine transformer and convolutional neural networks often use skip connections in U-shaped networks, which may limit their ability to capture contextual information in medical images. To address this limitation, we propose a coordinated mobile and residual transformer UNet (MRC-TransUNet) that combines the strengths of transformer and UNet architectures. Our approach uses a lightweight MR-ViT to address the semantic gap and a reciprocal attention module to compensate for the potential loss of details. To better explore long-range contextual information, we use skip connections only in the first layer and add MR-ViT and RPA modules in the subsequent downsampling layers. In our study, we evaluated the effectiveness of our proposed method on three different medical image segmentation datasets, namely, breast, brain, and lung. Our proposed method outperformed state-of-the-art methods in terms of various evaluation metrics, including the Dice coefficient and Hausdorff distance. These results demonstrate that our proposed method can significantly improve the accuracy of medical image segmentation and has the potential for clinical applications. Illustration of the proposed MRC-TransUNet. For the input medical images, we first subject them to an intrinsic downsampling operation and then replace the original jump connection structure using MR-ViT. The output feature representations at different scales are fused by the RPA module. Finally, an upsampling operation is performed to fuse the features to restore them to the same resolution as the input image.

A hybrid enhanced attention transformer network for medical ultrasound image segmentation

Ultrasound (US) technology is a widely utilized for clinical screening owing to its cost-effectiveness, painlessness, and convenience. However, the automatic segmentation of lesions or organs in US images remains to be challenging due to speckle artifacts, blurred boundaries, and low contrast. Recently, transformer-based methods have shown to be effective in long-range dependency, which is a good complement to Convolutional Neural Networks (CNN). In this article, we present a novel U-shape segmentation model based on a hybrid CNN-transformer structure that can effectively integrate CNN local features and transformer long-range contextual information of US images. To begin with, we design a coordinate residual block (CdRB) to encode the absolute position information of lesions. Further, we develop a channel enhanced self-attention-based transformer (ECAT) to help enhance the response of extracted global features. Finally, we adopt a comprehensive dual attention module (CDAM) to enhance skip connection features, which can learn feature correlations and capture more accurate edge features in US images. Results based on four public US datasets demonstrate that our method outperforms state-of-the-art segmentation methods, with 0.741 Dice on BUSI for breast lesion segmentation, 0.827 Dice on DDTI for thyroid lesion segmentation, 0.895 Dice on TN3k for thyroid lesion segmentation and 0.940 Dice on CAMUS for left ventricle segmentation. Furthermore, the robustness of our network is further demonstrated by an external validation dataset for breast lesion segmentation. In summary, our method showcases excellent adaptability and robustness in US image segmentation and can potentially be a general US segmentation tool.

FCD-AttResU-Net: An improved forest change detection in Sentinel-2 satellite images using attention residual U-Net

Forest Change Detection (FCD) is a critical component of natural resource monitoring and conservation strategies, enabling informed decision-making. Various methods utilizing the power of artificial intelligence (AI) have been developed for detecting and categorizing changes in forest cover using remote sensing (RS) data. One prominent AI-powered approach is the U-Net, a deep learning (DL) architecture famous for its segmentation proficiency. However, the standard U-Net architecture fails to effectively capture intricate spatial dependencies and long-range contextual information present in remote sensing imagery. To address this research gap, we introduce an attention-residual-based novel DL model which leverages the U-Net architecture and Sentinel-2 satellite images to map alterations in forest vegetation cover in the tropical region. Our novel model enhances the U-Net architecture by seamlessly integrating the strengths of the U-Net, harnessing attention mechanisms strategically to amplify crucial features, and leveraging cutting-edge residual connections to facilitate the smooth flow of information and gradient propagation. These meticulous design choices enabled the precise feature extraction, resulting in improved computational performance of the proposed method compared to the Standard U-Net, Deeplabv3+, Deep Res-U-Net, and Attention U-Net. The classification results demonstrate the enhanced efficiency of our model, achieving a Mean Intersection over Union (MIoU) of 0.9330 on our test dataset. This performance surpasses the Attention U-Net (0.9146), Standard U-Net (0.9029), Deeplabv3+ (0.9247), and Deep Res-U-Net (0.9282). The comparative analysis of ground truth reproductions unveiled the superior detection capabilities of our model in accurately identifying forest and non-forest polygons, surpassing both the standard U-Net, and the U-Net augmented with attention mechanism, along with other state-of-the-art techniques, thereby highlighting its enhanced efficacy. The model’s broad applicability can support forest managers and ecologists in rapidly evaluating the long-term ramifications of infrastructure initiatives, such as roads, on tropical forests, including those in Brunei.

MFAFNet: A Lightweight and Efficient Network with Multi-Level Feature Adaptive Fusion for Real-Time Semantic Segmentation.

Currently, real-time semantic segmentation networks are intensely demanded in resource-constrained practical applications, such as mobile devices, drones and autonomous driving systems. However, most of the current popular approaches have difficulty in obtaining sufficiently large receptive fields, and they sacrifice low-level details to improve inference speed, leading to decreased segmentation accuracy. In this paper, a lightweight and efficient multi-level feature adaptive fusion network (MFAFNet) is proposed to address this problem. Specifically, we design a separable asymmetric reinforcement non-bottleneck module, which designs a parallel structure to extract short- and long-range contextual information and use optimized convolution to increase the inference speed. In addition, we propose a feature adaptive fusion module that effectively balances feature maps with multiple resolutions to reduce the loss of spatial detail information. We evaluate our model with state-of-the-art real-time semantic segmentation methods on the Cityscapes and Camvid datasets. Without any pre-training and post-processing, our MFAFNet has only 1.27 M parameters, while achieving accuracies of 75.9% and 69.9% mean IoU with speeds of 60.1 and 82.6 FPS on the Cityscapes and Camvid test sets, respectively. The experimental results demonstrate that the proposed method achieves an excellent trade-off between inference speed, segmentation accuracy and model size.

Dilated Convolution-based Feature Refinement Network for Crowd Localization

As an emerging computer vision task, crowd localization has received increasing attention due to its ability to produce more accurate spatially predictions. However, continuous scale variations in complex crowd scenes lead to tiny individuals at the edges, so that existing methods cannot achieve precise crowd localization. Aiming at alleviating the above problems, we propose a novel Dilated Convolution-based Feature Refinement Network (DFRNet) to enhance the representation learning capability. Specifically, the DFRNet is built with three branches that can capture the information of each individual in crowd scenes more precisely. More specifically, we introduce a Feature Perception Module to model long-range contextual information at different scales by adopting multiple dilated convolutions, thus providing sufficient feature information to perceive tiny individuals at the edge of images. Afterwards, a Feature Refinement Module is deployed at multiple stages of the three branches to facilitate the mutual refinement of feature information at different scales, thus further improving the expression capability of multi-scale contextual information. By incorporating the above modules, DFRNet can locate individuals in complex scenes more precisely. Extensive experiments on multiple datasets demonstrate that the proposed method has more advanced performance compared to existing methods and can be more accurately adapted to complex crowd scenes.

A Two-Branch Neural Network for Short-Axis PET Image Quality Enhancement.

The axial field of view (FOV) is a key factor that affects the quality of PET images. Due to hardware FOV restrictions, conventional short-axis PET scanners with FOVs of 20 to 35 cm can acquire only low-quality PET (LQ-PET) images in fast scanning times (2-3 minutes). To overcome hardware restrictions and improve PET image quality for better clinical diagnoses, several deep learning-based algorithms have been proposed. However, these approaches use simple convolution layers with residual learning and local attention, which insufficiently extract and fuse long-range contextual information. To this end, we propose a novel two-branch network architecture with swin transformer units and graph convolution operation, namely SW-GCN. The proposed SW-GCN provides additional spatial- and channel-wise flexibility to handle different types of input information flow. Specifically, considering the high computational cost of calculating self-attention weights in full-size PET images, in our designed spatial adaptive branch, we take the self-attention mechanism within each local partition window and introduce global information interactions between nonoverlapping windows by shifting operations to prevent the aforementioned problem. In addition, the convolutional network structure tends to consider the information in each channel equally during the feature extraction process. In our designed channel adaptive branch, we use a Watts Strogatz topology structure to connect each feature map to its most relevant features in each graph convolutional layer, substantially reducing information redundancy. Moreover, ensemble learning is adopted in our SW-GCN for mapping distinct features from the two well-designed branches to the enhanced PET images. We carried out extensive experiments on three single-bed position scans for 386 patients. The test results demonstrate that our proposed SW-GCN approach outperforms state-of-the-art methods in both quantitative and qualitative evaluations.

VLTENet: A Deep-Learning-Based Vertebra Localization and Tilt Estimation Network for Automatic Cobb Angle Estimation.

Scoliosis diagnosis and assessment rely upon Cobb angle estimation from X-ray images of the spine. Recently, automated scoliosis assessment has been greatly improved using deep learning methods. However, in such methods, the Cobb angle is usually predicted based on regression models that don't account for information of the spine structure. Alternatively, the Cobb angle can be estimated indirectly through landmark-detection and vertebra-segmentation, but this approach is still highly sensitive to small detection and segmentation errors. This paper proposes a novel deep-learning architecture, called the vertebra localization and tilt estimation network (VLTENet). This network boosts the Cobb angle estimation accuracy through employing vertebra localization and tilt estimation as network prediction goals. In particular, the VLTENet model innovatively combines a deep high-resolution network (HRNet) and a fully-convolutional U-Net architecture for capturing long-range contextual information, the overall structure, and local details in spinal X-ray images. A feature fusion channel attention (FFCA) module is also proposed to selectively emphasize more informative features and suppress less informative ones. In addition, a joint spine loss function (JS-Loss) is designed to account for the spine shape and other spatial constraints, so that the network focuses more on spine-related regions and ignore irrelevant background regions. Finally, we propose a new Cobb angle estimation method conforms with the clinical Cobb angle calculation guidelines, and produces accurate estimates for different types of scoliosis. Extensive experiments on the publically-available AASCE challenge dataset and on an in-house dataset demonstrated the superiority of our method for the task of automatic assessment of scoliosis.

A Swin Transformer-Based Model for Thyroid Nodule Detection in Ultrasound Images.

In recent years, the incidence of thyroid cancer has been increasing. Thyroid nodule detection is critical for both the detection and treatment of thyroid cancer. Convolutional neural networks (CNNs) have achieved good results in thyroid ultrasound image analysis tasks. However, due to the limited valid receptive field of convolutional layers, CNNs fail to capture long-range contextual dependencies, which are important for identifying thyroid nodules in ultrasound images. Transformer networks are effective in capturing long-range contextual information. Inspired by this, we propose a novel thyroid nodule detection method that combines the Swin Transformer backbone and Faster R-CNN. Specifically, an ultrasound image is first projected into a 1D sequence of embeddings, which are then fed into a hierarchical Swin Transformer. The Swin Transformer backbone extracts features at five different scales by utilizing shifted windows for the computation of self-attention. Subsequently, a feature pyramid network (FPN) is used to fuse the features from different scales. Finally, a detection head is used to predict bounding boxes and the corresponding confidence scores. Data collected from 2,680 patients were used to conduct the experiments, and the results showed that this method achieved the best mAP score of 44.8%, outperforming CNN-based baselines. In addition, we gained better sensitivity (90.5%) than the competitors. This indicates that context modeling in this model is effective for thyroid nodule detection.

Deep convolutional neural network for hippocampus segmentation with boundary region refinement.

Accurately segmenting the hippocampus from magnetic resonance (MR) brain images is a crucial step in studying brain disorders. However, this task is challenging due to the low signal contrast of hippocampal images, the irregular shape, and small structural size of the hippocampi. In recent years, several deep convolutional networks have been proposed for hippocampus segmentation, which have achieved state-of-the-art performance. These methods typically use large image patches for training the network, as larger patches are beneficial for capturing long-range contextual information. However, this approach increases the computational burden and overlooks the significance of the boundary region. In this study, we propose a deep learning-based method for hippocampus segmentation with boundary region refinement. Our method involves two main steps. First, we propose a convolutional network that takes large image patches as input for initial segmentation. Then, we extract small image patches around the hippocampal boundary for training the second convolutional neural network, which refines the segmentation in the boundary regions. We validate our proposed method on a publicly available dataset and demonstrate that it significantly improves the performance of convolutional neural networks that use single-size image patches as input. In conclusion, our study proposes a novel method for hippocampus segmentation, which improves upon the current state-of-the-art methods. By incorporating a boundary refinement step, our approach achieves higher accuracy in hippocampus segmentation and may facilitate research on brain disorders.