Label Map Research Articles

LmGa: Combining label mapping method with graph attention network for agricultural recognition

LmGa: Combining label mapping method with graph attention network for agricultural recognition

PTE: Prompt tuning with ensemble verbalizers

Prompt tuning has achieved remarkable success in facilitating the performance of Pre-trained Language Models (PLMs) across various downstream NLP tasks, particularly in scenarios with limited downstream data. Reframing tasks as fill-in-the-blank questions represents an effective approach within prompt tuning. However, this approach necessitates the mapping of labels through a verbalizer consisting of one or more label tokens, constrained by manually crafted prompts. Furthermore, most existing automatic crafting methods either introduce external resources or rely solely on discrete or continuous optimization strategies. To address this issue, we have proposed a methodology for optimizing discrete verbalizers based on gradient descent, which we refer to this approach as PTE. This method integrates discrete tokens into verbalizers that can be continuously optimized, combining the distinct advantages of both discrete and continuous optimization strategies. In contrast to prior approaches, ours eschews reliance on prompts generated by other models or prior knowledge, merely augmenting a matrix. This approach boasts remarkable simplicity and flexibility, enabling prompt optimization while preserving the interpretability of output label tokens without constraints imposed by discrete vocabularies. Finally, employing this method in text classification tasks, we observe that PTE achieves results comparable to, if not surpassing, previous methods even under extreme conciseness. This furnishes a simple, intuitive, and efficient solution for automatically constructing verbalizers. Moreover, through quantitative analysis of optimized verbalizers, we uncover that language models likely rely not only on semantic information but also on other features for text classification. This revelation unveils new avenues for future research and model enhancements.

Relationship between left atrial isolated surface area and early-term recurrence in patients with persistent atrial fibrillation after cryoballoon ablation

ObjectiveTo investigate the effect of pulmonary vein antrum enlargement combined with left atrial roof cryoballoon ablation in patients with persistent atrial fibrillation (PeAF) by analyzing the relationship between left atrial isolation area surface area (ISA) and early postoperative recurrence.Methods93 patients with PeAF were classified into recurrence and non-recurrence groups according to the results of the 1-year follow-up. Three-dimensional electroanatomical labeling map was constructed and merged with that of the left atrial pulmonary vein CTA, and the ISA and the left atrial surface area (LASA) were measured and analyzed to determine the relationship between ISA/LASA in relation to early postoperative recurrence.Results93 patients were included and followed up for 1 year with AF-free recurrence rate of 75.3%. The ISA of the recurrence group was lower than that of the non-recurrence group. Left atrial internal diameter (LAD), left common pulmonary vein, the ISA, the ISA/LASA and early-term recurrence had statistical significance in both groups. The factors that significantly predicted early-term recurrence were left common pulmonary vein and the ISA/LASA. ISA/LASA (HR 0, 95% CI 0–0.005, P = 0.008) and left common pulmonary vein trunk (HR 7.754, 95% CI 2.256–25.651, P = 0.001) were the independent risk factors for early recurrence. ROC curve analysis showed that ISA/LASA predicted the best early recurrence after operation with a cut-off value of 15.2%.ConclusionA greater ISA/LASA reduces early recurrence after cryoablation in patients with PeAF. An ISA/LASA of 15.2% may be the best cut-off value for predicting early recurrence after cryoablation for PeAF.

High-resolution segmentations of the hypothalamus and its subregions for training of segmentation models

Segmentation of brain structures on magnetic resonance imaging (MRI) is a highly relevant neuroimaging topic, as it is a prerequisite for different analyses such as volumetry or shape analysis. Automated segmentation facilitates the study of brain structures in larger cohorts when compared with manual segmentation, which is time-consuming. However, the development of most automated methods relies on large and manually annotated datasets, which limits the generalizability of these methods. Recently, new techniques using synthetic images have emerged, reducing the need for manual annotation. Here we provide a dataset composed of label maps built from publicly available ultra-high resolution ex vivo MRI from 10 whole hemispheres, which can be used to develop segmentation methods using synthetic data. The label maps are obtained with a combination of manual labels for the hypothalamic regions and automated segmentations for the rest of the brain, and mirrored to simulate entire brains. We also provide the pre-processed ex vivo scans, as this dataset can support future projects to include other structures after these are manually segmented.

Boundary points guided 3D object detection for point clouds

3D object detection in LiDAR point clouds is crucial for computer vision tasks such as autonomous driving. The two-stage approach with point cloud completion achieves remarkable performance by generating semantic surface points for foreground objects or learning bird’s eye view shape heat map labels. However, these methods require additional completion datasets, leading to substantial computation and memory demands. In this context, we propose a Boundary Points Guided 3D (BPG3D) object detection method that complements point cloud boundary information without the need for additional data. Specifically, we generate Region of Interest (RoI) boundary points to aggregate the neighbor voxel information at the RoI boundary during the refinement stage to complement the missing boundary information. Meanwhile, we design a Dual Feature Selection (DFS) module to adaptively fuse RoI grid point features and RoI boundary point features for bounding box refinement with negligible computational cost. Additionally, inspired by tensor decomposition theory, we use low-rank tensors to reconstruct high-rank tensors in the point cloud feature encoder to enhance contextual semantic information. The proposed method achieves 65.81% mAP on KITTI Test Set, obtaining a good trade-off between accuracy and efficiency.

Kidney Tumor Segmentation Based on DWR-SegFormer

Kidney cancer is a malignant tumor with a high mortality rate. The accurate segmentation of tumors from computed tomography (CT) scans can assist physicians in clinical diagnosis. We introduced a new segmentation network called DWR-SegFormer to address the challenge of accurately segmenting kidney tumors in CT images. The method involved binarizing the label maps of clear cell renal cell carcinoma and papillary renal cell carcinoma CT images for identification, and the cancer lesion area was obtained by the label so that the model could accurately identify the area and enhance the feature extraction ability. Secondly, an optimized segmentation model combining a DWR attention mechanism and SegFormer network was constructed. MiT-B0 was used as the encoder of the model to establish long-distance feature dependencies and effectively extract feature information at different resolutions. The decoder with a multi-branch DWR module was implemented to utilize multi-scale feature information effectively and enhance segmentation accuracy. Comparing the experimental results with other existing models shows that the model significantly outperformed the comparison methods in all evaluation metrics on the CT image dataset of clear cell renal cancer. Furthermore, the experimental findings highlight the robustness of the proposed model across other datasets.

Transformer models for Land Cover Classification with Satellite Image Time Series

In this paper we address the task of pixel-wise land cover (LC) classification using satellite image time series (SITS). For that purpose, we use a supervised deep learning model and focus on combining spatial and temporal features. Our method is based on the Swin Transformer and captures global temporal features by using self-attention and local spatial features by convolutions. We extend the architecture to receive multi-temporal input to generate one output label map for every input image. In our experiments we focus on the application of pixel-wise LC classification from Sentinel‑2 SITS over the whole area of Lower Saxony (Germany). The experiments with our new model show that by using convolutions for spatial feature extraction or a temporal weighting module in the skip connections the performance improves and is more stable. The combined usage of both adaptations results in the overall best performance although this improvement is only minimal. Compared to a fully convolutional neural network without any self-attention layers our model improves the results by 2.1% in the mean F1-Score on a corrected test dataset. Additionally, we investigate different types of temporal position encoding, which do not have a significant impact on the performance.

Exploring graph learning techniques for enhancing cross-domain applications in computer vision and natural language processing

With the rapid development of artificial intelligence, graph learning technology has become a research hotspot because of its unique data processing ability. This paper discusses the application of graph learning technology in computer vision (CV) and natural language processing (NLP), especially, it is applied to image segmentation and recognition, visual relationship detection, dynamic scene understanding, semantic role labeling and knowledge map enhancement. The graph neural network (GNN) can effectively learn the deep features of image or text data by defining pixels or hyperpixels as nodes and constructing the edges between nodes according to the spatial proximity or similarity between pixels. This paper also discusses the application of graph learning technology in improving model interpretability, including feature relation visualization, error analysis and diagnosis, and model decision path interpretation. By comparing the application effects of different graph learning models and algorithms, this paper aims to provide reference and inspiration for the follow-up research and application.

Evaluation of Digital Nautical Chart data for confirmation and expansion of GeoNames data

ABSTRACT This work examines how Digital Nautical Chart (DNC) data may contribute to the evolution and refinement of GeoNames data for near-shore features. GeoNames features are point data with one or more possible place names. DNC Earth Cover Text (ECRText) objects are map labels positioned nearby their real word counterpart. ECRText feature map position strikes a compromise between association with real features and cartographic readability. This work explores whether ECRText features can confirm (or expand names for) existing locations or contribute new locations through data conflation. Due to name variations and spatial position, conflating these data are nontrivial. Previous work engaged in a brief examination using the trigram string matching algorithm under coarse proximity constraints, indicating that ECRText could provide additional value to GeoNames. This work builds on that study, by engaging in a deeper examination of spatial proximity and exploring conflation agreement across an ensemble of string matching approaches. The result finds strong ensemble agreement about ECRText features which already exist in GeoNames but mixed results about which features contribute new information, as well as exploring why some of these matching techniques fail. With an eye toward automation, computational efficiency was found not to be a constraint in sustaining updates.

Anatomy-aware and acquisition-agnostic joint registration with SynthMorph.

Affine image registration is a cornerstone of medical-image analysis. While classical algorithms can achieve excellent accuracy, they solve a time-consuming optimization for every image pair. Deep-learning (DL) methods learn a function that maps an image pair to an output transform. Evaluating the function is fast, but capturing large transforms can be challenging, and networks tend to struggle if a test-image characteristic shifts from the training domain, such as the resolution. Most affine methods are agnostic to the anatomy the user wishes to align, meaning the registration will be inaccurate if algorithms consider all structures in the image. We address these shortcomings with SynthMorph, a fast, symmetric, diffeomorphic, and easy-to-use DL tool for joint affine-deformable registration of any brain image without preprocessing. First, we leverage a strategy that trains networks with widely varying images synthesized from label maps, yielding robust performance across acquisition specifics unseen at training. Second, we optimize the spatial overlap of select anatomical labels. This enables networks to distinguish anatomy of interest from irrelevant structures, removing the need for preprocessing that excludes content which would impinge on anatomy-specific registration. Third, we combine the affine model with a deformable hypernetwork that lets users choose the optimal deformation-field regularity for their specific data, at registration time, in a fraction of the time required by classical methods. This framework is applicable to learning anatomy-aware, acquisition-agnostic registration of any anatomy with any architecture, as long as label maps are available for training. We analyze how competing architectures learn affine transforms and compare state-of-the-art registration tools across an extremely diverse set of neuroimaging data, aiming to truly capture the behavior of methods in the real world. SynthMorph demonstrates high accuracy and is available at https://w3id.org/synthmorph, as a single complete end-to-end solution for registration of brain magnetic resonance imaging (MRI) data.

DeepWaterFraction: A globally applicable, self-training deep learning approach for percent surface water area estimation from Landsat mission imagery

Surface water area estimation is essential for understanding global environmental dynamics, yet it presents significant challenges, particularly when dealing with small water bodies like ponds and narrow width rivers. Surface water areas for these small bodies are often inaccurately represented by existing methods due to the spatial resolution limitations in commonly used remote sensing images. This study introduces DeepWaterFraction (DWF), a deep learning approach, to estimate percent surface water area from Landsat mission imagery. DWF is trained with a self-training method, which creates training data by upscaling remote sensing images and water map labels to a lower resolution, enabling the creation of a large-scale, global coverage training dataset. DWF demonstrates superior accuracy in estimating areas for small water bodies compared to several existing methods for surface water area estimation, with a pixel-wise root mean squared error of 14.3 %. Specifically, it reduces error rates by 54.3 % for water bodies with a minimum area of 0.001 km2 and by 22.6 % for those with a minimum area of 0.01 km2. DWF’s application in global river discharge inversion is also explored, showcasing its capability to capture width variations in narrow rivers (<90 m) better than existing methods, and its robustness across environments including wetland, tree covers, and urban areas. Even for wider rivers (>150 m), DWF’s performance remains superior, as its ability to accurately quantify mixed water pixel areas effectively reflects discharge variations when the variation area is small. We find that self-training is an effective strategy for generating extensive global training datasets for water mapping, with a high upscaling factor being critical for ensuring label accuracy. This study presents a step forward in the accurate global mapping of water resources.

Occlusion-aware deep convolutional neural network via homogeneous Tanh-transforms for face parsing

Face parsing infers a pixel-wise label map for each semantic facial component. Previous methods generally work well for uncovered faces, however, they overlook facial occlusion and ignore some contextual areas outside a single face, especially when facial occlusion has become a common situation during the COVID-19 epidemic. Inspired by the lighting phenomena in everyday life, where illumination from four distinct lamps provides a more uniform distribution than a single central light source, we propose a novel homogeneous tanh-transform for image preprocessing, which is made up of four tanh-transforms. These transforms fuse the central vision and the peripheral vision together. Our proposed method addresses the dilemma of face parsing under occlusion and compresses more information from the surrounding context. Based on homogeneous tanh-transforms, we propose an occlusion-aware convolutional neural network for occluded face parsing. It combines information in both Tanh-polar space and Tanh-Cartesian space, capable of enhancing receptive fields. Furthermore, we introduce an occlusion-aware loss to focus on the boundaries of occluded regions. The network is simple, flexible, and can be trained end-to-end. To facilitate future research of occluded face parsing, we also contribute a new cleaned face parsing dataset. This dataset is manually purified from several academic or industrial datasets, including CelebAMask-HQ, Short-video Face Parsing, and the Helen dataset, and will be made public. Experiments demonstrate that our method surpasses state-of-the-art methods in face parsing under occlusion.

Stereo sample generation‐based domain generalization network for stereo matching

AbstractRecently, deep learning‐based stereo matching has achieved great success. However, models trained on the source domain dataset encounter substantial performance degradation when directly tested on an unseen target domain dataset because of neglecting the generalization to out‐of‐distribution (OOD) stereo samples. This paper proposes a stereo sample generation‐based domain generalization network (SGDG‐Net) for stereo matching. Specifically, to expand the distribution span of training samples, OOD stereo samples are generated to assist training. To effectively generate OOD left samples, a style transfer‐based generation mechanism is proposed to transmit perturbations to the source left samples. In addition, to generate the OOD right samples, a disparity‐assisted generation strategy is proposed by using disparity map labels as auxiliary information. Experimental results demonstrate that the proposed SGDG‐Net produces remarkable results on four benchmark datasets.

BOOSTING SKULL-STRIPPING PERFORMANCE FOR PEDIATRIC BRAIN IMAGES.

Skull-stripping is the removal of background and non-brain anatomical features from brain images. While many skull-stripping tools exist, few target pediatric populations. With the emergence of multi-institutional pediatric data acquisition efforts to broaden the understanding of perinatal brain development, it is essential to develop robust and well-tested tools ready for the relevant data processing. However, the broad range of neuroanatomical variation in the developing brain, combined with additional challenges such as high motion levels, as well as shoulder and chest signal in the images, leaves many adult-specific tools ill-suited for pediatric skull-stripping. Building on an existing framework for robust and accurate skull-stripping, we propose developmental SynthStrip (d-SynthStrip), a skull-stripping model tailored to pediatric images. This framework exposes networks to highly variable images synthesized from label maps. Our model substantially outperforms pediatric baselines across scan types and age cohorts. In addition, the <1-minute runtime of our tool compares favorably to the fastest baselines. We distribute our model at https://w3id.org/synthstrip.

MMSeaIce: a collection of techniques for improving sea ice mapping with a multi-task model

Abstract. The AutoICE challenge, organized by multiple national and international agencies, seeks to advance the development of near-real-time sea ice products with improved spatial resolution, broader spatial and temporal coverage, and enhanced consistency. In this paper, we present a detailed description of our solutions and experimental results for the challenge. We have implemented an automated sea ice mapping pipeline based on a multi-task U-Net architecture, capable of predicting sea ice concentration (SIC), stage of development (SOD), and floe size (FLOE). The AI4Arctic dataset, which includes synthetic aperture radar (SAR) imagery, ancillary data, and ice-chart-derived label maps, is utilized for model training and evaluation. Among the submissions from over 30 teams worldwide, our team achieved the highest combined score of 86.3 %, as well as the highest scores on SIC (92.0 %) and SOD (88.6 %). Notably, the result analysis and ablation studies demonstrate that instead of model architecture design, a collection of strategies/techniques we employed led to substantial enhancement in accuracy, efficiency, and robustness within the realm of deep-learning-based sea ice mapping. Those techniques include input SAR variable downscaling, input feature selection, spatial–temporal encoding, and the choice of loss functions. By highlighting the various techniques employed and their impacts, we aim to underscore the scientific advancements achieved in our methodology.

Multi3D: 3D-aware multimodal image synthesis

3D-aware image synthesis has attained high quality and robust 3D consistency. Existing 3D controllable generative models are designed to synthesize 3D-aware images through a single modality, such as 2D segmentation or sketches, but lack the ability to finely control generated content, such as texture and age. In pursuit of enhancing user-guided controllability, we propose Multi3D, a 3D-aware controllable image synthesis model that supports multi-modal input. Our model can govern the geometry of the generated image using a 2D label map, such as a segmentation or sketch map, while concurrently regulating the appearance of the generated image through a textual description. To demonstrate the effectiveness of our method, we have conducted experiments on multiple datasets, including CelebAMask-HQ, AFHQ-cat, and shapenet-car. Qualitative and quantitative evaluations show that our method outperforms existing state-of-the-art methods.

Multi-Label Supervised Contrastive Learning

Multi-label classification is an arduous problem given the complication in label correlation. Whilst sharing a common goal with contrastive learning in utilizing correlations for representation learning, how to better leverage label information remains challenging. Previous endeavors include extracting label-level presentations or mapping labels to an embedding space, overlooking the correlation between multiple labels. It exhibits a great ambiguity in determining positive samples with different extent of label overlap between samples and integrating such relations in loss functions. In our work, we propose Multi-Label Supervised Contrastive learning (MulSupCon) with a novel contrastive loss function to adjust weights based on how much overlap one sample shares with the anchor. By analyzing gradients, we explain why our method performs better under multi-label circumstances. To evaluate, we conduct direct classification and transfer learning on several multi-label datasets, including widely-used image datasets such as MS-COCO and NUS-WIDE. Validation indicates that our method outperforms the traditional multi-label classification method and shows a competitive performance when comparing to other existing approaches.

RAPHIA: A deep learning pipeline for the registration of MRI and whole-mount histopathology images of the prostate

Image registration can map the ground truth extent of prostate cancer from histopathology images onto MRI, facilitating the development of machine learning methods for early prostate cancer detection. Here, we present RAdiology PatHology Image Alignment (RAPHIA), an end-to-end pipeline for efficient and accurate registration of MRI and histopathology images. RAPHIA automates several time-consuming manual steps in existing approaches including prostate segmentation, estimation of the rotation angle and horizontal flipping in histopathology images, and estimation of MRI-histopathology slice correspondences. By utilizing deep learning registration networks, RAPHIA substantially reduces computational time. Furthermore, RAPHIA obviates the need for a multimodal image similarity metric by transferring histopathology image representations to MRI image representations and vice versa. With the assistance of RAPHIA, novice users achieved expert-level performance, and their mean error in estimating histopathology rotation angle was reduced by 51% (12 degrees vs 8 degrees), their mean accuracy of estimating histopathology flipping was increased by 5% (95.3% vs 100%), and their mean error in estimating MRI-histopathology slice correspondences was reduced by 45% (1.12 slices vs 0.62 slices). When compared to a recent conventional registration approach and a deep learning registration approach, RAPHIA achieved better mapping of histopathology cancer labels, with an improved mean Dice coefficient of cancer regions outlined on MRI and the deformed histopathology (0.44 vs 0.48 vs 0.50), and a reduced mean per-case processing time (51 vs 11 vs 4.5 min). The improved performance by RAPHIA allows efficient processing of large datasets for the development of machine learning models for prostate cancer detection on MRI. Our code is publicly available at: https://github.com/pimed/RAPHIA.

Raw Spectral Filter Array Imaging for Scene Recognition.

Scene recognition is the task of identifying the environment shown in an image. Spectral filter array cameras allow for fast capture of multispectral images. Scene recognition in multispectral images is usually performed after demosaicing the raw image. Along with adding latency, this makes the classification algorithm limited by the artifacts produced by the demosaicing process. This work explores scene recognition performed on raw spectral filter array images using convolutional neural networks. For this purpose, a new raw image dataset is collected for scene recognition with a spectral filter array camera. The classification is performed using a model constructed based on the pretrained Places-CNN. This model utilizes all nine channels of spectral information in the images. A label mapping scheme is also applied to classify the new dataset. Experiments are conducted with different pre-processing steps applied on the raw images and the results are compared. Higher-resolution images are found to perform better even if they contain mosaic patterns.

Medical Image Description Based on Multimodal Auxiliary Signals and Transformer

Medical image description can be applied to clinical medical diagnosis, but the field still faces serious challenges. There is a serious problem of visual and textual data bias in medical datasets, which are the imbalanced distribution of health and disease data. This can greatly affect the learning performance of data-driven neural networks and finally lead to errors in the generated medical image descriptions. To address this problem, we propose a new medical image description network architecture named multimodal data-assisted knowledge fusion network (MDAKF), which introduces multimodal auxiliary signals to guide the Transformer network to generate more accurate medical reports. In detail, audio auxiliary signals provide clear abnormal visual regions to alleviate the visual data bias problem. However, the audio modality signals with similar pronunciation lack recognizability, which may lead to incorrect mapping of audio labels to medical image regions. Therefore, we further fuse the audio with text features as the auxiliary signal to improve the overall performance of the model. Through the experiments on two medical image description datasets, IU-X-ray and COV-CTR, it is found that the proposed model is superior to the previous models in terms of language generation evaluation indicators.