Color Information Research Articles

Integrated Transcriptome and Metabolome Analysis Revealed Mechanism Underlying Anthocyanin Biosynthesis During Flower Color Formation in Lagerstroemia indica

Lagerstroemia indica is a widely used ornamental woody plant known for its summer flowering and significant ornamental and economic value. While L. indica boasts a variety of rich flower colors, the molecular mechanisms underlying this color formation remain unclear. In this study, we selected three different flower colors of L. indica—white (W), red (R), and purple (P)—for transcriptome and metabolome analysis. The metabolome analysis identified 538 flavonoids, with 22 anthocyanins highly accumulated in the red and purple flowers. RNA-seq analysis annotated a total of 35,505 genes. Furthermore, we identified 42 differentially expressed genes (DEGs) involved in anthocyanin biosynthesis, with their expression levels aligning with anthocyanin content. Correlation analysis revealed that 19 MYB and 11 bHLH transcription factors are likely involved in anthocyanin biosynthesis. Additionally, we identified 59 auxin biosynthesis and signaling-related genes that are positively correlated with anthocyanin-related genes and metabolites, suggesting that auxin may play a role in regulating anthocyanin biosynthesis in L. indica. This study provides valuable insights into the regulatory mechanisms underlying anthocyanin accumulation and color formation in L. indica petals and identifies several potential genes, laying the groundwork for further research on regulatory mechanisms and genetic improvement of L. indica.

Category-Level Pose Estimation and Iterative Refinement for Monocular RGB-D Image

Category-level pose estimation is proposed to predict the 6D pose of objects under a specific category and has wide applications in fields such as robotics, virtual reality, and autonomous driving. With the development of VR/AR technology, pose estimation has gradually become a research hotspot in 3D scene understanding. However, most methods fail to fully utilize geometric and color information to solve intra-class shape variations, which leads to inaccurate prediction results. To solve the above problems, we propose a novel pose estimation and iterative refinement network, use an attention mechanism to fuse multi-modal information to obtain color features after a coordinate transformation, and design iterative modules to ensure the accuracy of object geometric features. Specifically, we use an encoder-decoder architecture to implicitly generate a coarse-grained initial pose and refine it through an iterative refinement module. In addition, due to the differences between rotation and position estimation, we design a multi-head pose decoder that utilizes the local geometry and global features. Finally, we design a transformer-based coordinate transformation attention module to extract pose-sensitive features from RGB images and supervise color information by correlating point cloud features in different coordinate systems. We train and test our network on the synthetic dataset CAMERA25 and the real dataset REAL275. Experimental results show that our method achieves state-of-the-art performance on multiple evaluation metrics.

Similarity in feature space dictates the efficiency of attentional selection during ensemble processing.

Humans can rapidly and accurately extract statistical information about features of the visual environment, an ability referred to as ensemble perception.However, little is known about how ensemble estimates are affected when task-irrelevant and distracting feature information is present. Here, we tested how effectively feature-based attention-when tuned to a specific color-can select a single item set out of two intermixed ensembles of colored lines. Participants were instructed to report the average orientation of a target-colored item set, while ignoring a second differently colored set. To assess how representational overlap between the two sets impacts color-based selection, we systematically varied the orientation similarity between the relevant and irrelevant items. Our results showed that participants' orientation reports were reliably biased towards the irrelevant items, but interestingly, these biases were only observed when the item sets overlapped in orientation space. In a second experiment, using a visual mask to disrupt access to color information at different time points, we found that these biases were stronger when less time was available to process the stimuli. Together, these results suggest that ensemble representations are rapidly formed based on all available information in the relevant feature dimension, regardless of task relevance, and that selective attention weights and separates these ensemble representations at a relatively later processing stage. This selection appears highly effective when the underlying population activity generated by the two sets is separable along the to-be-estimated feature dimension, but is dampened when relevant and irrelevant ensemble representations overlap in feature space.

Automated Weld Path Generation in Cluttered Environments Using Segmentation and Iterative Closest Point Workpiece Localization

Abstract A significant amount of manufacturing is performed by small to medium enterprises (SMEs), but these manufacturers often have lower adoption rates of automation. The cost and complexity associated with traditional robot systems slow the adoption of robotic welding operations for SMEs. The recent increase in collaborative robotic welding systems is bridging the gap, however, by reducing the complexity of installing, maintaining, and training operators to perform weld operations with collaborative robotics. These systems add flexibility in the range of operator use and ease of deployment. These systems however still rely on kinematic registration between the robot-mounted torch and workpiece on any open-loop weld. This requires precise placement of the workpiece prior to performing a weld. This work will discuss a method for part identification and registration in a welding task as a step toward automated weld path generation. The method is based on lower-resolution 3D cameras (RGBD cameras) using a combination of color and depth information. This information is used to both identify the workpiece within a workspace that may have other, non-workpiece items, and then provide registration or localization information of the workpiece within a resolution that could allow follow-on near-position strategies to achieve final weld path identification.

Avatar-based versus conventional patient monitoring with distant vision: a computer-based simulation study.

Patient monitoring in the perioperative setting can be challenging, especially when monitoring multiple patients simultaneously or managing dynamic situations that require movement around the operating room. We aimed to evaluate whether avatar-based patient monitoring, which presents vital signs in the form of changing colors, shapes and motion, improves remote vital sign recognition compared to conventional monitoring. We conducted a prospective, single-center, computer-based simulation study to evaluate how anesthesia providers recognize vital signs when using the Philips Visual Patient Avatar at different viewing distances (8 and 16m) compared to conventional monitoring. The primary outcome was the total number of correctly identified vital signs which were compared for the two distances and the two devices using mixed Poisson regression. We analyzed data from 28 anesthesia providers who participated in 112 simulations. The correct recognition rate using the Visual Patient Avatar compared to conventional monitoring at 8m was increased by 74% (rate ratio 1.74, 95% CI, 1.42 to 2.14, p < 0.001) and by 51% at 16-meter viewing distance (rate ratio 1.51, 95% CI, 1.23 to 1.87, p < 0.001). We observed scenario-specific superior performance for six vital signs at 8m. The results provide empirical evidence that avatar-based monitoring can significantly improve the perception of vital signs when using distant vision.

A Virtual View Acquisition Technique for Complex Scenes of Monocular Images Based on Layered Depth Images

With the rapid development of stereoscopic display technology, how to generate high-quality virtual view images has become the key in the applications of 3D video, 3D TV and virtual reality. The traditional virtual view rendering technology maps the reference view into the virtual view by means of 3D transformation, but when the background area is occluded by the foreground object, the content of the occluded area cannot be inferred. To solve this problem, we propose a virtual view acquisition technique for complex scenes of monocular images based on a layered depth image (LDI). Firstly, the depth discontinuities of the edge of the occluded area are reasonably grouped by using the multilayer representation of the LDI, and the depth edge of the occluded area is inpainted by the edge inpainting network. Then, the generative adversarial network (GAN) is used to fill the information of color and depth in the occluded area, and the inpainting virtual view is generated. Finally, GAN is used to optimize the color and depth of the virtual view, and the high-quality virtual view is generated. The effectiveness of the proposed method is proved by experiments, and it is also applicable to complex scenes.

No-Reference Quality Assessment Based on Dual-Channel Convolutional Neural Network for Underwater Image Enhancement

Underwater images are important for underwater vision tasks, yet their quality often degrades during imaging, promoting the generation of Underwater Image Enhancement (UIE) algorithms. This paper proposes a Dual-Channel Convolutional Neural Network (DC-CNN)-based quality assessment method to evaluate the performance of different UIE algorithms. Specifically, inspired by the intrinsic image decomposition, the enhanced underwater image is decomposed into reflectance with color information and illumination with texture information based on the Retinex theory. Afterward, we design a DC-CNN with two branches to learn color and texture features from reflectance and illumination, respectively, reflecting the distortion characteristics of enhanced underwater images. To integrate the learned features, a feature fusion module and attention mechanism are conducted to align efficiently and reasonably with human visual perception characteristics. Finally, a quality regression module is used to establish the mapping relationship between the extracted features and quality scores. Experimental results on two public enhanced underwater image datasets (i.e., UIQE and SAUD) show that the proposed DC-CNN method outperforms a variety of the existing quality assessment methods.

MORPHOLOGICAL AND ANATOMICAL RESEARCH OF COMMON MYRTLE LEAVES (MYRTUS COMMUNIS L.)

This paper presents the results of morphological-anatomical study of common myrtle leaves diagnostically significant features (Myrtus communis L.) harvested in the Nikitsky Botanical Garden (Republic of Crimea, Russia). Microscopic analysis of common myrtle leaf petioles was carried out and diagnostic features of dominant biologically active сompounds and tissues luminescence of this plant species leaf petioles were revealed. Morphology and anatomy of common myrtle leaves petioles, luminescence of their tissues were analyzed and summarized and additional diagnostic features of this plant object were revealed. Morphological and anatomical study of common myrtle leaves (Myrtus communis L.) was carried out in accordance with the OFS "Leaves" and "Technique of microscopic and microchemical study of medicinal plant raw materials and medicinal plant preparations" GF RF XIV edition using light microscopy in transmitted and polarized light, as well as luminescence microscopy, which has not been previously studied for this plant. The results of the study concluded that when comparing the luminescence of whole essential oils of myrtle, impure eucalyptus and their components provides a luminescence diagnosis due to the difference in luminescence color; 1,8-cineole is not a diagnostic element in morphological and anatomical luminescence for common myrtle due to the lack of obvious luminescence; euvimal-1 can be used as a diagnostic and luminescence as it has specific luminescence in the form of pink color; the peculiarity of pigmentation of the epithelium of the schizogenic receptacle of the leaf of common myrtle was revealed and the characteristic of its luminescence is species specific for myrtle and can be used as a diagnostic feature in contrast to the impurity raw material of eucalyptus. The results obtained during the study can be further used in the development of the section of the draft pharmacopoeial article “Microscopic features” and other regulatory documentation for a new type of medicinal plant raw material - “Myrtle leaves”, as well as in fundamental botany in the study of species of the family Myrtaceae.

An Efficient Dense Reconstruction Algorithm from LiDAR and Monocular Camera

Dense reconstruction have been studied for decades in the fields of computer vision and robotics, in which LiDAR and camera are widely used. However, vision-based methods are sensitive to illumination variation and lack direct depth, and LiDAR-based methods are limited by sparse LiDAR measurement and lacking color and texture information. In this paper, we propose a novel 3D reconstruction algorithm based on LiDAR and a monocular camera, which realizes dense reconstruction. In the algorithm, a LiDAR odometry is used to get accurate poses and poses calculated by the odometry module are used in the calculation of depth maps and fusion of depth maps, and then mesh and texture mapping are implemented. In addition, a semantic segmentation network and a depth completion network are used to obtain dense and accurate depth maps. The concept of symmetry is utilized to generate 3D models of objects or scenes; that is, the reconstruction and camera imaging of these objects or scenes are symmetrical. Experimental results on public dataset show that the proposed algorithm achieves higher accuracy, efficiency and completeness than existing methods.

Multiomics analysis reveals the involvement of OnDIVARICATA 3 in controlling dynamic flower coloring of Oncidium hybridum

Flower color is one of the main quality and economic traits of ornamental plants, and a large amount of research on flower color mainly focuses on the differences between varieties, while there were few reports on the change of flower color at different developmental stages. In this study, the metabolome and transcriptome of a new strain ‘XM-1’ with dynamic color changes of Oncidium were analyzed. The results showed that rutin, quercetin and carotenoids metabolism decreased significantly during the change of color from yellow to white. Analyzing the correlation network between metabolites and differential expressed genes, 25 key structural genes were detected and regulated by multiple MYB-related transcription factors. The MYB-related transcription factor Cluster-100966.1_OnDIVARICATA 3 was selected for further analysis. The phylogenetic tree of DIVARICATA in different species of Orchidaceae and Arabidopsis thaliana was constructed and the most closely related members were selected for sequence comparison. The results showed that OnDIVARICATA 3 contained MYB-like conserved domains. Subcellular localization results showed that OnDIVARICATA 3 was located in the nucleus. In overexpressing OnDIVARICATA 3 transgenic hairy roots, the expression of flower color related genes FLS, ZEP, and CHYB were significantly up-regulated. In summary, this study characterized the key metabolic pathways in the formation of the dynamic flower color of Oncidium hybridum, and constructed the regulatory network of the MYB-related. These results laid a theoretical foundation for the subsequent research on flower color and genetic engineering technology breeding of Oncidium hybridum.

Multi-omics analysis provides insights into the mechanism underlying fruit color formation in Capsicum.

Fruit color is a crucial attribute of fruit quality in peppers (Capsicum spp.). However, few studies have focused on the mechanism of color formation in immature pepper fruits. In this study, the light-yellow color observed in immature CSJ009 fruits compared to CSJ010 could be attributed to decreased chlorophyll and carotenoid pigments. Through integrated analysis of the transcriptome and metabolome of CSJ009 and CSJ010, we identified 23,930 differentially expressed genes (DEGs) and 345 differentially accumulated metabolites (DAMs). Furthermore, integrated analysis revealed a strong correlation between the HCT-like gene and metabolite MWS0178 (chlorogenic acid). Paraffin section assay revealed that the epidermal cells of immature CSJ010 fruits exhibited a more compact arrangement with significantly greater length than those of CSJ009. Quantitative determination of carotenoids showed that lutein emerged as the predominant carotenoid in immature pepper fruits. Additionally, missense mutation of LCYB2 is likely to lead to a decrease in β-carotene content in immature CSJ009 fruits, whereas CCS may directly catalyze the conversion of lycopene to β-carotene in mature fruits. The null mutation in CCS promoted the biosynthesis of β,ϵ-branch carotenoids leading to lutein being the most abundant carotenoid found in orange CSJ010 fruits. These findings provide important insights into the mechanism underlying color formation in pepper fruits and establish a foundation for the further exploration of color-related genes.

Light intensity plays the key role in the regulation of leaf color, anthocyanin and polyphenol profiles, as well as antioxidant activity of Photinia × fraseri leaves

Photinia × fraseri is an important ornamental plant species widely cultivated in China with vibrant leaf color. However, there is limited information about the difference of phenolic constituents and bioactivity among P. fraseri leaves (PFL) with varied color, and the key factor which determines its color formation. Therefore, the aim of the present study was devoted to establishing the correlation among leaf color, anthocyanins and polyphenols profiles, together with antioxidant activity of PFL under varied light intensity. Results of color and pigment analysis on the four PFL groups indicated that total anthocyanins content was positively correlated with red color formation. Meanwhile, red leaves further exhibited a higher level of total anthocyanins (0.66 ± 0.02 mg/g), phenols (24.86 ± 1.33 mg/g), tannins (6.98 ± 0.23 mg/g), phenolic acids (73.60 ± 3.56 mg/g), condensed tannins (131.24 ± 5.51 mg/g), flavanols (3.42 ± 0.03 mg/g), flavonols (57.58 ± 0.19 mg/g), and antioxidant activities (0.17 ± 0.00 mmol/g for DPPH, 0.99 ± 0.06 mmol/g for ABTS, 1.73 ± 0.03 mmol/g for FRAP). Furthermore, a total of 11 phenolic compounds were identified from PFL, while caffeoylquinic acid derivatives and cyanidin-3-O-glucoside were the key phenolic compounds rich in red leaves. Principal component, hierarchical cluster, and correlation analysis indicated that high light intensity was the key factor responsible for the red color formation, as well as the high levels of total anthocyanins, phenolic acids and flavanols and antioxidant activities in the red leaves. Overall, the present study shall contribute to the exploration of the medicinal potential of the important horticultural plant P. fraseri and provide theoretical guidance for its further industrial development and application.

Duplicate MADS-box genes with split roles and a genetic regulatory network of floral development in long-homostyle common buckwheat

The classic ABC model postulates how three classes of floral homeotic genes (A, B and C) work in a combinational way to confer organ identity to each whorl that make up a perfect flower in core eudicot plants. Fagopyrum esculentum (Polygonaceae) produces dimorphic flowers with single whorl showy tepals, representing a considerable difference with most core eudicots flowers. Here, we explain in detail the function of a duplicated pair of floral homeotic genes involved in the formation of tepals and stamens in the LH F. esculentum. FaesAP1_1 and FaesAP1_2 work together to specify tepal identity. FaesAP3_1/2 or FaesPI_1/2 have redundant function in specifying filament identity, while FaesAP3_2 and FaesPI_2 also retain a conserved role in specifying anther development and gain novel function in style length determination. However, FaesPI_1 gain novel function in floral color formation. In addition, FaesAG can directly regulate stamen and pistil development or binds to the CArG-box of pFaesPI_1 to indirectly regulate stamen and pistil development by a gene regulatory pathway involving FaesAP1_1/2, FaesAP3_1/2 and FaesPI_1/2. Moreover, FaesAP1_1/2 can directly or indirectly regulate B-class gene (FaesAP3_1/2 and FaesPI_1/2) expression to be involved in floral development. Our work has led to detailed insights into the MADS-box gene regulatory networks that control floral developmental process in LH F. esculentum.

Revisiting representation learning of color information: Color medical image segmentation incorporating quaternion

Currently, color medical image segmentation methods commonly extract color and texture features mixed together by default, however, the distribution of color information and texture information is different: color information is represented differently in different color channels of a color image, while the distribution of texture information remains the same. Such a simple and brute-force feature extraction pattern will inevitably result in a partial bias in the model's semantics understanding. In this paper, we decouple the representation learning for color-texture information, and propose a novel network for color medical image segmentation, named CTNet. Specifically, CTNet introduces the Quaternion CNN (QCNN) module to capture the correlation among different color channels of color medical images to generate color features, and uses a designed local-global texture feature integrator (LoG) to mine the textural features from local to global. Moreover, a multi-stage features interaction strategy is proposed to minimize the semantic understanding gap of color and texture features in CTNet, so that they can be subsequently fused to generate a unified and robust feature representation. Comparative experiments on four different color medical image segmentation benchmark datasets show that CTNet strikes an optimal trade-off between segmentation accuracy and computational overhead when compared to current state-of-the-art methods. We also conduct extensive ablation experiments to verify the effectiveness of the proposed components. Our code will be available at https://github.com/Notmezhan/CTNet.

The SAGA Survey. IV. The Star Formation Properties of 101 Satellite Systems around Milky Way–mass Galaxies

We present the star-forming properties of 378 satellite galaxies around 101 Milky Way analogs in the Satellites Around Galactic Analogs (SAGA) Survey, focusing on the environmental processes that suppress or quench star formation. In the SAGA stellar mass range of 106−10 M ⊙, we present quenched fractions, star-forming rates, gas-phase metallicities, and gas content. The fraction of SAGA satellites that are quenched increases with decreasing stellar mass and shows significant system-to-system scatter. SAGA satellite quenched fractions are highest in the central 100 kpc of their hosts and decline out to the virial radius. Splitting by specific star formation rate (sSFR), the least star-forming satellite quartile follows the radial trend of the quenched population. The median sSFR of star-forming satellites increases with decreasing stellar mass and is roughly constant with projected radius. Star-forming SAGA satellites are consistent with the star formation rate–stellar mass relationship determined in the Local Volume, while the median gas-phase metallicity is higher and median H i gas mass is lower at all stellar masses. We investigate the dependence of the satellite quenched fraction on host properties. Quenched fractions are higher in systems with larger host halo mass, but this trend is only seen in the inner 100 kpc; we do not see significant trends with host color or star formation rate. Our results suggest that lower-mass satellites and satellites inside 100 kpc are more efficiently quenched in a Milky Way–like environment, with these processes acting sufficiently slowly to preserve a population of star-forming satellites at all stellar masses and projected radii.

Proteomic analysis identified proteins that are differentially expressed in the flavonoid and carotenoid biosynthetic pathways of Camellia Nitidissima flowers

BackgroundCamellia nitidissima Chi is a popular ornamental plant because of its golden flowers, which contain flavonoids and carotenoids. To understand the regulatory mechanism of golden color formation, the metabolites of C. nitidissima petals at five different developmental stages were detected, a proteome map of petals was first constructed via tandem mass tag (TMT) analysis, and the accuracy of the sequencing data was validated via parallel reaction monitoring (PRM).ResultsNineteen color components were detected, and most of these components were carotenoids that gradually accumulated, while some metabolites were flavonoids that were gradually depleted. A total of 97,647 spectra were obtained, and 6,789 quantifiable proteins were identified. Then, 1,319 differentially expressed proteins (DEPs) were found, 55 of which belong to the flavonoid and carotenoid pathways, as revealed by pairwise comparisons of protein expression levels across the five developmental stages. Notably, most DEPs involved in the synthesis of flavonoids, such as phenylalanine ammonium lyase and 4-coumarate-CoA ligase, were downregulated during petal development, whereas DEPs involved in carotenoid synthesis, such as phytoene synthase, 1-deoxy-D-xylulose-5-phosphate synthase, and β-cyclase, tended to be upregulated. Furthermore, protein‒protein interaction (PPI) network analysis revealed that these 55 DEPs formed two distinct PPI networks closely tied to the flavonoid and carotenoid synthesis pathways. Phytoene synthase and chalcone synthase exhibited extensive interactions with numerous other proteins and displayed high connectivity within the PPI networks, suggesting their pivotal biological functions in flavonoid and carotenoid biosynthesis.ConclusionProteomic data on the flavonoid and carotenoid biosynthesis pathways were obtained, and the regulatory roles of the DEPs were analyzed, which provided a theoretical basis for further understanding the golden color formation mechanism of C. nitidissima.

Comparative Lipidomics Analysis Provides New Insights into the Metabolic Basis of Color Formation in Green Cotton Fiber.

Green fiber (GF) is a naturally colored fiber. A limited understanding of its color formation mechanism restricts the improvement of colored cotton quality. This experiment used upland cotton green fiber germplasm 1-4560 and genetic inbred line TM-1; the lipid profiles of green fibers at 30 (white stage) and 35 days post-anthesis (DPA) (early greening stage), as well as those of TM-1 at the same stages, were revealed. Among the 109 differential types of lipids (DTLs) unique to GF, the content of phosphatidylserine PS (16:0_18:3) was significantly different at 30 and 35 DPA. It is speculated that this lipid is crucial for the pigment accumulation and color formation process of green fibers. The 197 DTLs unique to TM-1 may be involved in white fiber (WF) development. Among the shared DTLs in GF35 vs. GF30 and WF35 vs. WF30, sulfoquinovosyldiacyl-glycerol SQDG (18:1_18:1) displays a significant difference in the content change between green fibers and white fibers, potentially affecting color formation through changes in content. The enriched metabolic pathways in both comparison groups are relatively conserved. In the most significantly enriched glycerophospholipid metabolic pathway, 1-acyl-sn-glycero-3-phosphocholine (C04230) only appears in white cotton. This indicates differences in the metabolic pathways between white and green fibers, potentially related to different mechanisms of color formation and fiber development. These findings provide a new theoretical basis for studying cotton fiber development and offer important insights into the specific mechanism of green fiber color formation.

Map construction algorithm based on dense point cloud

AbstractIn order to address the issue of the difficulty in constructing maps from sensor‐acquired point clouds, a map construction algorithm is proposed for the effective conversion from sparse point clouds to dense point clouds. First, it constructs a dense point cloud map in real‐time using colour and depth information of keyframes. Then, it generates an accurate point cloud map by voxel filtering and coordinate conversion. Finally, the point cloud map is converted into an octree map using OctoMap. This article uses the TUM dataset for simulation analysis to verify the effectiveness of the algorithm. The results demonstrate that the algorithm can construct dense point cloud maps with high positioning accuracy in sparse point cloud scenarios. The algorithm improves the positioning accuracy by more than 15% under the fr1_xyz, fr1_room, fr1_desk2, fr1_desk1, fr2_desk and fr3_str_tex_near sequences, compared to ORB‐SLAM2, ORB‐SLAM3 and GX ORB‐SLAM2 algorithm

Asymmetric Deformable Spatio-temporal Framework for Infrared Object Tracking

The Infrared Object Tracking (IOT) task aims to locate objects in infrared sequences. Since color and texture information is unavailable in infrared modality, most existing infrared trackers merely rely on capturing spatial contexts from the image to enhance feature representation, where other complementary information is rarely deployed. To fill this gap, we in this article propose a novel Asymmetric Deformable Spatio-Temporal Framework (ADSF) to fully exploit collaborative shape and temporal clues in terms of the objects. Firstly, an asymmetric deformable cross-attention module is designed to extract shape information, which attends to the deformable correlations between distinct frames in an asymmetric manner. Secondly, a spatio-temporal tracking framework is coined to learn the temporal variance trend of the object during the training process and store the template information closest to the tracking frame when testing. Comprehensive experiments demonstrate that ADSF outperforms state-of-the-art methods on three public datasets. Extensive ablation experiments further confirm the effectiveness of each component in ADSF. Furthermore, we conduct generalization validation to demonstrate that the proposed method also achieves performance gains in RGB-based tracking scenarios.

Multi-Modal LiDAR Point Cloud Semantic Segmentation with Salience Refinement and Boundary Perception

Point cloud segmentation is essential for scene understanding, which provides advanced information for many applications, such as autonomous driving, robots, and virtual reality. To improve the accuracy and robustness of point cloud segmentation, many researchers have attempted to fuze camera images to complement the color and texture information. The common fusion strategy is the combination of convolutional operations with concatenation, element-wise addition or element-wise multiplication. However, conventional convolutional operators tend to confine the fusion of modal features within their receptive fields, which can be incomplete and limited. In addition, the inability of encoder–decoder segmentation networks to explicitly perceive segmentation boundary information results in semantic ambiguity and classification errors at object edges. These errors are further amplified in point cloud segmentation tasks, significantly affecting the accuracy of point cloud segmentation. To address the above issues, we propose a novel self-attention multi-modal fusion semantic segmentation network for point cloud semantic segmentation. Firstly, to effectively fuze different modal features, we propose a self-cross fusion module (SCF), which models long-range modality dependencies and transfers complementary image information to the point cloud to fully leverage the modality-specific advantages. Secondly, we design the salience refinement module (SR), which calculates the importance of channels in the feature maps and global descriptors to enhance the representation capability of salient modal features. Finally, we propose the local-aware anisotropy loss measure the element-level importance in the data and explicitly provide boundary information for the model, which alleviates the inherent semantic ambiguity problem in segmentation networks. Extensive experiments on two benchmark datasets demonstrate that our proposed method surpasses current state-of-the-art methods.