RGB Branches Research Articles

An edge information fusion perception network for curtain wall frames segmentation

Automatically detection of curtain wall frames is a crucial part of building automation. However, the current curtain wall installation robots mainly rely on manual assistance for positioning, resulting in low efficiency and inaccuracy. To address these drawbacks, we propose an edge information fusion perception network (EIFP-Net) for RGB-D curtain wall frames detection. Specifically, a cross-modal context fusion module (CCFM) allows fusing of RGB and depth features to enhance cross-modal information complementation and capture multi-scale context information simultaneously. In addition, an edge features sensing module (EFSM) is derived from the RGB branch, which is committed to realizing edge feature extraction. Within this module, a differential enhancement module (DEM) is introduced to enhance the edge information. Finally, a multi-scale progressive refinement decoder (MPRD) is designed to refine the features, utilizing the edge information as a guide to capture comprehensive features. The experimental results on the constructed curtain wall frame dataset show that the proposed EIFP-Net performs better than the state-of-the-art detection models, achieving 85.94 %, 88.29 %, 96.84 %, 86.81 %, and 87.10 % on the five evaluation metrics of precision, recall, accuracy, mIoU, and F1-score, respectively.

Global feature-based multimodal semantic segmentation

Incorporating complementary modality into RGB branch can significantly improve the effectiveness of semantic segmentation. However, fusion between the two modalities faces huge challenge due to the difference of their optical dimensions. Existed fusion methods can't keep a balance between performance and efficiency in aggregating detailed features. To address this problem, we propose a global feature-based network (GFBN) for semantic segmentation that establishes mapping function and extraction relationship among the multi-modalities. The GFBN contains three important modules, which are used for feature correction, fusion and edge enhancement. Firstly, the cross-attention rectification module (CARM) adaptively extracts mapping relationships and rectifies the RGB and complementary features. Secondly, the cross-field fusion module (CFM) integrates long-range rectified features of two branches to obtain an optimal fusion feature. Finally, the boundary guidance module (BGM) sharpens the boundary information of the fused features to effectively improve the segmentation accuracy of object boundaries. We make the experiments of GFBN on the challenging MCubeS and ZJU-RGB-Ps datasets. The results show that GFBN outperforms state-of-the-art methods by at least 0.64 % and 0.7 % on mean intersection over union (mIoU), respectively. It demonstrates the performance and efficiency of our proposed method. The code corresponding to our method can be found at the following link: https://github.com/Sci-Epiphany/GFBNext.

Improving Text-Based Person Retrieval by Excavating All-Round Information Beyond Color.

Text-based person retrieval is the process of searching a massive visual resource library for images of a particular pedestrian, based on a textual query. Existing approaches often suffer from a problem of color (CLR) over-reliance, which can result in a suboptimal person retrieval performance by distracting the model from other important visual cues such as texture and structure information. To handle this problem, we propose a novel framework to Excavate All-round Information Beyond Color for the task of text-based person retrieval, which is therefore termed EAIBC. The EAIBC architecture includes four branches, namely an RGB branch, a grayscale (GRS) branch, a high-frequency (HFQ) branch, and a CLR branch. Furthermore, we introduce a mutual learning (ML) mechanism to facilitate communication and learning among the branches, enabling them to take full advantage of all-round information in an effective and balanced manner. We evaluate the proposed method on three benchmark datasets, including CUHK-PEDES, ICFG-PEDES, and RSTPReid. The experimental results demonstrate that EAIBC significantly outperforms existing methods and achieves state-of-the-art (SOTA) performance in supervised, weakly supervised, and cross-domain settings.

SGFNet: Semantic-Guided Fusion Network for RGB-Thermal Semantic Segmentation

Recently, semantic segmentation based on RGB and thermal infrared (TIR) images has become a research hotspot because of its stability in the weak light environment. However, most of the current methods ignore the differences between the two modalities of data and do not use semantic information in multi-modal fusion. In this paper, we propose a novel Semantic-Guided Fusion Network (SGFNet) for RGB-Thermal semantic segmentation, which makes full use of semantic information in the multi-modal fusion. Our SGFNet consists of an asymmetric encoder with TIR branch and RGB branch and a decoder. We concentrate on enhancing the multi-modal feature representation in the encoder with a pattern of fusion and enhancement. Specifically, considering that TIR images are stable under weak light conditions, we first propose a Semantic Guidance Head to extract semantic information in the TIR branch. In the RGB branch, we propose a Multi-modal Coordination and Distillation Unit to fuse multi-modal features first. Then, we propose a Cross-level and Semantic-guided Enhancement Unit to enhance the fused features with cross-level information and semantic information. We arrange these two units at all stages of the RGB branch to generate features with strong representation abilities at different levels. For the decoder, to obtain large receptive fields and fine edges, we improve the Lawin ASPP decoder by introducing edge information extracted from the low-level features, proposing the edge-aware Lawin ASPP decoder. With our encoder and decoder working together, our SGFNet can identify objects accurately and segment objects finely. Extensive experiments on the MFNet dataset demonstrate the superior performance of the proposed SGFNet compared with state-of-the-art methods. The code and results of our method are available at https://github.com/kw717/SGFNet.

TMTNet: A Transformer-Based Multimodality Information Transfer Network for Hyperspectral Object Tracking

Hyperspectral video with spatial and spectral information has great potential to improve object tracking performance. However, the limited hyperspectral training samples hinder the development of hyperspectral object tracking. Since hyperspectral data has multiple bands, from which any three bands can be extracted to form pseudocolor images, we propose a Transformer-based multimodality information transfer network (TMTNet), aiming to improve the tracking performance by efficiently transferring the information of multimodality data composed of RGB and hyperspectral in the hyperspectral tracking process. The multimodality information needed to be transferred mainly includes the RGB and hyperspectral multimodality fusion information and the RGB modality information. Specifically, we construct two subnetworks to transfer the multimodality fusion information and the robust RGB visual information, respectively. Among them, the multimodality fusion information transfer subnetwork is designed based on the dual Siamese branch structure. The subnetwork employs the pretrained RGB tracking model as the RGB branch to guide the training of the hyperspectral branch with little training samples. The RGB modality information transfer subnetwork is designed based on a pretrained RGB tracking model with good performance to improve the tracking network’s generalization and accuracy in unknown complex scenes. In addition, we design an information interaction module based on Transformer in the multimodality fusion information transfer subnetwork. The module can fuse multimodality information by capturing the potential interaction between different modalities. We also add a spatial optimization module to TMTNet, which further optimizes the object position predicted by the subject network by fully retaining and utilizing detailed spatial information. Experimental results on the only available hyperspectral tracking benchmark dataset show that the proposed TMTNet tracker outperforms the advanced trackers, demonstrating the effectiveness of this method.

Dual Mutual Learning for Cross-Modality Person Re-Identification

Cross-modality person re-identification (Re-ID) is more challenging than traditional visible Re-ID due to the huge cross-modality gap from heterogeneous images. To alleviate this problem, existing methods often utilize a dual path learning framework equipped with metric loss to learn discriminative features. Despite effectiveness, the inevitable degeneration of intra-modality discrimination by taking cross-modality discrimination into consideration is unsolvable. Such degeneration substantially hinders the model’s capability of further improving feature representations. To mitigate this degeneration, we propose a Dual Mutual Learning (DML) method for cross-modality Re-ID which conducts mutual learning between the cross-modality and each of two single modalities. We design a triple-branch deep model containing the RGB and IR branches and the cross-modality branch. The cross-modality branch is designed to learn modality-invariant feature subspace for appearance similarity measurement. Both the RGB branch and IR branch provide attention supervision information to the cross-modality branch for attention feature alignment so as to enhance the intra-modality discrimination. Experimental results on two standard benchmarks demonstrate DML is superior to state-of-the-art methods.

Weakly-Supervised Salient Object Detection on Light Fields.

Most existing salient object detection (SOD) methods are designed for RGB images and do not take advantage of the abundant information provided by light fields. Hence, they may fail to detect salient objects of complex structures and delineate their boundaries. Although some methods have explored multi-view information of light field images for saliency detection, they require tedious pixel-level manual annotations of ground truths. In this paper, we propose a novel weakly-supervised learning framework for salient object detection on light field images based on bounding box annotations. Our method has two major novelties. First, given an input light field image and a bounding-box annotation indicating the salient object, we propose a ground truth label hallucination method to generate a pixel-level pseudo saliency map, to avoid heavy cost of pixel-level annotations. This method generates high quality pseudo ground truth saliency maps to help supervise the training, by exploiting information obtained from the light field (including depths and RGB images). Second, to exploit the multi-view nature of the light field data in learning, we propose a fusion attention module to calibrate the spatial and channel-wise light field representations. It learns to focus on informative features and suppress redundant information from the multi-view inputs. Based on these two novelties, we are able to train a new salient object detector with two branches in a weakly-supervised manner. While the RGB branch focuses on modeling the color contrast in the all-in-focus image for locating the salient objects, the Focal branch exploits the depth and the background spatial redundancy of focal slices for eliminating background distractions. Extensive experiments show that our method outperforms existing weakly-supervised methods and most fully supervised methods.

Boundary-enhanced attention-aware network for detecting salient objects in RGB-depth images

In recent years, salient object detection (SOD) in RGB-depth (RGB-D) images has attracted considerable research interest. We present a boundary-enhanced attention-aware network (BANet) for RGB-D SOD by combining boundary and saliency detection networks. In particular, depth maps were used to detect the saliency boundaries effectively, and the corresponding RGB images were used to predict the salient objects. Considering that the data contained in depth maps are insufficient, HSV images were employed as complements to enhance the boundary detection performances. Subsequently, an attention module was used to adaptively weigh the features from the RGB branch and boundary network to improve the SOD performance of the proposed BANet. A loss function combining saliency supervision, background supervision, and boundary supervision was designed to optimize the parameters of the BANet. Extensive experiments were conducted to assess the robustness and effectiveness of the proposed BANet. The results suggest that the proposed BANet shows a significant improvement over other representative SOD approaches.

MFF-Net: Deepfake Detection Network Based on Multi-Feature Fusion.

Significant progress has been made in generating counterfeit images and videos. Forged videos generated by deepfaking have been widely spread and have caused severe societal impacts, which stir up public concern about automatic deepfake detection technology. Recently, many deepfake detection methods based on forged features have been proposed. Among the popular forged features, textural features are widely used. However, most of the current texture-based detection methods extract textures directly from RGB images, ignoring the mature spectral analysis methods. Therefore, this research proposes a deepfake detection network fusing RGB features and textural information extracted by neural networks and signal processing methods, namely, MFF-Net. Specifically, it consists of four key components: (1) a feature extraction module to further extract textural and frequency information using the Gabor convolution and residual attention blocks; (2) a texture enhancement module to zoom into the subtle textural features in shallow layers; (3) an attention module to force the classifier to focus on the forged part; (4) two instances of feature fusion to firstly fuse textural features from the shallow RGB branch and feature extraction module and then to fuse the textural features and semantic information. Moreover, we further introduce a new diversity loss to force the feature extraction module to learn features of different scales and directions. The experimental results show that MFF-Net has excellent generalization and has achieved state-of-the-art performance on various deepfake datasets.

Micro-Expression Recognition Based on Dual-Stream Networks InformationInteraction

<p indent=0mm>Scarcity of experimental databases for deep learning based micro-expression recognition leads to limited knowledge acquisition in learning process and increasing difficulty for improving accuracy and generalization capabilities. In view of the problem, a micro-expression algorithm based on dual-stream network information interaction is proposed. An improved deep mutual learning strategy is designed to guide the interactive training between different modalities of the image sequence to improve the recognition rate of the network, in which the main network is built based on the RGB image sequence, and the auxiliary network is built based on the optical flow. In the training phase, the training process is optimized via a mutual learning loss which includes the supervised learning loss and mimic loss, which ensures the prediction of each mode is consistent with the true identity of the training sample and the predictions of other modalities. In the testing phase, since the mutual learning mechanism enhances the discrimination ability of RGB branches, the optical flow branches can be tailored to improve the recognition speed while ensuring accuracy. The experimental results on the CASME, CASME Ⅱ and SMIC data sets show that the algorithm effectively improves the recognition accuracy and the overall performance is better than existing algorithms.

Learning a Twofold Siamese Network for RGB-T Object Tracking

Visual object tracking works as a key component for many instrumentation and measurement applications such as UAV systems, optical tracking and measuring systems. This paper investigates how to implement accurate RGB-T tracking by integrating the complementary information from RGB and thermal sources. Inspired by the success of Siamese networks in the RGB tracking field, we design a twofold Siamese network for RGB-T tracking, which is composed of an RGB branch and a thermal branch. Each branch is a Siamese network, which can be utilized to compute similarities between the search image and the exemplar image. The parameters in the RGB branch are kept the same as SiamFC. The thermal branch is initialized with parameter weights of the trained network in SiamFC and fine-tuned with constructed thermal image pairs to better capture the target characteristics in the thermal data. Two criteria are further proposed to measure the confidence degrees of response maps obtained by these two modalities. The final response map is computed by adaptively fusing them according to their confidence degrees. The maximum location on the final response map is identified as the target location, and the target scale is obtained through a simple multi-scale search. Experiments on the recently public benchmark RGB-T234 demonstrate the effectiveness of our proposed method when compared to other state-of-the-art trackers. The high performance of our proposed trakcer makes it easy to be implemented in embedded system.

Three-branch architecture for stereoscopic 3D salient object detection

Existing stereoscopic 3D (S3D) salient object detection (SOD) networks typically employ a two-branch architecture, in which the RGB and depth channels are learned independently. Conventional methods based on conventional neural networks generally fuse the two branches by combining their deep representations at a later stage with only one path, which can be inefficient and insufficient for retaining a large amount of cross-modal data. In this study, we combine the RGB branch and depth branch to generate a third branch. The first branch is the embedded attention branch containing the attention mechanism, and we introduce the embedded attention module in this branch to give the allocation of available processing resources to the most informative components of an input signal. The second branch is the boundary refinement branch combined with the low-level information of RGB and depth images. Additionally, we propose a new module, called the detail correlation module, to ensure clear object boundaries and salient object refinement. The third branch is the global deep-view branch containing the global view module, which fuses high-level information and expands the sensor field. We also use three different loss functions to match our special SOD network. Extensive experiments demonstrate the effectiveness and robustness of the proposed architecture and show that it represents a significant improvement over other state-of-the-art SOD approaches.

Triple U-net: Hematoxylin-aware nuclei segmentation with progressive dense feature aggregation.

Nuclei segmentation is a vital step for pathological cancer research. It is still an open problem due to some difficulties, such as color inconsistency introduced by non-uniform manual operations, blurry tumor nucleus boundaries and overlapping tumor cells. In this paper, we aim to leverage the unique optical characteristic of H&E staining images that hematoxylin always stains cell nuclei blue, and eosin always stains the extracellular matrix and cytoplasm pink. Therefore, we extract the Hematoxylin component from RGB images by Beer-Lambert's Law. According to the optical attribute, the extracted Hematoxylin component is robust to color inconsistency. With the Hematoxylin component, we propose a Hematoxylin-aware CNN model for nuclei segmentation without the necessity of color normalization. Our proposed network is formulated as a Triple U-net structure which includes an RGB branch, a Hematoxylin branch and a Segmentation branch. Then we propose a novel feature aggregation strategy to allow the network to fuse features progressively and to learn better feature representations from different branches. Extensive experiments are performed to qualitatively and quantitatively evaluate the effectiveness of our proposed method. In the meanwhile, it outperforms state-of-the-art methods on three different nuclei segmentation datasets.

Object Tracking in RGB-T Videos Using Modal-Aware Attention Network and Competitive Learning.

Object tracking in RGB-thermal (RGB-T) videos is increasingly used in many fields due to the all-weather and all-day working capability of the dual-modality imaging system, as well as the rapid development of low-cost and miniaturized infrared camera technology. However, it is still very challenging to effectively fuse dual-modality information to build a robust RGB-T tracker. In this paper, an RGB-T object tracking algorithm based on a modal-aware attention network and competitive learning (MaCNet) is proposed, which includes a feature extraction network, modal-aware attention network, and classification network. The feature extraction network adopts the form of a two-stream network to extract features from each modality image. The modal-aware attention network integrates the original data, establishes an attention model that characterizes the importance of different feature layers, and then guides the feature fusion to enhance the information interaction between modalities. The classification network constructs a modality-egoistic loss function through three parallel binary classifiers acting on the RGB branch, the thermal infrared branch, and the fusion branch, respectively. Guided by the training strategy of competitive learning, the entire network is fine-tuned in the direction of the optimal fusion of the dual modalities. Extensive experiments on several publicly available RGB-T datasets show that our tracker has superior performance compared to other latest RGB-T and RGB tracking approaches.

Hyperspectral image recovery based on fusion of coded aperture snapshot spectral imaging and RGB images by guided filtering

In compressive hyperspectral imaging systems, the key issue is to accurately and efficiently recover 3D spectral data cubes from spectrally undersampled 2D projections. In this paper, a guided-filtering-based reconstruction algorithm is proposed to recover hyperspectral data cube with a joint coded aperture snapshot spectral imaging (CASSI) and RGB imaging system. In order to preserve the spectral and spatial details in the CASSI and the RGB branch to the greatest extent, 2D projections from the two branches are reconstructed into hyperspectral images individually with a pre-trained dictionary. The appropriate combination of the information from the two branches is realized through guided image filtering, which is applied on the reconstruction result of the CASSI branch with the result from the RGB branch as the guidance image. In virtue of combination between the well-performed spectral fidelity of the CASSI branch and the high spatial accuracy of the RGB branch, the proposed guided-filtering-based compressive hyperspectral imaging (GFCHI) not only overcomes the block effect and blurring common in the state-of-the-art methods, but also improves spectral fidelity related to key applications of hyperspectral imaging. The quantitative metrics associated with spatial, spectral and overall performance demonstrate the state-of-the-art reconstruction quality of GFCHI.

Multiple Populations in NGC 1851: Abundance Variations and UV Photometric Synthesis in the Washington and HST/WFC3 Systems

Abstract The analysis of multiple populations (MPs) in globular clusters (GCs), both spectroscopically and photometrically, is key in understanding their formation and evolution. The relatively narrow Johnson U, F336W, and Stromgren and Sloan u filters have been crucial in exhibiting these MPs photometrically, but in Paper I we showed that the broader Washington C filter can more efficiently detect MPs in the test case GC NGC 1851. Additionally, In Paper I we detected a double main sequence (MS) that has not been detected in previous observations of NGC 1851. We now match this photometry to NGC 1851's published RGB abundances and find that the two RGB branches observed in C generally exhibit different abundance characteristics in a variety of elements (e.g., Ba, Na, and O) and in CN band strengths, but no single element can define the two RGB branches. However, simultaneously considering [Ba/Fe] or CN strength with either [Na/Fe], [O/Fe], or CN strength can separate the two photometric RGB branches into two distinct abundance groups. Matches of NGC 1851's published SGB and HB abundances to the Washington photometry shows consistent characterizations of the MPs, which can be defined as an O-rich/N-normal population and an O-poor/N-rich population. Photometric synthesis for both the Washington C filter and the F336W filter finds that these abundance characteristics, with appropriate variations in He, can reproduce for both filters the photometric observations in both the RGB and the MS. This photometric synthesis also confirms the throughput advantages that the C filter has in detecting MPs.

CHEMICAL ABUNDANCES FOR EVOLVED STARS IN M5: LITHIUM THROUGH THORIUM

We present analysis of high-resolution spectra of a sample of stars in the globular cluster M5 (NGC 5904). The sample includes stars from the red giant branch (seven stars), the red horizontal branch (two stars), and the asymptotic giant branch (eight stars), with effective temperatures ranging from 4000 K to 6100 K. Spectra were obtained with the HIRES spectrometer on the Keck I telescope, with a wavelength coverage from 3700 to 7950 angstroms for the HB and AGB sample, and 5300 to 7600 angstroms for the majority of the RGB sample. We find offsets of some abundance ratios between the AGB and the RGB branches. However, these discrepancies appear to be due to analysis effects, and indicate that caution must be exerted when directly comparing abundance ratios between different evolutionary branches. We find the expected signatures of pollution from material enriched in the products of the hot hydrogen burning cycles such as the CNO, Ne-Na, and Mg-Al cycles, but no significant differences within these signatures among the three stellar evolutionary branches especially when considering the analysis offsets. We are also able to measure an assortment of neutron-capture element abundances, from Sr to Th, in the cluster. We find that the neutron-capture signature for all stars is the same, and shows a predominately r-process origin. However, we also see evidence of a small but consistent extra s-process signature that is not tied to the light-element variations, pointing to a pre-enrichment of this material in the protocluster gas.