Complementary Information Research Articles

Multi-Modal LiDAR Point Cloud Semantic Segmentation with Salience Refinement and Boundary Perception

Point cloud segmentation is essential for scene understanding, which provides advanced information for many applications, such as autonomous driving, robots, and virtual reality. To improve the accuracy and robustness of point cloud segmentation, many researchers have attempted to fuze camera images to complement the color and texture information. The common fusion strategy is the combination of convolutional operations with concatenation, element-wise addition or element-wise multiplication. However, conventional convolutional operators tend to confine the fusion of modal features within their receptive fields, which can be incomplete and limited. In addition, the inability of encoder–decoder segmentation networks to explicitly perceive segmentation boundary information results in semantic ambiguity and classification errors at object edges. These errors are further amplified in point cloud segmentation tasks, significantly affecting the accuracy of point cloud segmentation. To address the above issues, we propose a novel self-attention multi-modal fusion semantic segmentation network for point cloud semantic segmentation. Firstly, to effectively fuze different modal features, we propose a self-cross fusion module (SCF), which models long-range modality dependencies and transfers complementary image information to the point cloud to fully leverage the modality-specific advantages. Secondly, we design the salience refinement module (SR), which calculates the importance of channels in the feature maps and global descriptors to enhance the representation capability of salient modal features. Finally, we propose the local-aware anisotropy loss measure the element-level importance in the data and explicitly provide boundary information for the model, which alleviates the inherent semantic ambiguity problem in segmentation networks. Extensive experiments on two benchmark datasets demonstrate that our proposed method surpasses current state-of-the-art methods.

Temperature Dependence of Hydrotropy.

The solubility of hydrophobic solutes increases dramatically with the temperature when hydrotropes are added to water. In this paper, the mechanism of this well-known observation will be explained via statistical thermodynamics through (i) enhanced enthalpy-hydrotrope number correlation locally (around the solute) that promotes the temperature dependence and (ii) hydrotrope self-association in the bulk solution that suppresses the temperature dependence. The contribution from (i), demonstrated to be dominant for urea as a hydrotrope, signifies the weakening of interaction energies around the solute (local) than in the bulk that accompanies incoming hydrotrope molecules. Thus, studying hydrotropic solubilization along the temperature and hydrotrope concentration provides complementary information on the local-bulk difference: the local accumulation of hydrotropes around the solute, driven by the enhanced local hydrotrope self-association, is also accompanied by the overall local weakening of energetic interactions, reflecting the fluctuational nature of hydrotrope association and the mediating role of water molecules.

DDNet: Depth Dominant Network for Semantic Segmentation of RGB-D Images

Convolutional neural networks (CNNs) have been widely applied to parse indoor scenes and segment objects represented by color images. Nonetheless, the lack of geometric and context information is a problem for most RGB-based methods, with which depth features are only used as an auxiliary module in RGB-D semantic segmentation. In this study, a novel depth dominant network (DDNet) is proposed to fully utilize the rich context information in the depth map. The critical insight is that obvious geometric information from the depth image is more conducive to segmentation than RGB data. Compared with other methods, DDNet is a depth-based network with two branches of CNNs to extract color and depth features. As the core of the encoder network, the depth branch is given a larger fusion weight to extract geometric information, while semantic information and complementary geometric information are provided by the color branch for the depth feature maps. The effectiveness of our proposed depth-based architecture has been demonstrated by comprehensive experimental evaluations and ablation studies on challenging RGB-D semantic segmentation benchmarks, including NYUv2 and a subset of ScanNetv2.

Proposal Semantic Relationship Graph Network for Temporal Action Detection

Temporal action detection, a critical task in video activity understanding, is typically divided into two stages: proposal generation and classification. However, most existing methods overlook the importance of information transfer among proposals during classification, often treating each proposal in isolation, which hampers accurate label prediction. In this paper, we propose a novel method for inferring semantic relationships both within and between action proposals, guiding the fusion of action proposal features accordingly. Building on this approach, we introduce the Proposal Semantic Relationship Graph Network (PSRGN), an end-to-end model that leverages intra-proposal semantic relationship graphs to extract cross-scale temporal context and an inter-proposal semantic relationship graph to incorporate complementary neighboring information, significantly improving proposal feature quality and overall detection performance. This is the first method to apply graph structure learning in temporal action detection, adaptively constructing the inter-proposal semantic graph. Extensive experiments on two datasets demonstrate the effectiveness of our approach, achieving state-of-the-art results. Code and results are available at http://github.com/Riiick2011/PSRGN .

The added value of cardiopulmonary exercise testing in the diagnosis of heart failure with preserved ejection fraction

Abstract Background Diagnosing heart failure with preserved ejection fraction (HFpEF) is challenging. Two validated algorithms, the HFA-PEFF and H2FPEF scores, have been proposed to guide our clinical practice. Many patients are categorized into an intermediate probability for HFpEF, representing a diagnostic "gray zone." This necessitates further evaluation, including right heart catheterization (RHC) at rest and possibly during exercise, to confirm or exclude the diagnosis. RHC, while informative, is resource-consuming and requires execution in specialized centers. Cardiopulmonary exercise testing (CPET) may offer a non-invasive and cost-effective alternative for differential dyspnea diagnosis. Purpose To evaluate the role of CPET in the diagnostic algorithm of unexplained dyspnea/suspected HFpEF. Methods This single-center retrospective study analyzed patients with a left ventricular ejection fraction (LVEF) ≥ 50% who underwent CPET for unexplained dyspnea from 2016 to 2020. We assessed HFpEF probability using the H2FPEF score and the HFA-PEFF algorithm. We hypothesized that specific CPET parameters (VO2 peak< 80% of predicted and/or VE/VCO2 slope > 35) might provide complementary information to the HFpEF probability scores. Additionally, we hypothesized that CPET could help identifying HFpEF in patients with intermediate probability in a subcohort of patients who underwent exercise RHC. Results Out of 103 patients, 61 (59%) were categorized as intermediate probability of HFpEF according to the HFA-PEFF score, and 59 (57%) according to the H2FPEF score. CPET results (VO2 peak and VE/VCO2 slope) were not associated with the pre-test probability of HFpEF as defined by either of the two scores. Exercise RHC was performed in 26 (43%) and in 27 (46%) patients with intermediate HFA-PEFF score and intermediate H2FPEF score, respectively. Among these patients with intermediate HFpEF probability, VO2 peak < 80% and/or VE/VCO2 slope > 35 had a good specificity and positive predictive value for HFpEF, as compared with exercise RHC (Table). Conclusions CPET provides complementary information to HFA-PEFF and H2FPEF scores. In particular, our results suggest that CPET may reclassify as HFpEF over 25% of patients with an intermediate HFpEF probability according to either the HFA-PEFF score or the H2FPEF score, potentially obviating the need for RHC. Thus, CPET might be incorporated into the HFpEF diagnostic algorithms.Table

Musculoskeletal Tissue Asymmetries Following Anterior Cruciate Ligament Reconstruction

OBJECTIVE: To compare musculoskeletal tissue asymmetries between those who had undergone ACLR and non-surgical controls, and to evaluate the influence of sex on asymmetries and perceptions of knee function in ACLR patients. DESIGN: Cross-sectional cohort analysis. METHODS: Individuals (female n=11, male n=11) with a history of ACLR and matched controls (female n=10, male n=9) completed Dual-energy X-ray Absorptiometry (DXA) scans to measure lower-body bone mineral content (BMC), Fat%, and lean mass (LM). Perceptions of knee function were captured via the International Knee Documentation Committee (IKDC) instrument and inter-limb asymmetry indices (AI%) were calculated for each DXA outcome. Surgical groups and sexes were compared using independent t-tests ( p<0.05) and Cohen’s effect sizes ( d). Additionally, Pearson’s Correlation Coefficients evaluated the association between AI% and IKDC scores. RESULTS: The ACLR group showed greater lower-body asymmetries, specifically less BMC and LM and greater Fat% in the ACLR limb ( p≤0.044; d=0.63–0.87). ACLR group sex comparisons found that females had lower IKDC scores, greater asymmetries for BMC and LM ( p≤0.027, d=0.40–0.74), and these asymmetries were negatively correlated with IKDC scores (all r≥0.755; p≤0.007) while males demonstrated no such correlations. CONCLUSIONS: Individuals with a history of ACLR had persistent musculoskeletal tissue asymmetries that favored the non-surgical leg. The asymmetries were greater in female versus male patients and were associated with lower perceptions of knee function. Thus, musculoskeletal imaging may provide complementary information to the current standard of care during ACLR rehabilitation.

Relationship between six-minute walk distance and cardiopulmonary exercise testing in non-obstructive hypertrophic cardiomyopathy

Abstract Background Recent studies have shown the promise of new approaches to the management of hypertrophic cardiomyopathy (HCM) symptoms. However, as HCM is a chronic disorder that evolves slowly over decades with low annual event rates, therapeutic trials in the disease rely on surrogate measures of disease severity and progression rather than hard endpoints such as mortality. Exercise capacity is one potential surrogate endpoint. Purpose To examine the relationship between peak oxygen uptake (VO2) and six-minute walk test distance (6MWD) in patients with non-obstructive hypertrophic cardiomyopathy (nHCM) and describe the association with clinical parameters and quality of life. Methods We examined baseline data from patients with nHCM enrolled in a randomised placebo-controlled trial of trimetazidine therapy. The study was conducted in a single European centre. Cardiopulmonary exercise testing (CPET) was performed to assess peak oxygen consumption (pVO2), ventilatory anaerobic threshold (VAT) and ventilatory equivalent for carbon dioxide (minute ventilation to carbon dioxide production (VEVCO2). Health related quality of life was assessed with the Minnesota Living with Heart Failure Questionnaire (MLHFQ). ECG and echocardiography were performed on the same day and biomarkers (NT-proBNP and cardiac Troponin T, cTnT) were measured prior to exercise testing. Results In 51 patients (age 50 ± 13 years; 71% male) with nHCM, pVO2 ranged from 10 to 24.3 ml/kg/min and 6MWD from 188 to 650 metres. 6MWD correlated with peak VO2 (Pearson r=0.41, 95% CI 0.15 to 0.67, p=0.003) and anaerobic threshold (r=0.32, 95%CI 0.04 to 0.55, p=0.024). After adjusting for age and sex, NT-proBNP concentration correlated with peak VO2 (radi=-0.35, 95% CI -0.69 to -0.01, p=0.042) but not 6MWD (radj=-0.05, 95% CI -0.42 to 0.32, p=0.779). Health related quality of life assessed by the Minnesota Living with Heart Failure Questionnaire (MLHFQ) correlated moderately with 6MWD (radi=-0.54, 95% CI -0.79 to -0.29, p<0.001) but weakly with peak VO2 (radi 0.28, 95% CI -0.55 to -0.01, p=0.042). I Conclusions There is a modest correlation between peak VO2 and 6MWD in nHCM. The pVO2 correlates more strongly with NT-proBNP whereas 6MWD better reflects quality of life. These data suggest pVO2 and 6MWD provide complementary information about health status in nHCM.Relationship between 6MWD and CPETCharacteristics of study cohort

Collaborate decision network based on cross-modal attention for social media microblog recognition

Recently, social media has gradually become an important topic in public opinion analysis field. Among social media, Microblog is one of the most important platforms because it is short, convenient, mobile and instantaneous. Social media microblog recognition well reflects the attitudes from an enormous big colony to a specific incident, either positive or negative, which can be used for deriving competitive intelligence, marketing strategies, detecting depression and so on. However, the existing methods usually use only text or image from internet but not take advantages of their complementary information to finalize the recognition, it limits the performance and robustness of the algorithms. In this paper, we present a collaborate decision network (CDN) based on cross-modal attention to exploit the discriminative attributes of multi modalities by data- and knowledge joint driven strategy in depth, and further improve the recognition performance. In addition, we collect and construct a visual-text microblog recognition dataset with 2854 samples to support the subsequent research of related fields. Finally, experimental reuslts on the collected dataset show the effectiveness and superiority of the proposed CDN.

A Multi-Hierarchical Complementary Feature Interaction Network for Accelerated Multi-Modal MR Imaging

Magnetic resonance (MR) imaging is widely used in the clinical field due to its non-invasiveness, but the long scanning time is still a bottleneck for its popularization. Using the complementary information between multi-modal imaging to accelerate imaging provides a novel and effective MR fast imaging solution. However, previous technologies mostly use simple fusion methods and fail to fully utilize their potential sharable knowledge. In this study, we introduced a novel multi-hierarchical complementary feature interaction network (MHCFIN) to realize joint reconstruction of multi-modal MR images with undersampled data and thus accelerate multi-modal imaging. Firstly, multiple attention mechanisms are integrated with a dual-branch encoder–decoder network to represent shared features and complementary features of different modalities. In the decoding stage, the multi-modal feature interaction module (MMFIM) acts as a bridge between the two branches, realizing complementary knowledge transfer between different modalities through cross-level fusion. The single-modal feature fusion module (SMFFM) carries out multi-scale feature representation and optimization of the single modality, preserving better anatomical details. Extensive experiments are conducted under different sampling patterns and acceleration factors. The results show that this proposed method achieves obvious improvement compared with existing state-of-the-art reconstruction methods in both visual quality and quantity.

Fault3DNnet: A lightweight 3D seismic fault detection network with bidirectional decoding

Fault detection from seismic data is a critical component of hydrocarbon exploration and production. In recent years, deep-learning techniques have made tremendous achievements and breakthroughs in seismic fault detection. Especially U-nets, represented by FaultSeg3D and its variants, are a dominant method for 3D deep-learning-based seismic fault interpretation. The incorporation of state-of-the-art network modules or training strategies with U-nets continuously improves the performance of fault detection for field seismic data, usually at the cost of increased computational intensity and hardware consumption. We design a lightweight bidirectional decoding network, Fault3DNnet, for fault detection from 3D seismic data volume. To make it simple and concise, our Fault3DNnet is equipped with basic conventional convolution and regular operation modules. The two decoding branches in Fault3DNnet could provide complementary information and improve fault detection performance. The superiority of Fault3DNnet is verified using publicly available synthetic and four field seismic data sets. The experimental results indicate that the method predicts fault structures with improved completeness and continuity compared with FaultSeg3D. Meanwhile, the parameter and computational quantity of the network are only 1/7 and 1/5 of FaultSeg3D, respectively. With the same graphics processing unit memory consumption, Fault3DNnet can handle the size of approximately six times the volume of 3D seismic data during inferencing as that of FaultSeg3D. It helps to improve prediction stability and decrease discontinuous artifacts at the stitching boundaries when predicting the large field data in chunks. In general, our Fault3DNnet delivers superior fault detection results while greatly reducing computational intensity and memory usage.

Utilizing Pseudo Color Image to Improve the Performance of Deep Transfer Learning-Based Computer-Aided Diagnosis Schemes in Breast Mass Classification.

The purpose of this study is to investigate the impact of using morphological information in classifying suspicious breast lesions. The widespread use of deep transfer learning can significantly improve the performance of themammogram based CADx schemes.However, digital mammograms are grayscale images, while deep learning models are typically optimized using the natural images containing three channels. Thus, it is needed to convert the grayscale mammograms into three channel images for the input of deep transfer models. This study aims to developa novel pseudo color image generation methodwhich utilizesthe mass contour information to enhance the classification performance.Accordingly, a total of 830 breast cancer cases were retrospectively collected, which contains 310 benign and 520 malignant cases, respectively. For each case, a total of four regions of interest (ROI) are collected from the grayscale images captured for both the CC and MLO views of the two breasts. Meanwhile, a total of seven pseudo color image sets are generated as the input of the deep learning models, which arecreated through a combination of the original grayscale image, a histogram equalized image, a bilaterally filtered image, and a segmented mass. Accordingly, the output features from four identical pre-trained deep learning models are concatenated and then processed by a support vector machine-based classifier to generate the final benign/malignant labels. The performance of each image set was evaluated and compared. The results demonstrate that the pseudo color sets containing the manually segmented mass performed significantly better than all other pseudo color sets, which achieved an AUC (area under the ROC curve) up to 0.889 ± 0.012 and an overall accuracy up to 0.816 ± 0.020, respectively. At the same time, the performance improvement is also dependent on the accuracy of the mass segmentation. The results of this study support our hypothesis that adding accurately segmented mass contours can provide complementary information, thereby enhancing the performance of the deep transfer model in classifying suspicious breast lesions.

Aggressiveness classification of clear cell renal cell carcinoma using registration-independent radiology-pathology correlation learning.

Renal cell carcinoma (RCC) is a common cancer that varies in clinical behavior. Clear cell RCC (ccRCC) is the most common RCC subtype, with both aggressive and indolent manifestations. Indolent ccRCC is often low-grade without necrosis and can be monitored without treatment. Aggressive ccRCC is often high-grade and can cause metastasis and death if not promptly detected and treated. While most RCCs are detected on computed tomography (CT) scans, aggressiveness classification is based on pathology images acquired from invasive biopsy orsurgery. CT imaging-based aggressiveness classification would be an important clinical advance, as it would facilitate non-invasive risk stratification and treatment planning. Here, we present a novel machine learning method, Correlated Feature Aggregation By Region (CorrFABR), for CT-based aggressiveness classification ofccRCC. CorrFABR is a multimodal fusion algorithm that learns from radiology and pathology images, and clinical variables in a clinically-relevant manner. CorrFABR leverages registration-independent radiology (CT) and pathology image correlations using features from vision transformer-based foundation models to facilitate aggressiveness assessment on CT images. CorrFABR consists of three main steps: (a) Feature aggregation where region-level features are extracted from radiology and pathology images at widely varying image resolutions, (b) Fusion where radiology features correlated with pathology features (pathology-informed CT biomarkers) are learned, and (c) Classification where the learned pathology-informed CT biomarkers, together with clinical variables of tumor diameter, gender, and age, are used to distinguish aggressive from indolent ccRCC using multi-layer perceptron-based classifiers. Pathology images are only required in the first two steps of CorrFABR, and are not required in the prediction module. Therefore, CorrFABR integrates information from CT images, pathology images, and clinical variables during training, but for inference, it relies solely on CT images and clinical variables, ensuring its clinical applicability. CorrFABR was trained with heterogenous, publicly-available data from 298 ccRCC tumors (136 indolent tumors, 162 aggressive tumors) in a five-fold cross-validation setup and evaluated on an independent test set of 74 tumors with a balanced distribution of aggressive and indolent tumors. Ablation studies were performed to test the utility of each component ofCorrFABR. CorrFABR outperformed the other classification methods, achieving an ROC-AUC (area under the curve) of 0.855±0.0005 (95% confidence interval: 0.775, 0.947), F1-score of 0.793±0.029, sensitivity of 0.741±0.058, and specificity of 0.876±0.032 in classifying ccRCC as aggressive or indolent subtypes. It was found that pathology-informed CT biomarkers learned through registration-independent correlation learning improves classification performance over using CT features alone, irrespective of the kind of features or the classification model used. Tumor diameter, gender, and age provide complementary clinical information, and integrating pathology-informed CT biomarkers with these clinical variables further improvesperformance. CorrFABR provides a novel method for CT-based aggressiveness classification of ccRCC by enabling the identification of pathology-informed CT biomarkers, and integrating them with clinical variables. CorrFABR enables learning of these pathology-informed CT biomarkers through a novel registration-independent correlation learning module that considers unaligned radiology and pathology images at widely varying imageresolutions.

Development of a Hyperspectral Imaging Protocol for Painting Applications at the University of Seville

In the last decade, the hyperspectral imaging (HSI) method allowed performing non-invasive analysis in the field of cultural heritage. However, a considerable limitation was given by redundant and time-consuming features, with the necessary application of statistical algorithms and image-processing tools to extract relevant information. In this study, the Centro Nacional de Aceleradores (CNA) and the Group of Interdisciplinary Physics (GFI) of the School of Engineering (ETSI) of the University of Seville tested the application of three different hyperspectral cameras in the visible and near-infrared (VNIR) and short-wave infrared (SWIR) range for the investigation of an ancient painting. A reference-based procedure was realised to build a starting personal library and to evaluate the best working conditions for non-invasive and non-destructive characterisation with data treatment using the commercially available software Evince® and Specim IQ® to apply, respectively, the Principal Component Analysis (PCA) model functions and the classification method. The evaluation of the protocol was tested by acquiring complementary information by X-ray fluorescence (XRF), Ultraviolet Luminescence (UVL) imaging, and Infrared Reflectography (IRR). This exploration established a simplified protocol to analyse the large collection of paintings of the Archbishop’s Palace and the Cathedral of Seville.

An ensemble deep learning model for medical image fusion with Siamese neural networks and VGG-19.

Multimodal medical image fusion methods, which combine complementary information from many multi-modality medical images, are among the most important and practical approaches in numerous clinical applications. Various conventional image fusion techniques have been developed for multimodality image fusion. Complex procedures for weight map computing, fixed fusion strategy and lack of contextual understanding remain difficult in conventional and machine learning approaches, usually resulting in artefacts that degrade the image quality. This work proposes an efficient hybrid learning model for medical image fusion using pre-trained and non-pre-trained networks i.e. VGG-19 and SNN with stacking ensemble method. The model leveraging the unique capabilities of each architecture, can effectively preserve the detailed information with high visual quality, for numerous combinations of image modalities in image fusion challenges, notably improved contrast, increased resolution, and lower artefacts. Additionally, this ensemble model can be more robust in the fusion of various combinations of source images that are publicly available from Havard-Medical-Image-Fusion Datasets, GitHub. and Kaggle. Our proposed model performance is superior in terms of visual quality and performance metrics to that of the existing fusion methods in literature like PCA+DTCWT, NSCT, DWT, DTCWT+NSCT, GADCT, CNN and VGG-19.

Internally-consistent and fully-unbiased multimodal MRI brain template construction from UK Biobank: Oxford-MM

Abstract Anatomical MRI templates of the brain are essential to group-level analyses and image processing pipelines, as they provide a reference space for spatial normalisation. While it has become common for studies to acquire multimodal MRI data, many templates are still limited to one type of modality, usually either scalar or tensor-based. Aligning each modality in isolation does not take full advantage of the available complementary information, such as strong contrast between tissue types in structural images, or axonal organisation in the white matter in diffusion tensor images. Most existing strategies for multimodal template construction either do not use all modalities of interest to inform the template construction process, or do not use them in a unified framework. Here, we present multimodal, cross-sectional templates constructed from UK Biobank data: the OMM-1 template, and age-dependent templates for each year of life between 45 to 81. All templates are fully unbiased to represent the average shape of the populations they were constructed from, and internally consistent through jointly informing the template construction process with T1, T2-FLAIR and DTI data. The OMM-1 template was constructed with a multi-resolution, iterative approach using 240 individuals in the 50-55 year age range. The age-dependent templates were estimated using a Gaussian Process, which describes the change in average brain shape with age in 37,330 individuals. All templates show excellent contrast and alignment within and between modalities. The global brain shape and size is not preconditioned on existing templates, although maximal possible compatibility with MNI-152 space was maintained through rigid alignment. We showed benefits in registration accuracy across two datasets (UK Biobank and HCP), when using the OMM-1 as the template compared with FSL’s MNI-152 template, and found that the use of age-dependent templates further improved accuracy to a small but detectable extent. All templates are publicly available and can be used as a new reference space for uni- or multimodal spatial alignment.

Ecological and taxonomic dissimilarity in species and higher taxa of reptiles in western Mexico.

Reptiles are one of the most diverse groups of vertebrates in the world that are distributed in almost all ecosystems. Many of these studies have focused on exploring their diversity patterns across different environments; and recent studies on reptile alpha and beta diversity have incorporated a multifaceted approach into their analysis to have more comprehensive evaluations. This study presents an assessment of the taxonomic diversity of reptile patterns using methods that incorporate the assessment of higher taxa. Likewise, the taxonomic dissimilarities between reptile communities in the physiographic regions of the state of Jalisco were analyzed. Evaluations for the groups of snakes and lizards independently are presented. We use the taxonomic distinctiveness index that assesses the complexity of the taxonomic structure of communities through hierarchical classification above the species level to measure the relationships between taxa. The dissimilarity of the taxonomic structure in each community was also analyzed. Beta diversity partitions were performed to identify the contribution of turnover and the differences in richness. We determined that alpha diversity of species and higher taxa maintain different patterns, indicating that Jalisco presents regions with overrepresentation of reptile families and genera, as well as regions with an opposite trend. The representation of higher taxa is higher in the lizard group, although in terms of species richness snakes are the most prominent group. The turnover is the most important component at species and higher taxa, with similar values for lizards and snakes. The findings presented show that incorporating phylogenetic information about species through taxonomic relationships provides complementary information that species diversity per se, especially at the level of alpha diversity.

Remote Sensing LiDAR and Hyperspectral Classification with Multi-Scale Graph Encoder–Decoder Network

The rapid development of sensor technology has made multi-modal remote sensing data valuable for land cover classification due to its diverse and complementary information. Many feature extraction methods for multi-modal data, combining light detection and ranging (LiDAR) and hyperspectral imaging (HSI), have recognized the importance of incorporating multiple spatial scales. However, effectively capturing both long-range global correlations and short-range local features simultaneously on different scales remains a challenge, particularly in large-scale, complex ground scenes. To address this limitation, we propose a multi-scale graph encoder–decoder network (MGEN) for multi-modal data classification. The MGEN adopts a graph model that maintains global sample correlations to fuse multi-scale features, enabling simultaneous extraction of local and global information. The graph encoder maps multi-modal data from different scales to the graph space and completes feature extraction in the graph space. The graph decoder maps the features of multiple scales back to the original data space and completes multi-scale feature fusion and classification. Experimental results on three HSI-LiDAR datasets demonstrate that the proposed MGEN achieves considerable classification accuracies and outperforms state-of-the-art methods.

Comprehensive liquid biopsy analysis for monitoring NSCLC patients under second-line osimertinib treatment

BackgroundThe heterogeneous and complex genetic landscape of NSCLC impacts the clinical outcomes of patients who will eventually develop resistance to osimertinib. Liquid biopsy (LB) analysis as a minimally invasive approach is a key step to efficiently identify resistance mechanisms and adjust to proper subsequent treatments. Materials and methodsIn the present study, we combined plasma-cfDNA and CTC analysis from 30 NSCLC patients in samples collected before treatment and at the progression of disease (PD). We detected molecular alterations at the DNA mutation (EGFR, PIK3CA, KRAS G12C, BRAF V600E), DNA methylation (RASSF1A, BRMS1, FOXA1, SLFN1, SHISA3, RARβ,, WIF-1, RASSF10 and APC), gene expression (CK-19, CK-18, CK-8, AXL, TWIST-1, PD-L1, PIM-1, Vimentin, ALDH-1, and B2M) and chromosomal level (HER2 and MET amplification) as possible resistance mechanisms and druggable targets. We also studied the expression of PD-L1 in single CTCs using immunofluorescence.ResultsIn some cases, T790M resistance EGFR mutation was detected at baseline in CTCs but not in the corresponding plasma cfDNA. PIK3CA mutations were detected only in plasma-cfDNA but not in corresponding CTCs. KRAS G12C and BRAF V600E mutations were not detected in the samples analyzed. MET amplification was detected in the CTCs of two patients before treatment whereas HER2 amplification was detected in the CTCs of three patients at baseline and in one patient at PD. DNA methylation analysis revealed low concordance between CTCs and cfDNA, indicating the complementary information obtained through parallel LB analysis. Results from gene expression analysis indicated high rates of vimentin-positive CTCs detected at all time points during osimertinib. Moreover, there was an increased number of NSCLC patients at PD harboring CTCs positive in PD-L1. AXL and PIM-1 expression detected in CTCs during treatment suggesting new possible therapeutic strategies. DiscussionOur results reveal that comprehensive liquid biopsy analysis can efficiently represent the heterogeneous molecular landscape and provide prominent information on subsequent treatments for NSCLC patients at PD since druggable molecular alterations were detected in CTCs.

Graph neural networks for multi-view learning: a taxonomic review

With the explosive growth of user-generated content, multi-view learning has become a rapidly growing direction in pattern recognition and data analysis areas. Due to the significant application value of multi-view learning, there has been a continuous emergence of research based on machine learning methods and traditional deep learning paradigms. The core challenge in multi-view learning lies in harnessing both consistent and complementary information to forge a unified, comprehensive representation. However, many multi-view learning tasks are based on graph-structured data, making existing methods unable to effectively mine the information contained in the input multiple data sources. Recently, graph neural networks (GNN) techniques have been widely utilized to deal with non-Euclidean data, such as graphs or manifolds. Thus, it is essential to combine the advantages of the powerful learning capability of GNN models and multi-view data. In this paper, we aim to provide a comprehensive survey of recent research works on GNN-based multi-view learning. In detail, we first provide a taxonomy of GNN-based multi-view learning methods according to the input form of models: multi-relation, multi-attribute and mixed. Then, we introduce the applications of multi-view learning, including recommendation systems, computer vision and so on. Moreover, several public datasets and open-source codes are introduced for implementation. Finally, we analyze the challenges of applying GNN models on various multi-view learning tasks and state new future directions in this field.

Multi-level feature fusion network for neuronal morphology classification

Neuronal morphology can be represented using various feature representations, such as hand-crafted morphometrics and deep features. These features are complementary to each other, contributing to improving performance. However, existing classification methods only utilize a single feature representation or simply concatenate different features without fully considering their complementarity. Therefore, their performance is limited and can be further improved. In this paper, we propose a multi-level feature fusion network that fully utilizes diverse feature representations and their complementarity to effectively describe neuronal morphology and improve performance. Specifically, we devise a Multi-Level Fusion Module (MLFM) and incorporate it into each feature extraction block. It can facilitate the interaction between different features and achieve effective feature fusion at multiple levels. The MLFM comprises a channel attention-based Feature Enhancement Module (FEM) and a cross-attention-based Feature Interaction Module (FIM). The FEM is used to enhance robust morphological feature presentations, while the FIM mines and propagates complementary information across different feature presentations. In this way, our feature fusion network ultimately yields a more distinctive neuronal morphology descriptor that can effectively characterize neurons than any singular morphological representation. Experimental results show that our method effectively depicts neuronal morphology and correctly classifies 10-type neurons on the NeuronMorpho-10 dataset with an accuracy of 95.18%, outperforming other approaches. Moreover, our method performs well on the NeuronMorpho-12 and NeuronMorpho-17 datasets and possesses good generalization.