Multimodal Dataset Research Articles

Dual-Pathway Deep Hashing-Based Adversarial Learning for Cross-Modal Retrieval

The application of deep hashing methods for cross-modal retrieval has seen growing interest due to their storage efficiency and fast query execution. However, the challenge posed by the “heterogeneity gap” in multi-modal datasets cannot be understated. To address this, we present a novel framework named Dual-Pathway Deep Hashing-Based Adversarial Learning (DP-DHAL), engineered to surmount this challenge. The architecture of DP-DHAL integrates three key components: (a) a dual-pathway representation learning module tasked with extracting modality-specific features; (b) an adversarial module working to align the distributions of cross-modal features; and (c) a deep hashing module responsible for generating hash codes that uphold the similarity relationships across different modalities. Additionally, we have developed a unique Hamming triplet-margin loss function to refine the assessment of content similarities. The DP-DHAL model is trained through an adversarial process where the adversarial module’s goal is to discern cross-modal features with the aim of reducing the heterogeneity gap. Simultaneously, the representation learning module is focused on producing representations that can both deceive the adversarial module and preserve cross-modal similarities to yield distinctive hash codes. Comprehensive experiments on varied datasets have shown that our proposed method outperforms other leading cross-modal hashing techniques.

An Overview of Behavioral Recognition

Human behavior recognition has become a popular research topic in the field of computer vision. With the introduction of deep learning and attention mechanisms, this field has been further promoted. However, issues such as dataset acquisition and preprocessing operations on multimodal datasets, modeling of long time information in videos, and fusion of temporal and spatial information still exist. In this paper, we first outline some video action recognition datasets and related preprocessing techniques, including frame extraction, optical flow extraction, and skeletal feature acquisition. Then, the relevant models are classified and parsed according to their characteristics and the types of input data modalities. In addition, we evaluate the performance of the models on several benchmark datasets to gain a deeper understanding of the model development process. Finally, we summarized the current challenges faced in the field of video behavior recognition, including model timeliness, data set subjectivity and effective fusion of multi-modal features, and proposed possible future improvement directions in order to provide more ideas and methods for subsequent research.

A multimodal fMRI dataset unifying naturalistic processes with a rich array of experimental tasks.

Cognitive neuroscience has advanced significantly due to the availability of openly shared datasets. Large sample sizes, large amounts of data per person, and diversity in tasks and data types are all desirable, but are difficult to achieve in a single dataset. Here, we present an open dataset with N = 101 participants and 6 hours of scanning per participant, with 6 multifaceted cognitive tasks including 2 hours of naturalistic movie viewing. This datasets' combination of ample sample size, extensive data per participant, more than 600 hours worth of data, and a wide range of experimental conditions - including cognitive, affective, social, and somatic/interoceptive tasks - positions it uniquely for probing important questions in cognitive neuroscience.

HumourHindiNet: Humour detection in Hindi web series using word embedding and convolutional neural network

Humour is a crucial aspect of human speech, and it is, therefore, imperative to create a system that can offer such detection. While data regarding humour in English speech is plentiful, the same cannot be said for a low-resource language like Hindi. Through this article, we introduce two multimodal datasets for humour detection in the Hindi web series. The dataset was collected from over 500 minutes of conversations amongst the characters of the Hindi web series Kota-Factory and Panchayat . Each dialogue is manually annotated as Humour or Non-Humour. Along with presenting a new Hindi language-based Humour detection dataset, we propose an improved framework for detecting humour in Hindi conversations. We start by preprocessing both datasets to obtain uniformity across the dialogues and datasets. The processed dialogues are then passed through the Skip-gram model for generating Hindi word embedding. The generated Hindi word embedding is then passed onto three convolutional neural network (CNN) architectures simultaneously, each having a different filter size for feature extraction. The extracted features are then passed through stacked Long Short-Term Memory (LSTM) layers for further processing and finally classifying the dialogues as Humour or Non-Humour. We conduct intensive experiments on both proposed Hindi datasets and evaluate several standard performance metrics. The performance of our proposed framework was also compared with several baselines and contemporary algorithms for Humour detection. The results demonstrate the effectiveness of our dataset to be used as a standard dataset for Humour detection in the Hindi web series. The proposed model yields an accuracy of 91.79 and 87.32 while an F1 score of 91.64 and 87.04 in percentage for the Kota-Factory and Panchayat datasets, respectively.

ROCOv2: Radiology Objects in COntext Version 2, an Updated Multimodal Image Dataset

Automated medical image analysis systems often require large amounts of training data with high quality labels, which are difficult and time consuming to generate. This paper introduces Radiology Object in COntext version 2 (ROCOv2), a multimodal dataset consisting of radiological images and associated medical concepts and captions extracted from the PMC Open Access subset. It is an updated version of the ROCO dataset published in 2018, and adds 35,705 new images added to PMC since 2018. It further provides manually curated concepts for imaging modalities with additional anatomical and directional concepts for X-rays. The dataset consists of 79,789 images and has been used, with minor modifications, in the concept detection and caption prediction tasks of ImageCLEFmedical Caption 2023. The dataset is suitable for training image annotation models based on image-caption pairs, or for multi-label image classification using Unified Medical Language System (UMLS) concepts provided with each image. In addition, it can serve for pre-training of medical domain models, and evaluation of deep learning models for multi-task learning.

Multimodal Data-Driven Intelligent Systems for Breast Cancer Prediction

Cancer, a malignant disease, results from abnormalities in the body cells that lead to uncontrolled growth and division, surpassing healthy growth and stability. In the case of breast cancer, this uncontrolled growth and division occurs in breast cells. Early identification of breast cancer is key to lowering mortality rates. Several new developments in artificial intelligence predictive models show promise for assisting decision-making. The primary goal of the proposed study is to build an efficient Breast Cancer Intelligent System using a multimodal dataset. The aim is to to establish Computer-Aided Diagnosis for breast cancer by integrating various data.This study uses the TCGA "The Cancer Genome Atlas Breast Invasive Carcinoma Collection" (TCGA-BRCA) dataset, which is part of an ongoing effort to create a community integrating cancer phenotypic and genotypic data. The TCGA- BRCA dataset includes: Clinical Data, RNASeq Gene Data, Mutation Data, and Methylation Data. Both clinical and genomic data are used in this study for breast cancer diagnosis. Integrating multiple data modalities enhances the robustness and precision of diagnostic and prognostic models in comparison with conventional techniques. The approach offers several advantages over unimodal models due to its ability to integrate diverse data sources. Additionally, these models can be employed to forecast the likelihood of a patient developing breast cancer in the near future, providing a valuable tool for early intervention and treatment planning.

CMR-net: A cross modality reconstruction network for multi-modality remote sensing classification.

In recent years, the classification and identification of surface materials on earth have emerged as fundamental yet challenging research topics in the fields of geoscience and remote sensing (RS). The classification of multi-modality RS data still poses certain challenges, despite the notable advancements achieved by deep learning technology in RS image classification. In this work, a deep learning architecture based on convolutional neural network (CNN) is proposed for the classification of multimodal RS image data. The network structure introduces a cross modality reconstruction (CMR) module in the multi-modality feature fusion stage, called CMR-Net. In other words, CMR-Net is based on CNN network structure. In the feature fusion stage, a plug-and-play module for cross-modal fusion reconstruction is designed to compactly integrate features extracted from multiple modalities of remote sensing data, enabling effective information exchange and feature integration. In addition, to validate the proposed scheme, extensive experiments were conducted on two multi-modality RS datasets, namely the Houston2013 dataset consisting of hyperspectral (HS) and light detection and ranging (LiDAR) data, as well as the Berlin dataset comprising HS and synthetic aperture radar (SAR) data. The results demonstrate the effectiveness and superiority of our proposed CMR-Net compared to several state-of-the-art methods for multi-modality RS data classification.

Two-stage zero-shot sparse hashing with missing labels for cross-modal retrieval

Recently, zero-shot cross-modal hashing has gained significant popularity due to its ability to effectively realize the retrieval of emerging concepts within multimedia data. Although the existing approaches have shown impressive results, the following limitations still need to be solved: (1) Labels in large-scale multimodal datasets in real scenes are usually incomplete or partially missing. (2) The existing methods ignore the influence of features-wise low-level similarity and label distribution on retrieval performance. (3) The representation ability of dense hash codes limits its discriminative potential. To solve these issues, we introduce an effective cross-modal retrieval framework called two-stage zero-shot sparse hashing with missing labels (TZSHML). Specifically, we learn a classifier through the partially known labeled samples to predict the labels of unlabeled data. Then, we use the reliable information in the correctly marked labels to recover the missing labels. The predicted and recovered labels are combined to obtain more accurate labels for the samples with missing labels. In addition, we employ sample-wise fine-grained similarity and cluster-wise similarity to learn hash codes. Therefore, TZSHML ensures that more samples with similar semantics are clustered together. Besides, we apply high-dimensional sparse hash codes to explore richer semantic information. Finally, the drift and interaction terms are introduced into the learning of the hash function to further narrow the gap between different modalities. Extensive experimental results demonstrate the competitiveness of our approach over other state-of-the-art methods in zero-shot retrieval scenarios with missing labels. The source code of this paper can be obtained from https://github.com/szq0816/TZSHML.

Processing and Integration of Multimodal Image Data Supporting the Detection of Behaviors Related to Reduced Concentration Level of Motor Vehicle Users

This paper introduces a comprehensive framework for the detection of behaviors indicative of reduced concentration levels among motor vehicle operators, leveraging multimodal image data. By integrating dedicated deep learning models, our approach systematically analyzes RGB images, depth maps, and thermal imagery to identify driver drowsiness and distraction signs. Our novel contribution includes utilizing state-of-the-art convolutional neural networks (CNNs) and bidirectional long short-term memory (Bi-LSTM) networks for effective feature extraction and classification across diverse distraction scenarios. Additionally, we explore various data fusion techniques, demonstrating their impact on improving detection accuracy. The significance of this work lies in its potential to enhance road safety by providing more reliable and efficient tools for the real-time monitoring of driver attentiveness, thereby reducing the risk of accidents caused by distraction and fatigue. The proposed methods are thoroughly evaluated using a multimodal benchmark dataset, with results showing their substantial capabilities leading to the development of safety-enhancing technologies for vehicular environments. The primary challenge addressed in this study is the detection of driver states not relying on the lighting conditions. Our solution employs multimodal data integration, encompassing RGB, thermal, and depth images, to ensure robust and accurate monitoring regardless of external lighting variations

HARWE: A multi-modal large-scale dataset for context-aware human activity recognition in smart working environments

In recent years, deep neural networks (DNNs) have provided high performances for various tasks, such as human activity recognition (HAR), in view of their end-to-end training process between the input data and output labels. However, the performances of the DNNs are highly dependent on the availability of large-scale data for their training processes. In this paper, we propose a novel dataset for the task of HAR, in which the labels are specified for the working environments (WE). Our proposed dataset, namely HARWE, considers multiple signal modalities, including visual signal, audio signal, inertial sensor signals, and biological signals, that are acquired using four different electronic devices. Furthermore, our HARWE dataset is acquired from a large number of participants while considering the realistic disturbances that can occur in the wild. Our HARWE data is context-driven, which means there exist a number of labels in it that even though they are correlated with each other, they have contextual differences. A deep conventional multi-modal neural network provides an accuracy of 99.06% and 68.60%, for the cases of the easy and difficult settings of our dataset, respectively, which indicates its applicability for the task of human activity recognition.

VIFNet: An end-to-end visible–infrared fusion network for image dehazing

Image dehazing poses significant challenges in environmental perception. Recent research mainly focus on deep learning-based methods with single modality, while they may result in severe information loss especially in dense-haze scenarios. The infrared image exhibits robustness to the haze, however, existing methods have primarily treated the infrared modality as auxiliary information, failing to fully explore its rich information in dehazing. To address this challenge, the key insight of this study is to design a visible–infrared fusion network for image dehazing. In particular, we propose a multi-scale Deep Structure Feature Extraction (DSFE) module, which incorporates the Channel-Pixel Attention Block (CPAB) to restore more spatial and marginal information within the deep structural features. Additionally, we introduce an inconsistency weighted fusion strategy to merge the two modalities by leveraging the more reliable information. To validate this, we construct a visible–infrared multimodal dataset called AirSim-VID based on the AirSim simulation platform. Extensive experiments performed on challenging real and simulated image datasets demonstrate that VIFNet can outperform many state-of-the-art competing methods. The code and dataset are available at https://github.com/mengyu212/VIFNet_dehazing.

TDC-2: Multimodal Foundation for Therapeutic Science.

Therapeutics Data Commons (tdcommons.ai) is an open science initiative with unified datasets, AI models, and benchmarks to support research across therapeutic modalities and drug discovery and development stages. The Commons 2.0 (TDC-2) is a comprehensive overhaul of Therapeutic Data Commons to catalyze research in multimodal models for drug discovery by unifying single-cell biology of diseases, biochemistry of molecules, and effects of drugs through multimodal datasets, AI-powered API endpoints, new multimodal tasks and model frameworks, and comprehensive benchmarks. TDC-2 introduces over 1,000 multimodal datasets spanning approximately 85 million cells, pre-calculated embeddings from 5 state-of-the-art single-cell models, and a biomedical knowledge graph. TDC-2 drastically expands the coverage of ML tasks across therapeutic pipelines and 10+ new modalities, spanning but not limited to single-cell gene expression data, clinical trial data, peptide sequence data, peptidomimetics protein-peptide interaction data regarding newly discovered ligands derived from AS-MS spectroscopy, novel 3D structural data for proteins, and cell-type-specific protein-protein interaction networks at single-cell resolution. TDC-2 introduces multimodal data access under an API-first design using the model-view-controller paradigm. TDC-2 introduces 7 novel ML tasks with fine-grained biological contexts: contextualized drug-target identification, single-cell chemical/genetic perturbation response prediction, protein-peptide binding affinity prediction task, and clinical trial outcome prediction task, which introduce antigen-processing-pathway-specific, cell-type-specific, peptide-specific, and patient-specific biological contexts. TDC-2 also releases benchmarks evaluating 15+ state-of-the-art models across 5+ new learning tasks evaluating models on diverse biological contexts and sampling approaches. Among these, TDC-2 provides the first benchmark for context-specific learning. TDC-2, to our knowledge, is also the first to introduce a protein-peptide binding interaction benchmark.

A multimodal dataset for investigating working memory in presence of music: a pilot study.

Decoding an individual's hidden brain states in responses to musical stimuli under various cognitive loads can unleash the potential of developing a non-invasive closed-loop brain-machine interface (CLBMI). To perform a pilot study and investigate the brain response in the context of CLBMI, we collect multimodal physiological signals and behavioral data within the working memory experiment in the presence of personalized musical stimuli. Participants perform a working memory experiment called the n-back task in the presence of calming music and exciting music. Utilizing the skin conductance signal and behavioral data, we decode the brain's cognitive arousal and performance states, respectively. We determine the association of oxygenated hemoglobin (HbO) data with performance state. Furthermore, we evaluate the total hemoglobin (HbT) signal energy over each music session. A relatively low arousal variation was observed with respect to task difficulty, while the arousal baseline changes considerably with respect to the type of music. Overall, the performance index is enhanced within the exciting session. The highest positive correlation between the HbO concentration and performance was observed within the higher cognitive loads (3-back task) for all of the participants. Also, the HbT signal energy peak occurs within the exciting session. Findings may underline the potential of using music as an intervention to regulate the brain cognitive states. Additionally, the experiment provides a diverse array of data encompassing multiple physiological signals that can be used in the brain state decoder paradigm to shed light on the human-in-the-loop experiments and understand the network-level mechanisms of auditory stimulation.

StereoThermoLegs: label propagation with multimodal stereo cameras for automated annotation of posterior legs during running at different velocities

AbstractIn sports science, thermal imaging is applied to investigate various questions related to exercise-induced stress response, muscle fatigue, anomalies, and diseases. Infrared thermography monitors thermal radiation from the skin’s surface over time. For further analysis, regions of interest are extracted and statistically analyzed. Although computer vision algorithms have grown in recent years due to data-driven approaches, this is not the case for detailed segmentation in thermal images. In a supervised manner, machine learning optimizations require a large amount of training data with input and ground truth output data. Unfortunately, obtaining annotated data are a costly problem that increases with the complexity of the task. For semantic segmentation, pixel-wise label masks must be created by experts. Few datasets meet the needs of sports scientists and physicians to perform advanced applications of thermal computer vision during physical activity and generate new insights in their fields. In this paper, a new method is introduced to transfer segmentation masks from the vision domain to the thermal domain with a stereo-calibrated time-of-flight camera and high-resolution mid-wave infrared camera. A post-processing procedure is then utilized to obtain dense pixel masks for the posterior legs during walking and running on a treadmill. The developed StereoThermoLegs dataset is based on 14 participants and includes 11 subjects for training with 12,826 thermograms and the remaining three individuals for testing with 3433 images. A deep neural network was trained with the DeepLabv3+ architecture, the AdaBelief optimizer, and Dice loss as a benchmark. After 29 epochs, the test set achieved an average intersection over union of 0.66. The analysis of the posterior leg region, specifically the left and right calf, offered the most insights, with values of 0.83 and 0.83, respectively. The first multimodal stereo dataset containing synchronized visual and thermal images of a runner’s back provides a starting point for data-driven segmentation tasks in sports science and medicine. Our technique allows for automatic production of customized datasets for deep learning, accelerating the implementation of baseline outcomes for newly identified areas of interest in thermal imaging, while bypassing the requirement for extensive manual annotation. The approach is not exclusive to stereo rig and segmentation tasks utilizing RGBD and thermal cameras, but can be applied to other imaging tasks and modalities.

PATH-06. AN INTEGRATIVE MULTI-MODAL DIAGNOSTIC APPROACH TO PEDIATRIC CHOROID PLEXUS TUMORS

Abstract BACKGROUND Choroid plexus tumors (CPTs) consist of a diverse group of neoplasms, more prevalent in children. They include choroid plexus papillomas (CPPs, grade 1), atypical choroid plexus papillomas (aCPPs, grade 2), and choroid plexus carcinomas (CPCs, grade 3). Traditional grading of CPTs relies on histologic assessment, but morphologic criteria for determining tumor grade are subjective. Evidence suggests incorporation of clinical, radiologic, and molecular characteristics may increase diagnostic accuracy. Here, we investigated the multi-modal associations of CPT subtypes to develop a classification system to aid in CPT subtype diagnostics. METHODS We aggregated an integrative dataset that consists of digitized whole-slide images, NGS sequencing panels, methylation arrays, and magnetic resonance spectroscopy (MRS) spectra from in-house and publicly available data sources (n=110). We tested for clinically significant associations and established multi-modal profiles of CPT subtypes, as well as a machine-learning classifier to predict multi-modal based CPT subtypes. RESULTS CPT subtypes exhibit distinct characteristics across modalities with consistent correlations across anatomy, molecular, and metabolic datasets. Computer-vision models achieved high accuracy (89.7%) in classifying CPT subtypes using H&E images. Ki-67 immunostaining displayed increased reactivity in CPC compared to CPP and aCPP (p = 0.007, ANOVA). Methylation-based classifications confidently predicted CPP and CPC subtypes, while aCPP tumors clustered with either CPP or CPC due to the lack of a defined subtype. Molecular variants (i.e. TP53, MYCN) were significantly associated with CPC (p = 0.007, ANOVA). Metabolites such as inositol (p = 0.031) and choline (p = 0.004) were associated with CPT grade based on MRS-spectra. CONCLUSION Multi-modal analysis reliably separates CPTs by grade. The study underscores the significance of integrating multi-modal datasets in refining the accuracy of diagnosis and prognosis of CPTs. Continued research will focus on a multi-modal approach to improve prediction of aggressive behavior and understanding of inherent metabolic heterogeneity of CPT subtypes.

Cross-modal and multimodal data analysis based on functional mapping of spectral descriptors and manifold regularization

Multimodal manifold modeling methods extend spectral geometry-aware data analysis to incorporate learning from multiple related and complementary modalities. However, most methods assume an equal number of homogeneous data samples in each modality and partial correspondences between modalities as prior knowledge. This work introduces two new multimodal modeling methods. The first method introduces a comprehensive framework for handling multimodal information in heterogeneous data without requiring specific prior knowledge. To achieve this, we begin by extracting local descriptors using spectral graph wavelet signatures (SGWS) to identify manifold localities. Then, we propose a manifold regularization framework that incorporates functional mapping between SGWS descriptors (FMBSD) to determine pointwise correspondences. The second method involves manifold regularized multimodal classification based on pointwise correspondences (M2CPC). This method addresses the challenge of multiclass classification in multimodal heterogeneous data by determining correspondences between modalities using the FMBSD method. Experimental results evaluating the FMBSD method on three widely used cross-modal retrieval datasets and evaluating the M2CPC method on three benchmark multimodal multiclass classification datasets demonstrate their effectiveness and superiority over state-of-the-art methods.

Predicting conversion from mild cognitive impairment to Alzheimer's disease: a multimodal approach.

Successively predicting whether mild cognitive impairment patients will progress to Alzheimer's disease is of significant clinical relevance. This ability may provide information that can be leveraged by emerging intervention approaches and thus mitigate some of the negative effects of the disease. Neuroimaging biomarkers have gained some attention in recent years and may be useful in predicting the conversion of mild cognitive impairment to Alzheimer's disease. We implemented a novel multi-modal approach that allowed us to evaluate the potential of different imaging modalities, both alone and in different degrees of combinations, in predicting the conversion to Alzheimer's disease of mild cognitive impairment patients. We applied this approach to the imaging data from the Alzheimer's Disease Neuroimaging Initiative that is a multi-modal imaging dataset comprised of MRI, Fluorodeoxyglucose PET, Florbetapir PET and diffusion tensor imaging. We included a total of 480 mild cognitive impairment patients that were split into two groups: converted and stable. Imaging data were segmented into atlas-based regions of interest, from which relevant features were extracted for the different imaging modalities and used to construct machine-learning models to classify mild cognitive impairment patients into converted or stable, using each of the different imaging modalities independently. The models were then combined, using a simple weight fusion ensemble strategy, to evaluate the complementarity of different imaging modalities and their contribution to the prediction accuracy of the models. The single-modality findings revealed that the model, utilizing features extracted from Florbetapir PET, demonstrated the highest performance with a balanced accuracy of 83.51%. Concerning multi-modality models, not all combinations enhanced mild cognitive impairment conversion prediction. Notably, the combination of MRI with Fluorodeoxyglucose PET emerged as the most promising, exhibiting an overall improvement in predictive capabilities, achieving a balanced accuracy of 78.43%. This indicates synergy and complementarity between the two imaging modalities in predicting mild cognitive impairment conversion. These findings suggest that β-amyloid accumulation provides robust predictive capabilities, while the combination of multiple imaging modalities has the potential to surpass certain single-modality approaches. Exploring modality-specific biomarkers, we identified the brainstem as a sensitive biomarker for both MRI and Fluorodeoxyglucose PET modalities, implicating its involvement in early Alzheimer's pathology. Notably, the corpus callosum and adjacent cortical regions emerged as potential biomarkers, warranting further study into their role in the early stages of Alzheimer's disease.

DELTA: Integrating Multimodal Sensing with Micromobility for Enhanced Sidewalk and Pedestrian Route Understanding.

Urban environments are undergoing significant transformations, with pedestrian areas emerging as complex hubs of diverse mobility modes. This shift demands a more nuanced approach to urban planning and navigation technologies, highlighting the limitations of traditional, road-centric datasets in capturing the detailed dynamics of pedestrian spaces. In response, we introduce the DELTA dataset, designed to improve the analysis and mapping of pedestrian zones, thereby filling the critical need for sidewalk-centric multimodal datasets. The DELTA dataset was collected in a single urban setting using a custom-designed modular multi-sensing e-scooter platform encompassing high-resolution and synchronized audio, visual, LiDAR, and GNSS/IMU data. This assembly provides a detailed, contextually varied view of urban pedestrian environments. We developed three distinct pedestrian route segmentation models for various sensors-the 4K camera, stereocamera, and LiDAR-each optimized to capitalize on the unique strengths and characteristics of the respective sensor. These models have demonstrated strong performance, with Mean Intersection over Union (IoU) values of 0.84 for the reflectivity channel, 0.96 for the 4K camera, and 0.92 for the stereocamera, underscoring their effectiveness in ensuring precise pedestrian route identification across different resolutions and sensor types. Further, we explored audio event-based classification to connect unique soundscapes with specific geolocations, enriching the spatial understanding of urban environments by associating distinctive auditory signatures with their precise geographical origins. We also discuss potential use cases for the DELTA dataset and the limitations and future possibilities of our research, aiming to expand our understanding of pedestrian environments.

Functional connectome through the human life span.

The lifespan growth of the functional connectome remains unknown. Here, we assemble task-free functional and structural magnetic resonance imaging data from 33,250 individuals aged 32 postmenstrual weeks to 80 years from 132 global sites. We report critical inflection points in the nonlinear growth curves of the global mean and variance of the connectome, peaking in the late fourth and late third decades of life, respectively. After constructing a fine-grained, lifespan-wide suite of system-level brain atlases, we show distinct maturation timelines for functional segregation within different systems. Lifespan growth of regional connectivity is organized along a primary-to-association cortical axis. These connectome-based normative models reveal substantial individual heterogeneities in functional brain networks in patients with autism spectrum disorder, major depressive disorder, and Alzheimer's disease. These findings elucidate the lifespan evolution of the functional connectome and can serve as a normative reference for quantifying individual variation in development, aging, and neuropsychiatric disorders.

Prognostic prediction of ovarian cancer based on hierarchical sampling & fine-grained recognition convolution neural network

Ovarian cancer ranks among the deadliest gynecological malignancies, with high-grade serous adenocarcinoma (HGSA) constituting 75 % of ovarian cancer cases and accounting for 80 %–90 % of ovarian cancer-related fatalities. Accurate prognosis prediction for ovarian HGSA is of critical clinical significance. However, existing prognostic analysis methods exhibit suboptimal performance in identifying prognostically relevant pathological markers, making it challenging, even for experienced pathologists, to forecast the prognosis of ovarian HGSA patients accurately. Deep learning holds promise in enhancing prognostic prediction accuracy. However, a significant challenge in this field arises from the impracticality of directly inputting histopathology whole-slide images with millions to billions of pixels into existing deep learning networks for training and inference. To address this issue, we propose a prognostic analysis network for ovarian cancer based on hierarchical sampling and fine-grained recognition. This network comprises a two-stage hierarchical sampling sub-network, a fine-grained image recognition sub-network, and a prognostic analysis sub-network. We assess the system's performance using a pathological dataset of 450 cases of ovarian HGSA diagnosed and treated at the Pathology Department of the West China Second University Hospital of Sichuan University. Results indicate that: (1) The proposed prognostic analysis network based on two-stage hierarchical sampling sub-network can effectively analyze the histopathology whole-slide image; (2) the use of fine-grained image recognition and the introduction of clinical information can improve the performance of HGSA prognosis analysis method, with an improvement range of 4.77–10.66%; and (3) the proposed method can be used for the model analysis of pathological datasets of HGSA. Moreover, this method can be used to explore effective characteristics from the multi-modal dataset through automatic learning with prediction recall, accuracy, and precision rates of 80.0%, 81.1%, and 81.8%, respectively, underscoring its clinical potential. This study reveals the reliability and effectiveness of the proposed prognosis evaluation method of HGCA. Conclusions can help clinicians precisely evaluate the recurrence risk of patients, take the initiative to master the diagnosis and treatment, and increase the long-term survival rate of patients.