Modal Features Research Articles

An Enhanced Cross‐Attention Based Multimodal Model for Depression Detection

ABSTRACTDepression, a prevalent mental disorder in modern society, significantly impacts people's daily lives. Recently, there have been advancements in developing automated diagnosis models for detecting depression. However, data scarcity, primarily due to privacy concerns, has posed a challenge. Traditional speech features have limitations in representing knowledge for depression diagnosis, and the complexity of deep learning algorithms necessitates substantial data support. Furthermore, existing multimodal methods based on neural networks overlook the heterogeneity gap between different modalities, potentially resulting in redundant information. To address these issues, we propose a multimodal depression detection model based on the Enhanced Cross‐Attention (ECA) Mechanism. This model effectively explores text‐speech interactions while considering modality heterogeneity. Data scarcity has been mitigated by fine‐tuning pre‐trained models. Additionally, we design a modal fusion module based on ECA, which emphasizes similarity responses and updates the weight of each modal feature based on the similarity information between modal features. Furthermore, for speech feature extraction, we have reduced the computational complexity of the model by integrating a multi‐window self‐attention mechanism with the Fourier transform. The proposed model is evaluated on the public dataset, DAIC‐WOZ, achieving an accuracy of 80.0% and an average F1 value improvement of 4.3% compared with relevant methods.

A novel adaptive lightweight multimodal efficient feature inference network ALME-FIN for EEG emotion recognition.

Enhancing the accuracy of emotion recognition models through multimodal learning is a common approach. However, challenges such as insufficient modal feature learning in multimodal inference and scarcity of sample data continue to pose obstacles that need to be overcome. Therefore, we propose a novel adaptive lightweight multimodal efficient feature inference network (ALME-FIN). We introduce a time-domain lightweight adaptive network (TDLAN) and a two-dimensional dynamic focusing network (TDDFN) for multimodal feature learning. The TDLAN incorporates the denoising process as an integral part of network training, achieving adaptive denoising for each sample through the continuous optimization of the trainable filtering threshold. Simultaneously, it incorporates an interactive convolutional sampling module, enabling lightweight multi-scale feature extraction in the time domain. TDDFN effectively extracts core image features while filtering out redundancies. During the training process, the Multi-network dynamic gradient adjustment framework (MDGAF) dynamically monitors the feature learning efficacy across different modalities. It timely adjusts the training gradients of networks to allocate additional optimization time for under-optimized modalities, thereby maximizing the utilization of multimodal feature information. Moreover, the introduction of a Multi-class relationship interaction module prior to the classifier aids the model in clearly understanding the relationships among different category samples. This approach enables the model to achieve relatively accurate emotion recognition even in scenarios of limited sample availability. Compared to existing multimodal learning techniques, ALME-FIN exhibits a more efficient multimodal feature inference method that can achieve satisfactory emotional recognition performance even with a limited number of samples.

Research on Modelling Capability of English Multimodal File Search based on Transformer

With the exponential growth of file data in the multimedia era, file retrieval ability to achieve effective data management has become a hot research field. Based on people’s English file search needs, this paper proposes an English multimodal file search model based on transformer. Through ablation experiments on two public data sets and comparison experiments with the benchmark model, the effectiveness and superiority of the proposed transformers algorithm model in multi-modal data processing are verified. The multi-modal fusion retrieval system can usually achieve better performance than the single-modal retrieval system. This experiment focuses on three modes: Audio, Image and Text. The experimental results show that the proposed method can not only improve the efficiency of file search, but also extract modal features and perform feature fusion better. In the future, we can further explore different types of other attention mechanisms or integrate a variety of different architectures to further enhance the feasibility and superiority of multimodal file search

Efficient face anti-spoofing identifier network (FASIN) with depth and near infrared deep learning methods

Face anti-spoofing (FAS) is a crucial task in the field of face recognition practices, which aims to detect and prevent attempts to spoof/attack a facial recognition system using fake or manipulated images. In this work, we aimed to develop a novel Face Anti-Spoofing Identifier Network (FASIN) with Depth and Near Infrared (NIR) embeddings, trained on multi modalities using multi stream Convolutional Neural Networks (CNNs). The proposed FASIN model is capable of processing RGB, Depth, and NIR images to extract discriminative features and effectively distinguish between genuine and fake faces. The depth maps and the near infrared (NIR) images are acquired from RGB images by constructing the depth map construction network (DMCN) and near infrared construction network (NIRCN) respectively. The FASIN model comprises three sub-networks: one for processing RGB images, a second for processing depth images, and the other sub-network for processing NIR images. All the sub-networks consist of multiple CNN layers, which extract features at different scales and levels of abstraction. The inherent noise and other variables may reduce the efficacy of CNN, the wavelet spatial attention mechanism has been proposed to support the RGB CNN stream and it is named wavelet attention CNN (WA-CNN). The extracted features are then concatenated using a multi modal feature fusion module to obtain a robust feature representation that is used to classify real and fake faces. An ensemble learning mechanism has been attached to the model to learn the concatenated features effectively. Experimental results obtained on four benchmark datasets (namely, CelebA-Spoof, CASIA-SURF, WMCA, and MSU-MFSD) demonstrate the efficacy of the proposed FASIN model collated with the state-of-the-art methods. The proposed FASIN model achieves high accuracy and low average classification error rates (ACER), indicating its potential for real-world applications in face anti spoofing identification systems.

The Importance of Treatment Modality in Veterans with Opioid Use Disorder: Implications for Virtual Care.

This study examined treatment utilization across in-person and virtual treatment modalities in veterans who were on medications for opioid use disorder (MOUD; N = 139). Treatment records for veterans in addiction treatment on MOUD were examined for 3-months prior to telehealth conversions ("Pre-Telehealth," 12/02/2019-03/14/2020), 3-months during the initial telehealth transition ("Telehealth," 03/15/2020-06/30/2020) and 3-months during post-telehealth transition ("Re-Entry," 07/01/2020-10/01/2020). Analyses examined the relationship between treatment modality and demographic features, psychiatric comorbidities, treatment engagement, and illness severity as measured by psychiatric emergency room (PER) utilization. Results demonstrated that modality was not associated with PER utilization. Past-year PER visits, alcohol use disorder (AUD), and psychotic disorders were associated with PER utilization during Telehealth, while AUD was associated with Re-Entry PER utilization. Given the likelihood of virtual treatment in the future, frequent in-person visits may not be necessary for MOUD; however, individuals with comorbid AUD and psychotic disorders may need additional support to prevent emergency care.

Damage identification based on the inner product matrix and parallel convolution neural network for frame structure

Structural health monitoring based on vibration signal analysis has been extensively employed for damage identification. Mainstream machine learning techniques, such as convolutional neural networks (CNN), often rely on single-domain inputs, which may provide limited information for accurate damage identification. To overcome this limitation, this study proposes a novel approach that combines an inner product matrix (IPM) with a parallel CNN (IPM-PCNN) to extract multidimensional features for detecting structural damage in a steel frame structure. The proposed IPM-PCNN framework consists of a one-dimensional (1D) CNN branch for processing time series data, a two-dimensional (2D) CNN branch for handling structural modal data, and several fully connected layers. This unique combination leverages the strengths of both 1D and 2D CNNs to capture temporal and modal features of the signal effectively. To validate the effectiveness and superiority of the proposed method, a five-story steel frame model is used as the research object, and five comparative methods are evaluated under the same experimental conditions. The results demonstrate that the IPM-PCNN model can automatically extract relevant features from the signals to accurately identify structural damage, achieving an accuracy of 96.60% on the test set, outperforming machine learning methods in performance. Furthermore, the internal inference processes of these methods are explored and visualized to provide insights into their decision-making mechanisms.

Multimodal sentiment analysis based on multi‐stage graph fusion networks under random missing modality conditions

AbstractThe primary challenge of the multimodal sentiment analysis (MSA) task is the modal fusion, and the lack of modalities may exist in the fusion process, which leads to poor prediction results. Most of the previous research on multimodal fusion is single‐stage fusion, disregarding how various modality subsets interact, as well as rarely considering relative position relationship of modality sequences, causing the fragmentation of context info. Considering the aforementioned issues, this study introduces an MSA method based on the multi‐stage graph fusion network (MSGFN) under random missing modality conditions to improve the robustness of the model to MSA under the random missing modality conditions. Firstly, for each modality, its inter‐modal and intra‐modal multi‐head attention are used to learn robust representation of the modality sequence. Meanwhile, the relative position encoding (RPE) is introduced into mechanism of attention that enables model to perceive and learn the relative position before and after the modality sequence when calculating attention, thereby better understanding the contextual info of the sequence. After that, the transformer encoder receives the learned modality features and uses the pre‐trained model to supervise the reconstruction of the missing modality information. Finally, the feature representations of different modalities have effectively fused using multi‐stage graph fusion network, and the output is used for the ultimate sentiment classification. Wide experiments are conducted on two publicly available datasets, CMU‐MOSI and IEMOCAP, and the findings indicate that the proposed method can better handle the challenges caused by modality fusion and modality missing compared with several baseline methods.

Analyzing Arabic and English Media Coverage of Attempt Assassination of Donald Trump

This study is concerned with how Arabic and English media report a security incident by former President Trump through Halliday’s Systemic Functional Linguistics Halliday analysis in lexical features of mood and modality. It demonstrates differences in the cultivation and language of the narrative between countries and how these variations shape perceptions of the event. Through declarative and modal expressions, the study advances insights into how these media contexts avail authority, urgency, and control, influencing audience perceptions. This research adds to the study of media and cross-cultural communication, showcasing the extent to which language choices embody cultural and social values, even in politically high-stakes situations.

Bayesian-based multi-class damage identification on prestressed reinforced concrete bridges

Model-driven structural system identification techniques are effective for comprehensive damage assessment but relying on a single model for structural identification (St-Id) can lead to unreliable results due to the ill-posed nature of the inverse calibration problem. To address this issue, this work explores the potential of multi-class digital twins, derived from sets of competing model classes, each one characterised by a distinct failure mechanism. To accelerate the calibration process, Kriging metamodels are employed, facilitating the fusion of diverse data types, such as modal displacements, frequencies, and static rotations. The Bayesian Information Criterion (BIC) is utilised to identify the most suitable model class. This research specifically targets the damage assessment of prestressed reinforced concrete girder bridges, which represent a significant portion of the global bridge stock. To this end, a numerical model of a representative span is developed, followed by a parametric analysis to evaluate the proposed approach’s effectiveness. Various damage scenarios including stiffness reduction and tendon prestress losses are considered, demonstrating the method’s capability for quasi-real-time multi-class damage identification. The study emphasizes the combined use of modal features and static rotations within a Bayesian framework, underscoring the importance of data fusion in achieving effective damage identification for these structures.

Ambient vibration dataset of a short-span cable-stayed bridge

A full-scale short-span cable-stayed bridge on a wind-exposed hill in New South Wales in Australia, was instrumented to assess its dynamic response to ambient vibration. The primary goal was to generate extensive datasets for Operational Modal Analysis (OMA) and develop new Structural Health Monitoring (SHM) techniques. Wind, vehicular, and pedestrian traffic on the bridge, and highway traffic beneath it, provided sufficient ambient vibration excitation. Uni-axial accelerometers were installed on the deck and cables, and a shear strain sensor was placed at one end of the bridge to measure traffic volume. Continuous ambient temperature recording was included due to its known impact on structural modal features. This data note presents four comprehensive datasets: ‘A-Single-Day,’ ‘Seasonal,’ ‘Constant-Temperature,’ and ‘Damaged-Structure’ which correspond to different environmental, operational, and structural conditions. These datasets are intended to support researchers by providing high-quality, real-world data that can be used for validating and developing OMA frameworks and SHM techniques. This paper fills a critical gap in the literature by making these detailed datasets available for broader research applications. Detailed descriptions of the instrumentation and data collection procedures are provided, along with discussions on the validity and plausibility of the datasets.

Research on bearing fault diagnosis based on a multimodal method.

As an essential component of mechanical systems, bearing fault diagnosis is crucial to ensure the safe operation of the equipment. However, vibration data from bearings often exhibit non-stationary and nonlinear features, which complicates fault diagnosis. To address this challenge, this paper introduces a novel multi-scale time-frequency and statistical features fusion model (MTSF-FM). Specifically, the method first employs continuous wavelet transform to generate time-frequency images, capturing local and global features of the signal at different scales. Contrast enhancement techniques are then used to improve the visual quality of these images. Next, features are extracted from the time-frequency images using a visual geometry group network to obtain deep features of image modalities. In parallel, 13 key features are extracted from the original vibration data in the time-frequency domain. Convolutional neural networks are then employed for deep feature extraction. Experimental results demonstrate that MTSF-FM achieves accuracies of 98.5% and 95.1% on two public datasets. These findings highlight the effectiveness of MTSF-FM in analyzing complex vibration data and propose a novel method for bearing fault diagnosis.

Conti-Fuse: A novel continuous decomposition-based fusion framework for infrared and visible images

For better explore the relations of inter-modal and inner-modal, even in deep learning fusion framework, the concept of decomposition plays a crucial role. However, the previous decomposition strategies (base & detail or low-frequency & high-frequency) are too rough to present the common features and the unique features of source modalities, which leads to a decline in the quality of the fused images. The existing strategies treat these relations as a binary system, which may not be suitable for the complex generation task (e.g. image fusion). To address this issue, a continuous decomposition-based fusion framework (Conti-Fuse) is proposed. Conti-Fuse treats the decomposition results as few samples along the feature variation trajectory of the source images, extending this concept to a more general state to achieve continuous decomposition. This novel continuous decomposition strategy enhances the representation of complementary information of inter-modal by increasing the number of decomposition samples, thus reducing the loss of critical information. To facilitate this process, the continuous decomposition module (CDM) is introduced to decompose the input into a series continuous components. The core module of CDM, State Transformer (ST), is utilized to efficiently capture the complementary information from source modalities. Furthermore, a novel decomposition loss function is also designed which ensures the smooth progression of the decomposition process while maintaining linear growth in time complexity with respect to the number of decomposition samples. Extensive experiments demonstrate that our proposed Conti-Fuse achieves superior performance compared to the state-of-the-art fusion methods.

CMFX: Cross-modal fusion network for RGB-X crowd counting.

Currently, for obtaining more accurate counts, existing methods primarily utilize RGB images combined with features of complementary modality (X-modality) for counting. However, designing a model that can adapt to various sensors is still an unsolved issue due to the differences in features between different modalities. Therefore, this paper proposes a unified fusion framework called CMFX for RGB-X crowd counting. CMFX contains three core components: fast feature aggregation module (FFAM), cross-modal feature interaction module (CFIM), and cross-modal feature decoding module (CFDM). Specifically, FFAM aims to enhance the fusion representation capability of low-level multimodal features through lightweight mixed attention. CFIM can fully realize the interaction and fusion of high-level features by rectifying the feature information of two modalities and exploring their potential correlations. In addition, CFDM employs a novel graph convolution block for refining and preserving cross-modal features at high-level and low-level. To validate CMFX, this paper unifies, for the first time, two modalities complementary to RGB images, namely depth and thermal. After extensive experiments on three public datasets RGBT-CC, DroneRGBT, and ShanghaiTechRGBD, we found that CFMX performs excellently in these two multimodal combinations. Code: https://github.com/Duanxm9/CMFX.

Multimodal sentiment analysis based on multiple attention

The development of the Internet makes various types of data widely appear on various social platforms, multimodal data provides a new perspective for sentiment analysis. Although the data types are different, there are information expressing the same sentiment. The existing researches on extracting those information are static, and this means that there is a problem of extracting common information in a fixed amount. Therefore, to address this problem, we proposes a method named multimodal sentiment analysis based on multiple attention(MAMSA). Firstly, this method utilized the adaptive attention interaction module to dynamically determine the amount of information contributed by text and image features in multimodal fusion, and multimodal common representations are extracted through cross modal attention to improve the performance of each modal feature representation. Secondly, using sentiment information as a guide to extract text and image features related to sentiment. Finally, using hierarchical manner to fully learning the internal correlations between sentiment-text association representation, sentiment-image association representation, and multimodal common information to improve the performance of the model. We conducted extensive experiments using two public multimodal datasets, and the experimental results validated the availability of the proposed method.

Cyclic Evolution of Synergized Spin and Orbital Angular Momenta.

Spin angular momentum (SAM) and orbital angular momentum (OAM)are fundamental physical characteristics described by polarization and spatial degrees of freedom, respectively. Polarization is a feature of vector fields while spatial phase gradient determines the OAM ubiquitous to any scalar field. Common wisdom treats these two degrees of freedom as independent principles to manipulate wave propagations. Here, their synergy is demonstrated. This is achieved by introducing two orthogonal p-orbitals as eigenbases, whose spatial modal features are exploited to generate OAM, and the associated orbital orientations provide means to simultaneously manipulate polarizations. Through periodic modulation and directional coupling, a full cyclic evolution of synchronized and synergized SAM-OAM is realized. Remarkably, this evolution acquires a nontrivial geometric phase, leading to its representation on a Möbius strip. Experimentally, an acoustic cavity array is designed, whose dipole resonances precisely mimic the p-orbitals, with pressure fields featuring the spatial characteristics and velocity fields the vectorial orientations. Based on this synergy, a spin-orbital-Hall effect is further showcased, highlighting the intricate locking of handedness, directionality, spin density, and spatial mode profile. This study unveils a fundamental connection between SAM and OAM, promising avenues for their novel applications in trapping, coding, and communications.

MERGE: A Modal Equilibrium Relational Graph Framework for Multi-Modal Knowledge Graph Completion.

The multi-modal knowledge graph completion (MMKGC) task aims to automatically mine the missing factual knowledge from the existing multi-modal knowledge graphs (MMKGs), which is crucial in advancing cross-modal learning and reasoning. However, few methods consider the adverse effects caused by different missing modal information in the model learning process. To address the above challenges, we innovatively propose a Modal Equilibrium Relational Graph framEwork, called MERGE. By constructing three modal-specific directed relational graph attention networks, MERGE can implicitly represent missing modal information for entities by aggregating the modal embeddings from neighboring nodes. Subsequently, a fusion approach based on low-rank tensor decomposition is adopted to align multiple modal features in both the explicit structural level and the implicit semantic level, utilizing the structural information inherent in the original knowledge graphs, which enhances the interpretability of the fused features. Furthermore, we introduce a novel interpolation re-ranking strategy to adjust the importance of modalities during inference while preserving the semantic integrity of each modality. The proposed framework has been validated on four publicly available datasets, and the experimental results have demonstrated the effectiveness and robustness of our method in the MMKGC task.

First-order GBT for tapered regular convex polygonal tubes

This paper presents an accurate and efficient first-order Generalized Beam Theory (GBT) for linearly tapered regular convex polygonal tubes, such as those widely employed in the construction industry. Even though tapered members require a significantly involved formulation, it is shown that it is possible to enforce the standard GBT assumptions exactly, without additional simplifications, a key aspect that (i) is essential for the accuracy and computational performance of the formulation and (ii) allows identifying the deformed configurations pertaining to inextensible deformation. Consequently, very accurate solutions are achieved even for complex cases, such as tubes with a high taper angle and undergoing localized deformation. The GBT deformation modes for the prismatic case are directly used, meaning that the proposed approach for tapered tubes does not require a specific GBT cross-section analysis procedure. All expressions are presented in a straightforward vector–matrix format, for implementation purposes. The excellent performance of the resulting displacement-based beam finite element and the advantages of the GBT modal decomposition features are highlighted through several numerical examples, where results obtained with refined shell finite element models are used for comparison purposes.

MMAformer: Multiscale Modality-Aware Transformer for Medical Image Segmentation

The segmentation of medical images, particularly for brain tumors, is essential for clinical diagnosis and treatment planning. In this study, we proposed MMAformer, a Multiscale Modality-Aware Transformer model, which is designed for segmenting brain tumors by utilizing multimodality magnetic resonance imaging (MRI). Complementary information between different sequences helps the model delineate tumor boundaries and distinguish different tumor tissues. To enable the model to acquire the complementary information between related sequences, MMAformer employs a multistage encoder, which uses a cross-modal downsampling (CMD) block for learning and integrating the complementary information between sequences at different scales. In order to effectively fuse the various information extracted by the encoder, the Multimodal Gated Aggregation (MGA) block combines the dual attention mechanism and multi-gated clustering to effectively fuse the spatial, channel, and modal features of different MRI sequences. In the comparison experiments on the BraTS2020 and BraTS2021 datasets, the average Dice score of MMAformer reached 86.3% and 91.53%, respectively, indicating that MMAformer surpasses the current state-of-the-art approaches. MMAformer’s innovative architecture, which effectively captures and integrates multimodal information at various scales, offers a promising solution for tackling complex medical image segmentation challenges.

Integrating scRNA-seq and scATAC-seq with inter-type attention heterogeneous graph neural networks.

Single-cell multi-omics techniques, which enable the simultaneous measurement of multiple modalities such as RNA gene expression and Assay for Transposase-Accessible Chromatin (ATAC) within individual cells, have become a powerful tool for deciphering the intricate complexity of cellular systems. Most current methods rely on motif databases to establish cross-modality relationships between genes from RNA-seq data and peaks from ATAC-seq data. However, these approaches are constrained by incomplete database coverage, particularly for novel or poorly characterized relationships. To address these limitations, we introduce single-cell Multi-omics Integration (scMI), a heterogeneous graph embedding method that encodes both cells and modality features from single-cell RNA-seq and ATAC-seq data into a shared latent space by learning cross-modality relationships. By modeling cells and modality features as distinct node types, we design an inter-type attention mechanism to effectively capture long-range cross-modality interactions between genes and peaks. Benchmark results demonstrate that embeddings learned by scMI preserve more biological information and achieve comparable or superior performance in downstream tasks including modality prediction, cell clustering, and gene regulatory network inference compared to methods that rely on databases. Furthermore, scMI significantly improves the alignment and integration of unmatched multi-omics data, enabling more accurate embedding and improved outcomes in downstream tasks.

MORE: a multi-omics data-driven hypergraph integration network for biomedical data classification and biomarker identification.

High-throughput sequencing methods have brought about a huge change in omics-based biomedical study. Integrating various omics data is possibly useful for identifying some correlations across data modalities, thus improving our understanding of the underlying biological mechanisms and complexity. Nevertheless, most existing graph-based feature extraction methods overlook the complementary information and correlations across modalities. Moreover, these methods tend to treat the features of each omics modality equally, which contradicts current biological principles. To solve these challenges, we introduce a novel approach for integrating multi-omics data termed Multi-Omics hypeRgraph integration nEtwork (MORE). MORE initially constructs a comprehensive hyperedge group by extensively investigating the informative correlations within and across modalities. Subsequently, the multi-omics hypergraph encoding module is employed to learn the enriched omics-specific information. Afterward, the multi-omics self-attention mechanism is then utilized to adaptatively aggregate valuable correlations across modalities for representation learning and making the final prediction. We assess MORE's performance on datasets characterized by message RNA (mRNA) expression, Deoxyribonucleic Acid (DNA) methylation, and microRNA (miRNA) expression for Alzheimer's disease, invasive breast carcinoma, and glioblastoma. The results from three classification tasks highlight the competitive advantage of MORE in contrast with current state-of-the-art (SOTA) methods. Moreover, the results also show that MORE has the capability to identify a greater variety of disease-related biomarkers compared to existing methods, highlighting its advantages in biomedical data mining and interpretation. Overall, MORE can be investigated as a valuable tool for facilitating multi-omics analysis and novel biomarker discovery. Our code and data can be publicly accessed at https://github.com/Wangyuhanxx/MORE.