Modal Features Research Articles

Neural networks-based damage detection for symmetrical structures considering modal feature similarity

Symmetric structures are commonly used in various fields. To ensure the safety of these structures, it is crucial to employ appropriate damage detection methods. Among the methods used for structural health detection, the combination of structural dynamics and neural networks based on finite element models has gained popularity due to its non-destructive nature. This study presents a neural network-based damage detection approach for symmetrical structures, taking into account the similarity of modal features. By considering the interference of modal feature similarity, the proposed method effectively detects damage in symmetrical structures. In this approach, the rates of mode shape change before and after damage are used as inputs to the neural network, as they are found to be less sensitive to modeling errors compared to the mode shapes themselves. The effectiveness and adaptability of the method are validated through a numerical case analysis of a symmetrical load-bearing base plate used in walkers for patients with spinal cord injuries.

EEG-eye movement based subject dependence, cross-subject, and cross-session emotion recognition with multidimensional homogeneous encoding space alignment

The joint learning of multimodal is helpful to extract the general information cross-modality in improving the performance of multimodal emotion recognition. However, focusing on a single common pattern can cause multimodal data to deviate from its original distribution and fail to fully capture the potential representation of the data. Therefore, we propose a multi-dimensional homogenous encoding spatial alignment (MHESA) method, which consists of two parts: multi-modal joint learning and modal knowledge transfer. To obtain a common projection space of EEG-EM features, we use a multimodal joint space encoder to learn the homogeneous joint space of EEG-Eye Movement (EM). To obtain a homogeneous encoding space based on modal knowledge, the knowledge transfer module learns the spatial distribution of EM features while retaining the original EEG features. The output of each module is used to construct a multidimensional homogeneous encoding space. The weight and multi-task loss function of the multi-dimensional homogeneous encoding space are dynamically adjusted by the Multi-task Joint Optimization Strategy (MJOS). By analyzing the effect of multi-task optimization, we found that compared with the subject dependence scene, the cross-subject scene has an advantage in the construction of joint encoding space, and the modal knowledge transfer feature has a higher contribution degree in cross-session. The experimental results show the MHESA method can make the model achieve more stable performance in three emotion recognition scenes.

SLE diagnosis research based on SERS combined with a multi-modal fusion method

As artificial intelligence technology gains widespread adoption in biomedicine, the exploration of integrating biofluidic Raman spectroscopy for enhanced disease diagnosis opens up new prospects for the practical application of Raman spectroscopy in clinical settings. However, for systemic lupus erythematosus (SLE), origin Raman spectral data (ORS) have relatively weak signals, making it challenging to obtain ideal classification results. Although the surface enhancement technique can enhance the scattering signal of Raman spectroscopic data, the sensitivity of the SERS substrate to airborne impurities and the inhomogeneous distribution of hotspots degrade part of the signal. To fully utilize both kinds of data, this paper proposes a two-branch residual-attention network (DBRAN) fusion technique, which allows the ORS to complement the degraded portion and thus improve the model's classification accuracy. The features are extracted using the residual module, which retains the original features while extracting the deep features. At the same time, the study incorporates the attention module in both the upper and lower branches to handle the weight allocation of the two modal features more efficiently. The experimental results demonstrate that both the low-level fusion method and the intermediate-level fusion method can significantly improve the diagnostic accuracy of SLE disease classification compared with a single modality, in which the intermediate-level fusion of DBRAN achieves 100% classification accuracy, sensitivity, and specificity. The accuracy is improved by 10% and 7% compared with the ORS unimodal and the SERS unimodal modalities, respectively. The experiment, by fusing the multimodal spectral, realized rapid diagnosis of SLE disease by fusing multimodal spectral data, which provides a reference idea in the field of Raman spectroscopy and can be further promoted to clinical practical applications in the future.

SigFormer: Sparse Signal-Guided Transformer for Multi-Modal Action Segmentation

Multi-modal human action segmentation is a critical and challenging task with a wide range of applications. Nowadays, the majority of approaches concentrate on the fusion of dense signals (i.e., RGB, optical flow, and depth maps). However, the potential contributions of sparse IoT sensor signals, which can be crucial for achieving accurate recognition, have not been fully explored. To make up for this, we introduce a S parse s i gnal- g uided Transformer ( SigFormer ) to combine both dense and sparse signals. We employ mask attention to fuse localized features by constraining cross-attention within the regions where sparse signals are valid. However, since sparse signals are discrete, they lack sufficient information about the temporal action boundaries. Therefore, in SigFormer, we propose to emphasize the boundary information at two stages to alleviate this problem. In the first feature extraction stage, we introduce an intermediate bottleneck module to jointly learn both category and boundary features of each dense modality through the inner loss functions. After the fusion of dense modalities and sparse signals, we then devise a two-branch architecture that explicitly models the interrelationship between action category and temporal boundary. Experimental results demonstrate that SigFormer outperforms the state-of-the-art approaches on a multi-modal action segmentation dataset from real industrial environments, reaching an outstanding F1 score of 0.958. The codes and pre-trained models have been available at https://github.com/LIUQI-creat/SigFormer.

Modal acoustic emission-based circumferential crack feature extractions for pipeline welds with L-shaped flexible sensor array

ABSTRACT The collection and analysis of acoustic emission (AE) from weld circumferential cracks are crucial for ensuring pipeline safety. However, the unclear propagation characteristics and mode features of AE present challenges in array design and mode identification. In this manuscript, a novel mode feature extraction method for circumferential crack AE signals is proposed. This method involves three primary steps: firstly, a L-shaped flexible sensor array is designed to capture raw AE signals from both axial and circumferential directions. Next, the collected signals are filtered and evaluated to extract effective multi-mode components through the Kalman filtering and recursive plot (RP). Finally, the time–frequency features and mode types are extracted through combining the Hilbert-Huang transform (HHT) and dispersion curves. Results indicate that during the crack growths, both axial and circumferential direction AE signals contain multi-mode components, specifically L(0,1), F(1,1) and F(1,2) modes, with durations spanning 30–500 μs. Additionally, the axial modes predominantly occur within 200–300 kHz range, whereas the circumferential modes span both low and high frequency bands, specifically 40–50 kHz and 200–300 kHz, respectively. Modes across different frequencies indicate distinct structural behaviours, including crack growths and plastic deformations. The proposed method provides attempts for the pipe weld AE monitoring and multi-mode analysis.

Text-based person search via cross-modal alignment learning

Text-based person search aims to use text descriptions to search for corresponding person images. However, due to the obvious pattern differences in image and text modalities, it is still a challenging problem to align the two modalities. Most existing approaches only consider semantic alignment within a global context or partial parts, lacking consideration of how to match image and text in terms of differences in model information. Therefore, in this paper, we propose an efficient Modality-Aligned Person Search network (MAPS) to address this problem. First, we suppress image-specific information by image feature style normalization to achieve modality knowledge alignment and reduce information differences between text and image. Second, we design a multi-granularity modal feature fusion and optimization method to enrich the modal features. To address the problem of useless and redundant information in the multi-granularity fused features, we propose a Multi-granularity Feature Self-optimization Module (MFSM) to adaptively adjust the corresponding contributions of different granularities in the fused features of the two modalities. Finally, to address the problem of information inconsistency in the training and inference stages, we propose a Cross-instance Feature Alignment (CFA) to help the network enhance category-level generalization ability and improve retrieval performance. Extensive experiments demonstrate that our MAPS achieves state-of-the-art performance on all text-based person search datasets, and significantly outperforms other existing methods.

Epileptic spasms relapse is associated with response latency but not conventional attributes of post-treatment EEG.

Relapse of epileptic spasms after initial treatment of infantile epileptic spasms syndrome (IESS) is common. However, past studies of small cohorts have inconsistently linked relapse risk to etiology, treatment modality, and EEG features upon response. Using a large single-center IESS cohort, we set out to quantify the risk of epileptic spasms relapse and identify specific risk factors. We identified all children with epileptic spasms at our center using a clinical EEG database. Using the electronic medical record, we confirmed IESS syndrome classification and ascertained treatment, response, time to relapse, etiology, EEG features, and other demographic factors. Relapse-free survival analysis was carried out using Cox proportional hazards regression. Among 599 children with IESS, 197 specifically responded to hormonal therapy and/or vigabatrin (as opposed to surgery or other second-line treatments). In this study, 41 (21%) subjects exhibited relapse of epileptic spasms within 12 months of response. Longer duration of IESS prior to response (>3 months) was strongly associated with shorter latency to relapse (hazard ratio = 3.11; 95% CI 1.59-6.10; p = 0.001). Relapse was not associated with etiology, developmental status, or any post-treatment EEG feature. This study suggests that long duration of IESS before response is the single largest clinical predictor of relapse risk, and therefore underscores the importance of prompt and successful initial treatment. Further study is needed to evaluate candidate biomarkers of epileptic spasms relapse and identify treatments to mitigate this risk. Relapse of infantile spasms is common after initially successful treatment. With study of a large group of children with infantile spasms, we determined that relapse is linked to long duration of infantile spasms. In contrast, relapse was not associated with the cause of infantile spasms, developmental measures, or EEG features at the time of initial response. Further study is needed to identify tools to predict impending relapse of infantile spasms.

A muti-modal feature fusion method based on deep learning for predicting immunotherapy response

Immune checkpoint therapy (ICT) has greatly improved the survival of cancer patients in the past few years, but only a small number of patients respond to ICT. To predict ICT response, we developed a multi-modal feature fusion model based on deep learning (MFMDL). This model utilizes graph neural networks to map gene-gene relationships in gene networks to low dimensional vector spaces, and then fuses biological pathway features and immune cell infiltration features to make robust predictions of ICT. We used five datasets to validate the predictive performance of the MFMDL. These five datasets span multiple types of cancer, including melanoma, lung cancer, and gastric cancer. We found that the prediction performance of multi-modal feature fusion model based on deep learning is superior to other traditional ICT biomarkers, such as ICT targets or tumor microenvironment-associated markers. In addition, we also conducted ablation experiments to demonstrate the necessity of fusing different modal features, which can improve the prediction accuracy of the model.

Analysis and optimization of pre-stressed modal features of ship anchor support parts

In order to solve the problems of excessive weight and unreasonable structure of anchor machine parts caused by traditional design methods, a lightweight optimization method was proposed based on pre-stressed modal analysis. The design variables were determined, and the parameterized model was established by using ANSYS Workbench. Under ultimate load conditions, the strength of wall frame board and lower box bodies was simulated and calculated. Through modal analysis, the discretized natural frequencies under different design variables could be obtained. The multi-objective genetic algorithm and sequence quadratic programming were respectively used to calculate the lightweight analysis model. The results showed that the weight of the supporting components in ship anchor can be reduced by more than 5 % without reducing strength and equivalent stiffness.

Emotional analysis of joint sports quality expansion tasks based on multi-modal feature fusion

A multi-modal feature based motion emotion analysis model based on a fusion deep learning model is proposed for the problem of analyzing the motion emotions of participants in the joint exercise quality expansion task. This model involves three major modalities: EEG signals, peripheral physiological signals, and facial expression signals, and processes and fuses the information of these three main modalities to achieve the effect of processing multi-dimensional motor emotional information. At the same time, this study introduces the design concept of residual networks, using self attention modules and multi head mutual attention modules to process different modal features. The results showed that the combination of peripheral physiological modality and facial expression modality had the highest accuracy among the three modality combinations, with an accuracy rate of 88.8 %. The feature fusion method based on the cascaded residual attention mechanism module has better accuracy and F1 Score performance than other methods. In addition, different emotional states can be effectively identified and distinguished in these three modalities, indicating that the model has a wide range of possibilities in practical applications.

Deep Multimodal Fusion of Data with Heterogeneous Dimensionality via Projective Networks.

The use of multimodal imaging has led to significant improvements in the diagnosis and treatment of many diseases. Similar to clinical practice, some works have demonstrated the benefits of multimodal fusion for automatic segmentation and classification using deep learning-based methods. However, current segmentation methods are limited to fusion of modalities with the same dimensionality (e.g., 3D+3D, 2D+2D), which is not always possible, and the fusion strategies implemented by classification methods are incompatible with localization tasks. In this work, we propose a novel deep learning-based framework for the fusion of multimodal data with heterogeneous dimensionality (e.g., 3D+2D) that is compatible with localization tasks. The proposed framework extracts the features of the different modalities and projects them into the common feature subspace. The projected features are then fused and further processed to obtain the final prediction. The framework was validated on the following tasks: segmentation of geographic atrophy (GA), a late-stage manifestation of age-related macular degeneration, and segmentation of retinal blood vessels (RBV) in multimodal retinal imaging. Our results show that the proposed method outperforms the state-of-the-art monomodal methods on GA and RBV segmentation by up to 3.10% and 4.64% Dice, respectively.

Dynamic Weighted Combiner for Mixed-Modal Image Retrieval

Mixed-Modal Image Retrieval (MMIR) as a flexible search paradigm has attracted wide attention. However, previous approaches always achieve limited performance, due to two critical factors are seriously overlooked. 1) The contribution of image and text modalities is different, but incorrectly treated equally. 2) There exist inherent labeling noises in describing users' intentions with text in web datasets from diverse real-world scenarios, giving rise to overfitting. We propose a Dynamic Weighted Combiner (DWC) to tackle the above challenges, which includes three merits. First, we propose an Editable Modality De-equalizer (EMD) by taking into account the contribution disparity between modalities, containing two modality feature editors and an adaptive weighted combiner. Second, to alleviate labeling noises and data bias, we propose a dynamic soft-similarity label generator (SSG) to implicitly improve noisy supervision. Finally, to bridge modality gaps and facilitate similarity learning, we propose a CLIP-based mutual enhancement module alternately trained by a mixed-modality contrastive loss. Extensive experiments verify that our proposed model significantly outperforms state-of-the-art methods on real-world datasets. The source code is available at https://github.com/fuxianghuang1/DWC.

A Learnable Discrete-Prior Fusion Autoencoder with Contrastive Learning for Tabular Data Synthesis

The actual collection of tabular data for sharing involves confidentiality and privacy constraints, leaving the potential risks of machine learning for interventional data analysis unsafely averted. Synthetic data has emerged recently as a privacy-protecting solution to address this challenge. However, existing approaches regard discrete and continuous modal features as separate entities, thus falling short in properly capturing their inherent correlations. In this paper, we propose a novel contrastive learning guided Gaussian Transformer autoencoder, termed GTCoder, to synthesize photo-realistic multimodal tabular data for scientific research. Our approach introduces a transformer-based fusion module that seamlessly integrates multimodal features, permitting for mining more informative latent representations. The attention within the fusion module directs the integrated output features to focus on critical components that facilitate the task of generating latent embeddings. Moreover, we formulate a contrastive learning strategy to implicitly constrain the embeddings from discrete features in the latent feature space by encouraging the similar discrete feature distributions closer while pushing the dissimilar further away, in order to better enhance the representation of the latent embedding. Experimental results indicate that GTCoder is effective to generate photo-realistic synthetic data, with interactive interpretation of latent embedding, and performs favorably against some baselines on most real-world and simulated datasets.

LGMRec: Local and Global Graph Learning for Multimodal Recommendation

The multimodal recommendation has gradually become the infrastructure of online media platforms, enabling them to provide personalized service to users through a joint modeling of user historical behaviors (e.g., purchases, clicks) and item various modalities (e.g., visual and textual). The majority of existing studies typically focus on utilizing modal features or modal-related graph structure to learn user local interests. Nevertheless, these approaches encounter two limitations: (1) Shared updates of user ID embeddings result in the consequential coupling between collaboration and multimodal signals; (2) Lack of exploration into robust global user interests to alleviate the sparse interaction problems faced by local interest modeling. To address these issues, we propose a novel Local and Global Graph Learning-guided Multimodal Recommender (LGMRec), which jointly models local and global user interests. Specifically, we present a local graph embedding module to independently learn collaborative-related and modality-related embeddings of users and items with local topological relations. Moreover, a global hypergraph embedding module is designed to capture global user and item embeddings by modeling insightful global dependency relations. The global embeddings acquired within the hypergraph embedding space can then be combined with two decoupled local embeddings to improve the accuracy and robustness of recommendations. Extensive experiments conducted on three benchmark datasets demonstrate the superiority of our LGMRec over various state-of-the-art recommendation baselines, showcasing its effectiveness in modeling both local and global user interests.

Tri-CLT: Learning Tri-Modal Representations with Contrastive Learning and Transformer for Multimodal Sentiment Recognition

Multimodal Sentiment Analysis (MSA) has become an essential area of research to achieve more accurate sentiment analysis by integrating multiple perceptual modalities such as text, vision, and audio. However, most previous studies failed to align the various modalities well and ignored the differences in semantic information, leading to inefficient fusion between modalities and generating redundant information. In order to solve the above problems, this paper proposes a transformer-based network model, Tri-CLT. Specifically, this paper designs Integrating Fusion Block to fuse modal features to enhance their semantic information and mitigate the secondary complexity of paired sequences in the transformer. Meanwhile, the cross-modal attention mechanism is utilized for complementary learning between modalities to enhance the model performance. In addition, contrastive learning is introduced to improve the model's representation of learning ability. Finally, this paper conducts experiments on CMU-MOSEI aligned and unaligned data, and the experimental results show that the proposed method outperforms the existing methods.

DHFNet: Decoupled Hierarchical Fusion Network for RGB-T dense prediction tasks

The fusion of RGB and thermal data for dense prediction tasks has been demonstrated to be an effective and robust approach in autonomous driving. Nevertheless, the challenge lies in fusing features from different modalities, as a simple fusion strategy may lead to redundant or conflicting semantic information. In this paper, we delve into the hierarchical connections between multi-modal features and propose a novel fusion paradigm termed DHFNet, which decouples multi-modal features into similar global long-distance and discrepant local detail features for hierarchical feature fusion. At each fusion stage, a Lightweight Global Self-attention (LGSA) module is designed to decouple the global long-distance features with cheap computational complexity cost, and a Cross Modal Long-distance Feature Fusion (CMLFF) module is designed to eliminate redundant features by facilitating information interaction between different modalities. During the process of decoupling local detail features, a Cross Modal Deformable Convolution (CMDC) module is proposed to dynamically extract effective local features and capture misaligned features between different modalities. Finally, the fused global long-distance and local detail features are recoupled to achieve efficient hierarchical fusion. The results on RGB-T semantic segmentation and object detection tasks demonstrate the effectiveness of proposed method. The code will be available at: https://github.com/donggaomu/DHFNet.

Automatic classification of immune-mediated glomerular diseases based on multi-modal multi-instance learning

To develop a multi-modal deep learning method for automatic classification of immune-mediated glomerular diseases based on images of optical microscopy (OM), immunofluorescence microscopy (IM), and transmission electron microscopy (TEM). We retrospectively collected the pathological images from 273 patients and constructed a multi-modal multi- instance model for classification of 3 immune-mediated glomerular diseases, namely immunoglobulin A nephropathy (IgAN), membranous nephropathy (MN), and lupus nephritis (LN). This model adopts an instance-level multi-instance learning (I-MIL) method to select the TEM images for multi-modal feature fusion with the OM images and IM images of the same patient. By comparing this model with unimodal and bimodal models, we explored different combinations of the 3 modalities and the optimal methods for modal feature fusion. The multi-modal multi-instance model combining OM, IM, and TEM images had a disease classification accuracy of (88.34±2.12)%, superior to that of the optimal unimodal model [(87.08±4.25)%] and that of the optimal bimodal model [(87.92±3.06)%]. This multi- modal multi- instance model based on OM, IM, and TEM images can achieve automatic classification of immune-mediated glomerular diseases with a good classification accuracy.

Characteristics of Dry and Wet Changes and Future Trends in the Tarim River Basin Based on the Standardized Precipitation Evapotranspiration Index

Global changes in drought and wetness and their future trends in arid regions have recently become a major focus of research attention. The Tarim River Basin (TRB) in Xinjiang, China, is among the most climate-sensitive regions in the world. This study uses data from the past 60 years (1962–2021) to analyze the spatial and temporal features of drought and wetness conditions in the TRB, calculating the Standardized Precipitation Evapotranspiration Index (SPEI). Trend detection for SPEI is performed using the BEAST mutation test, identification of drought events using the theory of operations, and spatial and temporal analyses of dry and wet changes using Empirical Orthogonal Function (EOF) decomposition. Additionally, the CMIP6 dataset is used to estimate future changes. The study results indicate the following: (1) From 1962 to 1998, the TRB exhibited a “warm and wet” trend that suddenly shifted from “wet-to-dry” in 1998 and subsequently transitioned to a pronounced “warm and dry” trend. (2) After the “wet-to-dry” shift, the frequency of drought events noticeably increased. The northern section of the basin witnessed more frequent drought events, albeit with lower severity, while the southern part had fewer occurrences but with higher severity. The spatial distribution of drought event frequency and severity is inconsistent. (3) The EOF decomposition results for SPEI-variable fields at 1-, 3-, and 6-month time scales show that the cumulative variance contribution rate of the first three principal spatial modal feature vectors exceeds 70%. The spatial distribution of the modes includes a consistent pattern across the entire basin, a north–south opposite pattern, and an east–west opposite pattern. (4) The future trend of drought in the TRB is expected to intensify, manifesting a spatial pattern characterized by dryness in the middle of the basin and wetness around the periphery. These research findings can provide support for decisions addressing regional drought risks.

Sûdî’nin Bostan Şerhi’nde Nesir Dili

Sûdî is one of the representatives of 16th century Turkish classical prose literature. In his annotated work of Bostan he preferred quite a simple language, free from internal rhyme and compound words. He made his explanations short, using defining annotation style. As he pursued overall goal of language teaching, there are oft-repeated sentences of information in the work. His mastery over Persian language enabled Sûdî to meticulously work on the derivational and inflectional affixes. As the main part of speech necessary to understand a statement is the verb, there is quite a lot of information embedded in the work regarding the modality features of Persian verbs, whether they are transitive or intransitive, or whether they are affirmative or negative. There is information given as to in what sense phrases are used, which words pronouns refer to in the text, and how a pair of lines that conveys figures of speech should be inferred and understood. He severely criticized the authors who previously worked on Bostan due to their mistakes in their annotated works, their inadequate knowledge of Bostan, their sheer lack of comprehension of the sentences. It is obvious that he made use of Turkish, Persian and Arabic conjunctions, gerunds, participles and adverbials. There does not seem to be any particular use of artistic language in his prose. He produced a kind of prose a lot easier to understand than those used in his age. Key words: Sûdî, Annotated Work of Bostan, language of prose, sentence construction

Different gait combinations based on multi-modal deep CNN architectures

Gait recognition is the process of identifying a person from a distance based on their walking patterns. However, the recognition rate drops significantly under cross-view angle and appearance-based variations. In this study, the effectiveness of the most well-known gait representations in solving this problem is investigated based on deep learning. For this purpose, a comprehensive performance evaluation is performed by combining different modalities, including silhouettes, optical flows, and concatenated image of the Gait Energy Image (GEI) head and leg region, with GEI itself. This evaluation is carried out across different multimodal deep convolutional neural network (CNN) architectures, namely fine-tuned EfficientNet-B0, MobileNet-V1, and ConvNeXt-base models. These models are trained separately on GEIs, silhouettes, optical flows, and concatenated image of GEI head and leg regions, and then extracted GEI features are fused in pairs with other extracted modality features to find the most effective gait combination. Experimental results on the two different datasets CASIA-B and Outdoor-Gait show that the concatenated image of GEI head and leg regions significantly increased the recognition rate of the networks compared to other modalities. Moreover, this modality demonstrates greater robustness under varied carrying (BG) and clothing (CL) conditions compared to optical flows (OF) and silhouettes (SF). Codes available at https://github.com/busrakckugurlu/Different-gait-combinations-based-on-multi-modal-deep-CNN-architectures.git