Processing Of Multimodal Data Research Articles

Implementation and Evaluation of Attention Aggregation Technique for Pear Disease Detection

In this study, a novel approach integrating multimodal data processing and attention aggregation techniques is proposed for pear tree disease detection. The focus of the research is to enhance the accuracy and efficiency of disease detection by fusing data from diverse sources, including images and environmental sensors. The experimental results demonstrate that the proposed method outperforms in key performance metrics such as precision, recall, accuracy, and F1-Score. Specifically, the model was tested on the Kaggle dataset and compared with existing advanced models such as RetinaNet, EfficientDet, Detection Transformer (DETR), and the You Only Look Once (YOLO) series. The experimental outcomes indicate that the proposed model achieves a precision of 0.93, a recall of 0.90, an accuracy of 0.92, and an F1-Score of 0.91, surpassing those of the comparative models. Additionally, detailed ablation experiments were conducted on the multimodal weighting module and the dynamic regression loss function to verify their specific contributions to the model performance. These experiments not only validated the effectiveness of the proposed method but also demonstrate its potential application in pear tree disease detection. Through this research, an effective technological solution is provided for the agricultural disease detection domain, offering substantial practical value and broad application prospects.

Multimodal Environmental Sensing Using AI & IoT Solutions: A Cognitive Sound Analysis Perspective.

This study presents a novel audio compression technique, tailored for environmental monitoring within multi-modal data processing pipelines. Considering the crucial role that audio data play in environmental evaluations, particularly in contexts with extreme resource limitations, our strategy substantially decreases bit rates to facilitate efficient data transfer and storage. This is accomplished without undermining the accuracy necessary for trustworthy air pollution analysis while simultaneously minimizing processing expenses. More specifically, our approach fuses a Deep-Learning-based model, optimized for edge devices, along with a conventional coding schema for audio compression. Once transmitted to the cloud, the compressed data undergo a decoding process, leveraging vast cloud computing resources for accurate reconstruction and classification. The experimental results indicate that our approach leads to a relatively minor decrease in accuracy, even at notably low bit rates, and demonstrates strong robustness in identifying data from labels not included in our training dataset.

ЗАСТОСУВАННЯ ГЛИБОКИХ ШТУЧНИХ НЕЙРОННИХ МЕРЕЖ ДЛЯ КЛАСИФІКАЦІЇ МУЛЬТИМОДАЛЬНИХ ДАНИХ

Multimodal data analysis is gaining attention in recent research. Pu Liang et al. (2023) provide a comprehensive overview on multimodal machine learning, highlighting its founda-tions, challenges and achievements in recent years. More problem-oriented works propose new methods and applications for multimodal ML, such a Ngiam et al. (2011) propose to use joint audio and video data to improve speech recognition accuracy; Sun, Wand and Li (2018) describe application of multimodal classification for breast cancer prognosis prediction; Mao et al. (2014) propose an architecture of multimodal recurrent network to generate text de-scription of images and so on. However, such works usually focus on the task itself and meth-ods therein, and not on integrating multimodal data processing into other software systems. The goal of this research is to propose a way to conduct multimodal data processing, specifically as a part of a digital twin systems, thus efficiency and near-real-time operation are required. The paper presents an approach to conduct parallel multimodal data classification, adapting to available computing power. The method is modular and scalable and intended for in digital twin application as a part of analysis and modeling tools. Later, the detailed example of such a software module is discussed. It uses multimodal data from open datasets to detect and classify the behavior of pets using deep learning mod-els. Videos are processed using two artificial neural networks: YOLOv3 object detection net-work to process individual frames of the video and a relatively simple convolutional network to classify sounds based on their frequency spectra. Constructed module uses a producer-consumer parallel processing pattern and allows processing 5 frames per second of a video on available hardware, which can be sufficiently improved by using GPU acceleration or more paralleled processing threads.

Early Detection of Extreme Storm Tide Events Using Multimodal Data Processing

Sea-level rise is a well-known consequence of climate change. Several studies have estimated the social and economic impact of the increase in extreme flooding. An efficient way to mitigate its consequences is the development of a flood alert and prediction system, based on high-resolution numerical models and robust sensing networks. However, current models use various simplifying assumptions that compromise accuracy to ensure solvability within a reasonable timeframe, hindering more regular and cost-effective forecasts for various locations along the shoreline. To address these issues, this work proposes a hybrid model for multimodal data processing that combines physics-based numerical simulations, data obtained from a network of sensors, and satellite images to provide refined wave and sea-surface height forecasts, with real results obtained in a critical location within the Port of Santos (the largest port in Latin America). Our approach exhibits faster convergence than data-driven models while achieving more accurate predictions. Moreover, the model handles irregularly sampled time series and missing data without the need for complex preprocessing mechanisms or data imputation while keeping low computational costs through a combination of time encoding, recurrent and graph neural networks. Enabling raw sensor data to be easily combined with existing physics-based models opens up new possibilities for accurate extreme storm tide events forecast systems that enhance community safety and aid policymakers in their decision-making processes.

Artificial intelligence-assisted repair of peripheral nerve injury: a new research hotspot and associated challenges.

Artificial intelligence can be indirectly applied to the repair of peripheral nerve injury. Specifically, it can be used to analyze and process data regarding peripheral nerve injury and repair, while study findings on peripheral nerve injury and repair can provide valuable data to enrich artificial intelligence algorithms. To investigate advances in the use of artificial intelligence in the diagnosis, rehabilitation, and scientific examination of peripheral nerve injury, we used CiteSpace and VOSviewer software to analyze the relevant literature included in the Web of Science from 1994-2023. We identified the following research hotspots in peripheral nerve injury and repair: (1) diagnosis, classification, and prognostic assessment of peripheral nerve injury using neuroimaging and artificial intelligence techniques, such as corneal confocal microscopy and coherent anti-Stokes Raman spectroscopy; (2) motion control and rehabilitation following peripheral nerve injury using artificial neural networks and machine learning algorithms, such as wearable devices and assisted wheelchair systems; (3) improving the accuracy and effectiveness of peripheral nerve electrical stimulation therapy using artificial intelligence techniques combined with deep learning, such as implantable peripheral nerve interfaces; (4) the application of artificial intelligence technology to brain-machine interfaces for disabled patients and those with reduced mobility, enabling them to control devices such as networked hand prostheses; (5) artificial intelligence robots that can replace doctors in certain procedures during surgery or rehabilitation, thereby reducing surgical risk and complications, and facilitating postoperative recovery. Although artificial intelligence has shown many benefits and potential applications in peripheral nerve injury and repair, there are some limitations to this technology, such as the consequences of missing or imbalanced data, low data accuracy and reproducibility, and ethical issues (e.g., privacy, data security, research transparency). Future research should address the issue of data collection, as large-scale, high-quality clinical datasets are required to establish effective artificial intelligence models. Multimodal data processing is also necessary, along with interdisciplinary collaboration, medical-industrial integration, and multicenter, large-sample clinical studies.

The hospital at home in the USA: current status and future prospects.

The annual cost of hospital care services in the US has risen to over $1 trillion despite relatively worse health outcomes compared to similar nations. These trends accentuate a growing need for innovative care delivery models that reduce costs and improve outcomes. HaH-a program that provides patients acute-level hospital care at home-has made significant progress over the past two decades. Technological advancements in remote patient monitoring, wearable sensors, health information technology infrastructure, and multimodal health data processing have contributed to its rise across hospitals. More recently, the COVID-19 pandemic brought HaH into the mainstream, especially in the US, with reimbursement waivers that made the model financially acceptable for hospitals and payors. However, HaH continues to face serious challenges to gain widespread adoption. In this review, we evaluate the peer-reviewed evidence and discuss the promises, challenges, and what it would take to tap into the future potential of HaH.

InClust+: the deep generative framework with mask modules for multimodal data integration, imputation, and cross-modal generation

BackgroundWith the development of single-cell technology, many cell traits can be measured. Furthermore, the multi-omics profiling technology could jointly measure two or more traits in a single cell simultaneously. In order to process the various data accumulated rapidly, computational methods for multimodal data integration are needed.ResultsHere, we present inClust+, a deep generative framework for the multi-omics. It’s built on previous inClust that is specific for transcriptome data, and augmented with two mask modules designed for multimodal data processing: an input-mask module in front of the encoder and an output-mask module behind the decoder. InClust+ was first used to integrate scRNA-seq and MERFISH data from similar cell populations, and to impute MERFISH data based on scRNA-seq data. Then, inClust+ was shown to have the capability to integrate the multimodal data (e.g. tri-modal data with gene expression, chromatin accessibility and protein abundance) with batch effect. Finally, inClust+ was used to integrate an unlabeled monomodal scRNA-seq dataset and two labeled multimodal CITE-seq datasets, transfer labels from CITE-seq datasets to scRNA-seq dataset, and generate the missing modality of protein abundance in monomodal scRNA-seq data. In the above examples, the performance of inClust+ is better than or comparable to the most recent tools in the corresponding task.ConclusionsThe inClust+ is a suitable framework for handling multimodal data. Meanwhile, the successful implementation of mask in inClust+ means that it can be applied to other deep learning methods with similar encoder-decoder architecture to broaden the application scope of these models.

Multi-modal tumor segmentation methods based on deep learning: a narrative review.

Automatic tumor segmentation is a critical component in clinical diagnosis and treatment. Although single-modal imaging provides useful information, multi-modal imaging provides a more comprehensive understanding of the tumor. Multi-modal tumor segmentation has been an essential topic in medical image processing. With the remarkable performance of deep learning (DL) methods in medical image analysis, multi-modal tumor segmentation based on DL has attracted significant attention. This study aimed to provide an overview of recent DL-based multi-modal tumor segmentation methods. In in the PubMed and Google Scholar databases, the keywords "multi-modal", "deep learning", and "tumor segmentation" were used to systematically search English articles in the past 5 years. The date range was from 1 January 2018 to 1 June 2023. A total of 78 English articles were reviewed. We introduce public datasets, evaluation methods, and multi-modal data processing. We also summarize common DL network structures, techniques, and multi-modal image fusion methods used in different tumor segmentation tasks. Finally, we conclude this study by presenting perspectives for future research. In multi-modal tumor segmentation tasks, DL technique is a powerful method. With the fusion methods of different modal data, the DL framework can effectively use the characteristics of different modal data to improve the accuracy of tumor segmentation.

Development of the Multimodal Handling Interface Based on Google API

Today, Artificial Intelligence is a daily routine, becoming deeply entrenched in our lives. One of the most popular and rapidly advancing technologies is speech recognition, which forms an integral part of the broader concept of multimodal data handling. Multimodal data encompasses voice, audio, and text data, constituting a multifaceted approach to understanding and processing information. This paper presents the development of a multimodal handling interface leveraging Google API technologies. The interface aims to facilitate seamless integration and management of diverse data modalities, including text, audio, and video, within a unified platform. Through the utilization of Google API functionalities, such as natural language processing, speech recognition, and video analysis, the interface offers enhanced capabilities for processing, analysing, and interpreting multimodal data. The paper discusses the design and implementation of the interface, highlighting its features and functionalities. Furthermore, it explores potential applications and future directions for utilizing the interface in various domains, including healthcare, education, and multimedia content creation. Overall, the development of the multimodal handling interface based on Google API represents a significant step towards advancing multimodal data processing and enhancing user experience in interacting with diverse data sources.

Comparative Study on BEV Vision and LiDAR Point Cloud Data Fusion Methods

With the gradual maturity of autonomous driving technology, efficient fusion and processing of multimodal sensor data has become an important research direction. This study mainly explores the strategy of integrating BEV (Bird's Eye View) vision with LiDAR point cloud data. We evaluated the performance and applicability of each of the three main data fusion methods through in-depth comparison: early fusion, mid-term fusion, and late fusion. First of all, we summarize the working principle and data characteristics of BEV vision and LiDAR, and emphasize their key roles in auto drive system. Then, the theoretical basis and implementation methods of the three fusion strategies were described in detail. The experimental results show that different fusion strategies exhibit their own advantages in different application scenarios and requirements. For example, early fusion performs well in high-precision tasks, but has a high demand for computing resources. And mid-term fusion is more suitable in scenarios with high real-time requirements. Overall, this study provides in-depth insights and practical suggestions on the fusion of BEV vision and LiDAR data in the field of autonomous driving, laying a solid foundation for future research and applications.

Semantic speech analysis using machine learning and deep learning techniques: a comprehensive review

AbstractHuman cognitive functions such as perception, attention, learning, memory, reasoning, and problem-solving are all significantly influenced by emotion. Emotion has a particularly potent impact on attention, modifying its selectivity in particular and influencing behavior and action motivation. Artificial Emotional Intelligence (AEI) technologies enable computers to understand a user's emotional state and respond appropriately. These systems enable a realistic dialogue between people and machines. The current generation of adaptive user interference technologies is built on techniques from data analytics and machine learning (ML), namely deep learning (DL) artificial neural networks (ANN) from multimodal data, such as videos of facial expressions, stance, and gesture, voice, and bio-physiological data (such as eye movement, ECG, respiration, EEG, FMRT, EMG, eye tracking). In this study, we reviewed existing literature based on ML and data analytics techniques being used to detect emotions in speech. The efficacy of data analytics and ML techniques in this unique area of multimodal data processing and extracting emotions from speech. This study analyzes how emotional chatbots, facial expressions, images, and social media texts can be effective in detecting emotions. PRISMA methodology is used to review the existing survey. Support Vector Machines (SVM), Naïve Bayes (NB), Random Forests (RF), Recurrent Neural Networks (RNN), Logistic Regression (LR), etc., are commonly used ML techniques for emotion extraction purposes. This study provides a new taxonomy about the application of ML in SER. The result shows that Long-Short Term Memory (LSTM) and Convolutional Neural Networks (CNN) are found to be the most useful methodology for this purpose.

Understanding the effect of group emotions on consumer instant order cancellation behavior in livestreaming E-commerce: Empirical evidence from TikTok

Livestreaming e-commerce (LSE) exhibits a noteworthy idiosyncrasy in the form of escalating instant order cancellations, wherein consumers swiftly revoke their purchases during livestreaming before receiving the products. This study leveraged the emotional contagion theory to construct an econometric model and investigate the factors influencing viewers' instant order cancellation behaviors. An extensive dataset was assembled, comprising 4984 product introduction video segments from 768 TikTok livestreaming rooms in China, along with 1.29 million bullet screen comments and 513,551 viewers' information. Employing advanced multimodal data processing technologies, the study extracted anchors' (livestreaming hosts) multimodal emotion features. The empirical findings demonstrate the significant impact of anchors' emotions, encompassing body movements, facial expressions, voices, and conversation contents, on the arousal and valence of group emotions, where the latter exhibit opposing effects on instant order cancellation behavior. This research establishes the paramount roles of anchor and group emotions in viewers' instant order cancellation behaviors, lending credence to the epistemological applicability of emotional contagion theory in the context of LSE. The study's findings offer LSE practitioners practical implications to enhance business decision-making, fostering more engaging and emotionally resonant livestreaming experiences to reduce cancellations and improve customer retention.

Multimodal video and IMU kinematic dataset on daily life activities using affordable devices

Human activity recognition and clinical biomechanics are challenging problems in physical telerehabilitation medicine. However, most publicly available datasets on human body movements cannot be used to study both problems in an out-of-the-lab movement acquisition setting. The objective of the VIDIMU dataset is to pave the way towards affordable patient gross motor tracking solutions for daily life activities recognition and kinematic analysis. The dataset includes 13 activities registered using a commodity camera and five inertial sensors. The video recordings were acquired in 54 subjects, of which 16 also had simultaneous recordings of inertial sensors. The novelty of dataset lies in: (i) the clinical relevance of the chosen movements, (ii) the combined utilization of affordable video and custom sensors, and (iii) the implementation of state-of-the-art tools for multimodal data processing of 3D body pose tracking and motion reconstruction in a musculoskeletal model from inertial data. The validation confirms that a minimally disturbing acquisition protocol, performed according to real-life conditions can provide a comprehensive picture of human joint angles during daily life activities.

Open access dataset integrating EEG and fNIRS during Stroop tasks

Conflict monitoring and processing are crucial components of the human cognitive system, with significant implications for daily life and the diagnosis of cognitive disorders. The Stroop task, combined with brain function detection technology, has been widely employed as a classical paradigm for investigating conflict processing. However, there remains a lack of public datasets that integrate Electroencephalogram (EEG) and functional Near-infrared Spectroscopy (fNIRS) to simultaneously record brain activity during a Stroop task. We introduce a dual-modality Stroop task dataset incorporating 34-channel EEG (sampling frequency is 1000 Hz) and 20-channel high temporal resolution fNIRS (sampling frequency is 100 Hz) measurements covering the whole frontal cerebral cortex from 21 participants (9 females/12 males, aged 23.0 ± 2.3 years). Event-related potential analysis of EEG recordings and activation analysis of fNIRS recordings were performed to show the significant Stroop effect. We expected that the data provided would be utilized to investigate multimodal data processing algorithms during cognitive processing.

Age of Incorrect Information in Semantic Communications for NOMA Aided XR Applications

As an evolving successor to the mobile Internet, the extended reality (XR) devices can generate a fully digital immersive environment similar to the real world, integrating integrating virtual and real-world elements. However, in addition to the difficulties encountered in traditional communications, there emerge a range of new challenges such as ultra-massive access, real-time synchronization as well as unprecedented amount of multi-modal data transmission and processing. To address these challenges, semantic communications might be harnessed in support of XR applications, whereas it lacks a practical and effective performance metric. For broadening a new path for evaluating semantic communications, in this paper, we construct a multi-user uplink non-orthogonal multiple access (NOMA) system to analyze its transmission performance by harnessing a novel metric called age of incorrect information (AoII). First, we derive the average semantic similarity of all the users based on DeepSC and obtain the closed-form expressions for the packets' age of information (AoI) relying on queue theory. Besides, we formulate a non-convex optimization problem for the proposed AoII which combines both error-and AoI-based performance under the constraints of semantic rate, transmit power and status update rate. Finally, in order to solve the problem, we apply an exact linear search based algorithm for finding the optimal policy. Simulation results show that the AoII metric can beneficially evaluate both the error- and AoI-based transmission performance simultaneously.

Prediction of Seedling Oilseed Rape Crop Phenotype by Drone-Derived Multimodal Data

In recent years, unmanned aerial vehicle (UAV) remote sensing systems have advanced rapidly, enabling the effective assessment of crop growth through the processing and integration of multimodal data from diverse sensors mounted on UAVs. UAV-derived multimodal data encompass both multi-source remote sensing data and multi-source non-remote sensing data. This study employs Image Guided Filtering Fusion (GFF) to obtain high-resolution multispectral images (HR-MSs) and selects three vegetation indices (VIs) based on correlation analysis and feature reduction in HR-MS for multi-source sensing data. As a supplement to remote sensing data, multi-source non-remote sensing data incorporate two meteorological conditions: temperature and precipitation. This research aims to establish remote sensing quantitative monitoring models for four crucial growth-physiological indicators during rapeseed (Brassica napus L.) seedling stages, namely, leaf area index (LAI), above ground biomass (AGB), leaf nitrogen content (LNC), and chlorophyll content (SPAD). To validate the monitoring effectiveness of multimodal data, the study constructs four model frameworks based on multimodal data input and employs Support Vector Regression (SVR), Partial Least Squares (PLS), Backpropagation Neural Network (BPNN), and Nonlinear Model Regression (NMR) machine learning models to create winter rapeseed quantitative monitoring models. The findings reveal that the model framework, which integrates multi-source remote sensing data and non-remote sensing data, exhibits the highest average precision (R2 = 0.7454), which is 28%, 14.6%, and 3.7% higher than that of the other three model frameworks, enhancing the model’s robustness by incorporating meteorological data. Furthermore, SVR consistently performs well across various multimodal model frameworks, effectively evaluating the vigor of rapeseed seedlings and providing a valuable reference for rapid, non-destructive monitoring of winter rapeseed.

Soft Tactile Sensor With Multimodal Data Processing for Texture Recognition

Soft Tactile Sensor With Multimodal Data Processing for Texture Recognition

QNMF: A quantum neural network based multimodal fusion system for intelligent diagnosis

The Internet of Medical Things (IoMT) has emerged as a significant research area in the medical field, enabling the transmission of various types of data to the cloud for analysis and diagnosis. Fusing data from multiple modalities can enhance accuracy but requires substantial computing power. Theoretically, quantum computers can rapidly process large volumes of high-dimensional medical data. Despite accelerated developments in quantum computing, research on quantum machine learning (QML) for multimodal data processing remains limited. Considering these factors, this paper presents a quantum neural network-based multimodal fusion system for intelligent diagnosis (QNMF) that can process multimodal medical data transmitted by IoMT devices, fuse data from different modalities, and improve the performance of intelligent diagnosis. This system employs a quantum convolutional neural network (QCNN) to efficiently extract features from medical images. These QCNN-based features are then fused with other modality features (such as blood test results or breast cell slices), and used to train an effective variational quantum classifier (VQC) for intelligent diagnosis. The experimental results demonstrate that a QCNN can effectively extract image data features. Furthermore, QNMF achieved an accuracy of 97.07% and 97.61% on breast cancer diagnosis and Covid-19 diagnosis experiments, respectively. In addition, the QNMF exhibits strong quantum noise robustness.

Models and Algorithms for Multimodal Data Processing

Information technologies and computer equipment are used in almost all areas of activity, which is why new areas of their use are emerging, and the level of ICT implementation is deepening, with more and more functions that were the prerogative of humans being assigned to computers. As science and technology develop, new technologies and technical means are emerging that enable a human-centered approach to software development, better adaptation of human-machine interfaces to user needs, and an increase in the ergonomics of software products, etc. These measures contribute to the formation of fundamentally new opportunities for presenting and processing information about real-world objects with which an individual interacts in production, educational and everyday activities in computer systems. The article aims to identify current models and algorithms for processing multimodal data in computer systems based on a survey of company employees and to analyze these models and algorithms to determine the benefits of using models and algorithms for processing multimodal data. Research methods: comparative analysis; systematization; generalization; survey. Results. It has been established that the recommended multimodal data representation models (the mixed model, the spatiotemporal linked model, and the multilevel ontological model) allow for representing the digital twin of the object under study at differentiated levels of abstraction, and these multimodal data processing models can be combined to obtain the most informative way to describe the physical twin. As a result of the study, it was found that the "general judgment of the experience of using models and algorithms for multimodal data processing" was noted by the respondents in the item "Personally, I would say that models and algorithms for multimodal data processing are practical" with an average value of 8.16 (SD = 0 1.70), in the item "Personally, I would say that models and algorithms for multimodal data processing are understandable (not confusing)" with an average value of 7.52. It has been determined that respondents positively evaluate (with scores above 5.0) models and algorithms for processing multimodal data in work environments as practical, understandable, manageable, and original. columns finish at the same distance from the top of the page.

PhiPipe: A multi-modal MRI data processing pipeline with test-retest reliability and predicative validity assessments.

Magnetic resonance imaging (MRI) has been one of the primary instruments to measure the properties of the human brain non-invasively in vivo. MRI data generally needs to go through a series of processing steps (i.e., a pipeline) before statistical analysis. Currently, the processing pipelines for multi-modal MRI data are still rare, in contrast to single-modal pipelines. Furthermore, the reliability and validity of the output of the pipelines are critical for the MRI studies. However, the reliability and validity measures are not available or adequate for almost all pipelines. Here, we present PhiPipe, a multi-modal MRI processing pipeline. PhiPipe could process T1-weighted, resting-state BOLD, and diffusion-weighted MRI data and generate commonly used brain features in neuroimaging. We evaluated the test-retest reliability of PhiPipe's brain features by computing intra-class correlations (ICC) in four public datasets with repeated scans. We further evaluated the predictive validity by computing the correlation of brain features with chronological age in three public adult lifespan datasets. The multivariate reliability and predictive validity of the PhiPipe results were also evaluated. The results of PhiPipe were consistent with previous studies, showing comparable or better reliability and validity when compared with two popular single-modality pipelines, namely DPARSF and PANDA. The publicly available PhiPipe provides a simple-to-use solution to multi-modal MRI data processing. The accompanied reliability and validity assessments could help researchers make informed choices in experimental design and statistical analysis. Furthermore, this study provides a framework for evaluating the reliability and validity of image processing pipelines.