Multimodal Solution Research Articles

New robotic tools for multimodal non-destructive analysis and characterization of 2D and 3D objects

The forensic field has until now been missing more versatile equipment for non-destructive characterization, analysis and inspection of 2D and 3D objects. Also, the need for increasingly frequent analysis of art forgeries, where non-destructive analysis is required at least in the first step, is calling for a multimodal solution. A prototype device for robotic analysis, imaging and mapping of 3D objects is being developed and tested to be used in these areas. The system is based on the principle of integrating imaging and analytical technologies onto six-axis robotic arms, which allow substantial flexibility in the sample size or shape. The system enables non-destructive examination of a wide spectrum of samples with complicated curvatures. The new generation of X-ray imaging detectors provide a high picture quality with a spatial resolution level in the micrometre range in 2D or 3D imaging. The basic version of the robotic scanner allows transmission X-ray imaging and mapping of the individual photons with high-sensitivity and high-resolution detectors. The broad capabilities of XRD imaging are now being complemented by X-ray fluorescence point analysis and mapping, multispectral macro imaging, and multispectral X-ray diffraction analysis.

Considering the imperfect cooperation among workers in the two-sided partial disassembly line balancing problem and the corresponding multi-modal multi-objective solution algorithm

Considering the imperfect cooperation among workers in the two-sided partial disassembly line balancing problem and the corresponding multi-modal multi-objective solution algorithm

Analysis of passenger perception heterogeneity and differentiated service strategy for air-rail intermodal travel

Air-rail intermodal services (ARISs) represent a highly promising multimodal solution within the transportation sector. Nonetheless, various uncertainties and challenges persist across multiple dimensions of air-rail interline travel, with discrepancies in passenger perceptions being a notable aspect. In an effort to pinpoint the pivotal factors contributing to these disparities among distinct passenger profiles, this study employs the Structural Equation Modeling-Multiple Indicator Multiple Cause-Artificial Neural Network (SEM-MIMIC-ANN) methodology. This approach explores the impact of numerous attributes on passenger perceptions in the context of air-rail intermodal travel, leveraging questionnaire data gathered from Shijiazhuang multimodal passengers. Furthermore, the study utilizes the Classification and Regression Tree (CART) decision tree algorithm to categorize actual passengers into distinct characteristic groups. Subsequently, the perception levels of these diverse passenger groups are quantified through the calculation of comprehensive evaluation function values. In conclusion, taking into account the real-world conditions of air-rail interline travel, this research formulates a tailored service strategy aimed at enhancing the overall passenger experience.

Multimodal spatiotemporal aggregation for point cloud accumulation

Point cloud accumulation is a crucial technique in point cloud analysis, facilitating various downstream tasks like surface reconstruction. Current methods merely rely on raw LiDAR points, yielding unsatisfactory performance due to the limited geometric information, particularly in complex scenarios characterized by intricate motions, diverse objects, and an increased number of frames. In this paper, we introduce camera modality data, which is usually acquired alongside LiDAR data at minimal expense. To this end, we present the Multimodal Spatiotemporal Aggregation solution (termed MSA) to thoroughly explore and aggregate these two distinct modalities (sparse 3D points and multi-view 2D images). Concretely, we propose a multimodal spatial aggregation module to bridge the data gap between different modalities in the Bird’s-Eye-View (BEV) space and further fuse them by learnable adaptive channel-wise weights. By assembling their respective strengths, this module generates a reliable and consistent scene representation. Subsequently, we design a temporal aggregation module to capture continuous motion information across consecutive sequences, which is beneficial for identifying the motion state of the foreground scene and enabling the model to extend robustly to longer sequences. Experiments demonstrate MSA outperforms state-of-the-art (SoTA) point cloud accumulation methods across all evaluation metrics in the public benchmark, especially with more frames.

Dynamic learning-based search for multi-criteria itinerary planning

Travelers expect integrated and multimodal itinerary planning while addressing their individual expectations. Besides common preferences such as travel time and price, further criteria such as walking and waiting times are of importance as well. The competing features of these preferences yield a variety of non-dominated itineraries. Finding the set of non-dominated multimodal travel itineraries in efficient run time remains a challenge in case multiple traveler preferences are considered.In this work, we present a sampling framework to approximate the set of non-dominated travel itineraries that scales well in terms of considered preferences. In particular, we guide the search process dynamically to uncertain areas of the complex multimodal solution space. To this end, we learn the structure of the Pareto front during the search with Gaussian Process Regression (GPR). The GPR sampling framework is evaluated integrating an extensive amount of real-world data on mobility services. We analyze long-distance trips between major cities in Germany. Furthermore, we take up to five traveler preferences into account. We observe that the framework performs well, revealing the origin and destination specifics of Pareto fronts of multimodal travel itineraries.

LOOP: An observational, prospective, multicenter study of patients with non-small cell lung cancer in the US receiving standard-of-care and initiating an approved therapy with risk of pneumonitis/ILD.

TPS12157 Background: A wide range of oncology treatments are associated with a risk of pneumonitis or ILD, which is an uncommon but serious side effect with potential for significant morbidity and mortality (Conte et al 2022). The development of pneumonitis/ILD remains unpredictable and relatively uncommon in a single practice or hospital setting. Risk of drug-induced pneumonitis/ILD remains a limiting factor to prescribing life-saving treatments. A non-invasive, patient-centric multimodal solution for remote data capture could support recognition and tracking of pulmonary symptoms in addition to providing insight for the clinical team to personalize the monitoring of patients according to their medical history or specific physiological markers. Methods: LOOP is an observational, prospective, multicenter study conducted in the US. Stage III and IV NSCLC patients receiving SoC and initiating treatment with an FDA-approved ICI (alone or in combination with other agents), ADC, or small molecule EGFR inhibitor are eligible to participate. The primary objective is to enable the development and refinement of an algorithm including documentation of its performance (specificity, sensitivity and odd ratio to inform risk of pulmonary/respiratory/thoracic-related events including ILD). A secondary objective proposes to characterize the risk factors, signs, and symptoms leading to onset, diagnosis, and treatment of pneumonitis/ILD in patients receiving SoC. The study will collect patient-generated data outside of point-of-care for six months, using questionnaires and a medical pulse-oximeter device . Clinical data available per SoC in the medical records will be collected. Approximately 600 patients will be enrolled over 13 months to accrue approximately 100 pneumonitis/ILD events. Events will include all grades of pneumonitis, including ILD and radiation pneumonitis, as determined by the Investigator. Qualitative interviews will be conducted to inform a more granular understanding of the patient specific experience of developing ILD. Logistic regression, area under curve, descriptive statistics and relevant visualizations will map the relationship between pneumonitis/ILD and patient’s health status. The study is currently enrolling; the protocol was approved by a central IRB. Clinical trial information: NCT06192004 .

An extended incremental technique for solving economic dispatch with practical considerations

This work develops a mixed-integer incremental model to solve economic dispatch (ED) with practical challenges of prohibited operating zones (POZs), transmission losses, and valve loading effects (VLE). The first challenge, the POZ, disconnects ED feasible space. The second challenge arises from transmission losses and imposes bilinear terms on ED. The VLE, in the third challenge, renders the ED objective function non-smooth and nonconvex. Due to the practical challenges, a disconnected and multimodal solution space arises, requiring a tailored solver. This work presents an extended incremental (Ext-Inc) technique to fulfill the requirement. The Ext-Inc enforces the POZs by using auxiliary variables and logical constraints. Moreover, the Ext-Inc uses linear combinatorial terms to represent the nonconvex VLEs and bilinear terms. The Ext-Inc is evaluated through ED case studies. The Ext-Inc finds the optimal cost of 1,016,279 $/h, 24,512 $/h, 62,456 $/h, 32,704 $/h, 121,412 $/h, 653,645 $/h in 10-unit, 13-unit, 15-unit, 20-unit, 40-unit, and 200-unit respectively. These findings either replicate or enhance the solution documented in previous studies, demonstrating the effectiveness of the Ext-Inc in addressing real-world challenges. Moreover, the proposed method demonstrates remarkable efficiency, solving single-period ED case studies in under 3 seconds and the multi-period case study within one minute.

Crack detection and dimensional assessment using smartphone sensors and deep learning

This paper addresses the crucial need for effective crack detection and dimensional assessment in civil infrastructure materials to ensure safety and functionality. It proposes a cost-effective crack detection and dimensional assessment solution by applying state-of-the-art deep learning on smartphone sensor imagery and positioning data. The proposed methodology integrates three-dimensional (3D) data from LiDAR sensors with Mask R-convolutional neural network (CNN) and YOLOv8 object detection networks, for automated crack detection in concrete structures, allowing for accurate measurement of crack dimensions, including length, width, and area. The study finds that YOLOv8 produces superior precision and recall results in crack detection compared to Mask R-CNN. Furthermore, the calculated crack-straight-length closely aligns with the ground-truth straight-length, with an average error of 1.5%. These research contributions include developing a multi-modal solution combining LiDAR observations with image masks for precise 3D crack measurements, establishing a dimensional assessment pipeline to convert segmented cracks into measurements, and comparing state-of-the-art CNN-based networks for crack detection in real-life images.

Multi-modal Battle Royale optimizer

Multimodal optimization poses a challenging problem in the field of optimization as it entails the discovery of multiple local and global optima, unlike unimodal optimization, which seeks a single global solution. In recent years, the significance of addressing multimodal optimization challenges has grown due to the real-world complexity of many problems. While numerous optimization methods are available for unimodal problems, multimodal optimization techniques have garnered increased attention. However, these approaches often grapple with a common issue: the determination of the niching parameter, necessitating prior knowledge of the problem space. This paper introduces a novel multimodal optimization approach that circumvents the need for prior problem space knowledge and avoids the challenge of predefining the niching parameter. Building upon the Battle Royal Optimization (BRO) algorithm, this extended version formulates a multimodal solution by utilizing Coulomb's law to identify suitable neighbors. The incorporation of Coulomb's law serves the dual purpose of identifying potential local and global optima based on fitness values and establishing optimal distances from solution candidates. A comparison study was done between the MBRO and seven well-known multimodal optimization algorithms using 14 benchmark problems from the CEC 2013 and CEC 2015 competitions to see how well it worked. The experimental results underscore MBRO's proficiency in successfully identifying most, if not all, local and global optima, positioning it as a superior solution when compared to its competitors.

Directivity of horns mounted in finite enclosures: A multimodal formulation.

The beamwidth is a primary directivity metric for the design of a constant directivity horn. To date, investigations on this property have predominantly been restricted to the half-space radiation or idealized geometries. This paper examines the beamwidth behavior of an axisymmetric horn mounted in a finite cylindrical enclosure by proposing an elegant multimodal solution to the far-field directivity pattern. The variation of beamwidth is examined for the frequency, dimensions of the enclosure, and shape of the horn. At low frequencies, a fitted model is proposed to precisely depict the intrinsic beam narrowing governed by the enclosure diffraction. The asymptotic behavior of the beamwidth is explored as the flange width increases. In the high-frequency range, the horn profile is a determinant of the directivity characteristics. We report the possibility of extending the bandwidth of a constant directivity horn by leveraging the enclosure diffraction effects. The proposed analytical method is highly accurate and much faster than the finite element method for wideband analysis. It allows for an arbitrary velocity distribution at the mouth of the horn and incorporates idealized flange configurations such as an infinite baffle, a zero-thickness closed baffle, and an infinitely long enclosure as limit cases.

Multi-modal deep learning networks for RGB-D pavement waste detection and recognition

To create a clean living environment, governments around the world have hired a large number of workers to clean up waste on pavements, which is inefficient for waste management. To better alleviate this problem, relevant scholars have proposed several deep learning methods based on RGB images to achieve waste detection and recognition. Considering the limitations of color images, we propose an efficient multi-modal learning solution for pavement waste detection and recognition. Specifically, we construct a high-quality outdoor pavement waste dataset called OPWaste, which is more in line with real needs. Compared to other waste datasets, OPWaste dataset not only has the advantages of rich background and high diversity, but also provides color and depth images. Meanwhile, we explore six different multi-modal fusion methods and propose a novel multi-modal multi-scale network (MM-Net) for RGB-D waste detection and recognition. MM-Net introduces a novel multi-scale refinement module (MRM) and multi-scale interaction module (MIM). MRM can effectively refine critical features using attention mechanisms. MIM can gradually realize information interaction between hierarchical features. In addition, we select several representative methods and perform comparative experiments. Experimental results show that MM-Net based on the image addition fusion method outperforms other deep learning models and reaches 97.3% and 84.4% on mAP0.5 and AR metrics. In fact, multi-modal learning plays an important role in intelligent waste recycling. As a promising auxiliary tool, our solution can be applied to intelligent cleaning robots for automatic outdoor waste management.

Geriatric Pain Protocol: Impact of Multimodal Pain Care for Elderly Orthopedic Trauma Patients

ABSTRACT Hip fractures are costly, and associated complications are the leading cause of injury-related deaths in persons 65 years or older. Uncontrolled pain leads to increased hospital length of stay (LOS), delayed physical therapy, and long-term functional impairment. The Geriatric Pain Protocol (GPP) is Cedars-Sinai's multimodal pain management solution, addressing the needs of older adult inpatients who have suffered fractures. Study participants included hip fracture patients admitted between February 1, 2019, and March 5, 2021. Inclusion criteria were patients 65 years or older with a hip fracture sustained from a ground-level fall and surgical candidate. Participants were divided into 2 categories: Geriatric Fracture Program (GFP) and non-GFP, with physician participation in the GFP being the differentiating factor. End points included postoperative pain, postoperative opioid utilization, LOS, complications, and 30-day readmission rates. The GPP decreased morphine milligram equivalent (MME) daily totals on days 1 and 2 and improved pain management compared with non-GPP patients. MMEs were lower in the GPP group than in the non-GPP group for both postoperative day 1 (POD1) (P = 0.007) and POD2 (P = 0.043); the Numerical Rating Scale (NRS) pain score on POD1 was lower in the GPP group (vs non-GPP, P = 0.013). There were no group differences in NRS POD2 pain or complications (all Ps > 0.1). The study sample (N = 453) had no significant difference between sex and LOS (all Ps > 0.3). Although not statistically significant, the 30-day readmission rate trended lower in patients treated in accordance with the GPP. Use of the GPP reduced pain levels and MME totals.

Routing of multi-modal autonomous vehicles for system optimal flows and average travel cost equilibrium over time

In the emerging era of autonomous vehicles (AVs), an important question is how to integrate AVs in a multi-modal framework. Recent studies on Transportation Management Centres (TMCs) for future AVs addressed the issue of providing system optimal (SO) solution. However, none of them considered the possibility of developing a multi-modal SO-based solution. In this regard, we propose a novel multi-modal Transportation Management Centre (MMTMC) for future AVs, which controls both the routing and mode use of each traveller and distributes the AVs in such a way that SO flows are maintained for the whole network every day, while travellers on the same origin-destination (OD) have the same Multi-modal Average Travel Time (MMAVT) over a cycle or period of time (in days). That is, travellers on the same OD may encounter different travel times or costs on different days, but over the planning cycle of many days, their travel times will all average to be the same MMAVT, hence preserving the MMAVT equilibrium. Travellers will travel on single occupancy vehicles (SOVs) or high occupancy vehicles (HOVs) on different routes for different days within the planning cycle, while maintaining the same MMAVT overall for the cycle. Travellers on SOVs will experience no crowdedness while on HOVs will experience crowdedness. We model both the cases of minimum system time and minimum system cost (including crowdedness cost), and provide link-based solution first for both cases. For minimum system time, we apply the conventional Frank Wolfe algorithm to generate feasible link flows, whereas for the minimum system cost problem, we incorporate a sensitivity-based equilibrium procedure to derive the SOV, HOV link flows and demand profiles. Afterwards, we utilize the most likely path method to compute unique SOV and HOV path flows and solve the MMAVT equilibrium. Then, we investigate the case of mixed-equilibrium (ME), where some travellers are given the option not to subscribe to the MMTMC and do their own private routing, i.e. user equilibrium (UE) travellers, who are penalized by a toll. The tolls imposed on these MMTMC non-subscribers are defined by the travel time/cost difference between the MMAVT of subscribers and UE users based on their value-of-time (VOT) distribution. This study investigates the properties of this MMTMC routing pattern, the relationship between MMAVT, tolling, and SOV and HOV market penetration, which offers insights to integrate AVs in a multi-modal framework.

Movie tag prediction: An extreme multi-label multi-modal transformer-based solution with explanation

Movie tag prediction: An extreme multi-label multi-modal transformer-based solution with explanation

Autofusion: Fusing Multi-Modalities with Interactions

In the context of escalating data flows through diverse channels, multimodal machine learning holds the potential to simultaneously process varied data formats from multiple sources, offering robust solutions for uncertainty in various applications. The study highlights the often under-explored correlation information among data modalities and emphasizes the importance of disentangling enriched interactions for more informed decision-making. This paper navigates the burgeoning field of multimodal artificial intelligence (AI) by proposing Autofusion, a pioneering framework addressing representation learning and fusion challenges. Our proposed approach integrates autoencoder structures to address overfitting issues in unimodal machine learning, simultaneously tackling information balancing challenges. The framework’s application in Alzheimer’s disease detection, using DementiaBank’s Pitt corpus, demonstrates promising results, outperforming unimodal methods and showcasing a substantial advantage over traditional fusion techniques. This research significantly contributes by introducing Autofusion as a comprehensive multimodal machine learning solution, demonstrating its efficacy through DementiaBank’s Pitt corpus to detect Alzheimer’s disease, and shedding light on the influential role of cross-modality interaction for enhanced performance in complex applications.

Multimodal Deep Learning for Robust Road Attribute Detection

Automatic inference of missing road attributes (e.g., road type and speed limit) for enriching digital maps has attracted significant research attention in recent years. A number of machine learning-based approaches have been proposed to detect road attributes from GPS traces, dash-cam videos, or satellite images. However, existing solutions mostly focus on a single modality without modeling the correlations among multiple data sources. To bridge this gap, we present a multimodal road attribute detection method, which improves the robustness by performing pixel-level fusion of crowdsourced GPS traces and satellite images. A GPS trace is usually given by a sequence of location, bearing, and speed. To align it with satellite imagery in the spatial domain, we render GPS traces into a sequence of multi-channel images that simultaneously capture the global distribution of the GPS points, the local distribution of vehicles’ moving directions and speeds, and their temporal changes over time, at each pixel. Unlike previous GPS-based road feature extraction methods, our proposed GPS rendering does not require map matching in the data preprocessing step. Moreover, our multimodal solution addresses single-modal challenges such as occlusions in satellite images and data sparsity in GPS traces by learning the pixel-wise correspondences among different data sources. On top of this, we observe that geographic objects and their attributes in the map are not isolated but correlated with each other. Thus, if a road is partially labeled, then the existing information can be of great help on inferring the missing attributes. To fully use the existing information, we extend our model and discuss the possibilities for further performance improvement when partially labeled map data is available. Extensive experiments have been conducted on two real-world datasets in Singapore and Jakarta. Compared with previous work, our method is able to improve the detection accuracy on road attributes by a large margin.

Exploring the Elusive Mind: A Multimodal Wearable Sensor Solution for Measuring Mind Wandering in University Students

AbstractMind‐wandering is a typical daily phenomenon during which attention shifts from external stimuli to internal trains of thought. It affect students' learning by impairing comprehension, diminishing academic achievement, impeding critical thinking, and encouraging a lack of attention and engagement in classroom activities. This study aims to introduce a new method of detecting and tracking mind wandering in university students. This approach involves using wearable sensors, including galvanic skin response (GSR), photoplethysmography (PPG), and eye‐trackers, along with machine learning techniques. The study provides a proof of concept for this multisensory approach. The association between longer fixation duration and mind wandering, and the influence of an instructor's presence on fixation allocation, and, consequently, the frequency and occurrence of mind wandering is investigated. Furthermore, the feasibility of using eye‐trackers in conjunction with GSR and PPG sensors for detecting mind wandering through a wearable multisensory data collection system is assessed. The wearable multisensory device is evaluated by ten participants (university students, males/females aged between 21‐30). Two distinct machine learning methods, support vector machine (SVM) and gated recurrent unit (GRU), are used as classification models. With sensor fusion, the SVM and GRU models yielded maximum accuracies of 86.53% and 89.86%, respectively. Moreover, participants are observed to fixate on instructors more often, just before instances of mind wandering.

Remote Assessment for ALS using Multimodal Dialog Agents: Data Quality, Feasibility and Task Compliance.

We investigate the feasibility, task compliance and audiovisual data quality of a multimodal dialog-based solution for remote assessment of Amyotrophic Lateral Sclerosis (ALS). 53 people with ALS and 52 healthy controls interacted with Tina, a cloud-based conversational agent, in performing speech tasks designed to probe various aspects of motor speech function while their audio and video was recorded. We rated a total of 250 recordings for audio/video quality and participant task compliance, along with the relative frequency of different issues observed. We observed excellent compliance (98%) and audio (95.2%) and visual quality rates (84.8%), resulting in an overall yield of 80.8% recordings that were both compliant and of high quality. Furthermore, recording quality and compliance were not affected by level of speech severity and did not differ significantly across end devices. These findings support the utility of dialog systems for remote monitoring of speech in ALS.

MyoKey: Inertial Motion Sensing and Gesture-Based QWERTY Keyboard for Extended Realities

Usability challenges and social acceptance of textual input in a context of extended realities (XR) motivate the research of novel input modalities. We investigate the fusion of inertial measurement unit (IMU) control and surface electromyography (sEMG) gesture recognition applied to text entry using a QWERTY-layout virtual keyboard. We design, implement and evaluate our proposed multi-modal solution, MyoBoard, which is an IMU-driven and EMG-based text entry system based on the QWERTY keyboard layout for XR headsets. The user can select characters with a combination of arm movements and hand gestures. MyoBoard employs a light-weight convolutional neural network classifier that can be deployed on a mobile device with insignificant inference time. We demonstrate the practicality of interruption-free text entry in mobile scenarios with MyoBoard, by recruiting 12 participants and by testing three sets of grasp micro-gestures in three scenarios - empty hand text input, input holding a pen, and a large object, such as an umbrella. With MyoBoard, users achieve an average text entry rate of 9.33 words per minute (WPM), 8.76 WPM, and 8.35 WPM for the freehand, pen, and umbrella conditions, respectively.

Alternative statistical modeling for radical prostatectomy data

Several statistical models have been proposed in recent years, among them is the semiparametric regression. In medicine, there are several situations in which it is impracticable to consider a linear regression for statistical modeling, especially when the data contain explanatory variables that present a nonlinear relationship with the response variable. Another common situation is when the response variable does not have a unimodal shape, and it is not possible to adopt distributions belonging to the symmetric or asymmetric classes. In this context, a semiparametric heteroskedastic regression is proposed based on an extension of the normal distribution. Then, we show the usefulness of this model to analyze the cost of prostate cancer surgery. The predictor variables refer to two groups of patients such that one group receives a multimodal local anesthetic solution (Preemptive Target Anesthetic Solution) and the second group is treated with neuraxial blockade (spinal anesthesia/traditional standard). The other relevant predictor variables are also evaluated, thus allowing for the in-depth interpretation of the predictor variables with a nonlinear effect on the dependent variable cost. The penalized maximum likelihood method is adopted to estimate the model parameters. The new regression is a useful statistical tool for analyzing medical data.