Multimodal Adaptation Research Articles

Multimodal Adaptation, Reconstruction, and Deviation of Immortal Ones

Since time immemorial, immortal ones, or Shenxians in Chinese, have been an integral part of the Chinese civilization, reflecting the ancient and everlasting pursuit of immortality, both physically and spiritually. In particular, the epic novel Journey to the West offers a panoramic view of their hierarchy, personality, and ways of life. For this reason, the novel has been adapted into different multimedia versions. The animated comedy Monkey King (2023), produced by Netflix, is one of the recent but less satisfactory examples, in which the images of immortals are reconstructed in a multimodal manner suffering from adaptational deviation. Based on previous studies on translation, adaptation, and multimodality, this article puts forward a model of multimodal adaptation and uses that framework to evaluate Monkey King’s major reconstructions of immortals in a transnational context. Focusing on explicit and implicit adaptations, the article discusses how the new presentations of these immortals deviate from those in Chinese culture, and how a world that would appear to be turned upside down, in the eyes of a Chinese audience, comes into being. The article also sheds some light on the effect of transnational adaptations on the culture of origin for the source text.

Wong Kar-wai’s Argentine Affair: Happy Together as a Translingual and Multimodal Adaptation

Abstract In 2021, a 4K-restored rendition of Wong Kar-wai’s iconic film Happy Together (1997) returned to theatres across Hong Kong, Taiwan, and Argentina, marking the film’s enduring transnational appeal two decades after its Cannes premiere. This article investigates the film’s intricate cultural ties to Argentina. I argue that Happy Together not only enhances the cultural connections between Hong Kong and Argentina but also showcases Wong’s translingual and multimodal approaches to film adaptation. ‘Translingual’ refers to the film’s literary source: I demonstrate how Wong skilfully wove elements from Manuel Puig's 1973 novel The Buenos Aires Affair into his film through a threefold Spanish-English-Chinese process of rewriting. Subsequently, I contend that various cultural facets of Argentina—the literature of Puig, the soundscape of Piazzolla’s tango, and the landscapes of the Iguazú Falls and Tierra del Fuego—contributed significantly to the Hong Kong auteur’s multimodal adaptation. In short, this article proposes a dialogic perspective on Hong Kong cinema by considering Argentina as a reflective mirror brimming with aesthetic and geopolitical implications. Through the lens of Happy Together, this study moves beyond Western-centric reception paradigms by spotlighting the cultural and socio-political affinities between Latin America and Asia, thus bringing to the forefront the often-overlooked transpacific connections within the Global South.

3MSF: A Multi-Modal Adaptation of the 6TiSCH Minimal Scheduling Function for the Industrial IoT.

Although wireless devices continuously gain communication capabilities, even state-of-the-art Industrial Internet of Things (IIoT) architectures, such as Internet Protocol version 6 over the Time-Slotted Channel Hopping (TSCH) mode of IEEE 802.15.4 (6TiSCH), continue to use network-wide, fixed link configurations. This presents a missed opportunity to (1) forego the need for rigorous manual setup of new deployments; and (2) provide full coverage of particularly heterogeneous and/or dynamic industrial sites. As such, we devised the Multi-Modal Minimal Scheduling Function (3MSF) for the TSCH link layer, which, combined with previous work on the routing layer, results in a 6TiSCH architecture able to dynamically exploit modern multi-modal hardware on a per-link basis through variable-duration timeslots, simultaneous transmission, and routing metric normalization. This paper describes, in great detail, its design and discusses the rationale behind every choice made. Finally, we evaluate three basic scenarios through simulations, showcasing both the functionality and flexibility of our 6TiSCH implementation.

Calibration of Transformer-Based Models for Identifying Stress and Depression in Social Media

In today’s fast-paced world, the rates of stress and depression present a surge. People use social media for expressing their thoughts and feelings through posts. Therefore, social media provide assistance for the early detection of mental health conditions. Existing methods mainly introduce feature extraction approaches and train shallow machine learning (ML) classifiers. For addressing the need of creating a large feature set and obtaining better performance, other research studies use deep neural networks or language models based on transformers. Despite the fact that transformer-based models achieve noticeable improvements, they cannot often capture rich factual knowledge. Although there have been proposed a number of studies aiming to enhance the pretrained transformer-based models with extra information or additional modalities, no prior work has exploited these modifications for detecting stress and depression through social media. In addition, although the reliability of a machine learning (ML) model’s confidence in its predictions is critical for high-risk applications, there is no prior work taken into consideration the model calibration. To resolve the above issues, we present the first study in the task of depression and stress detection in social media, which injects extra-linguistic information in transformer-based models, namely, bidirectional encoder representations from transformers (BERT) and MentalBERT. Specifically, the proposed approach employs a multimodal adaptation gate for creating the combined embeddings, which are given as input to a BERT (or MentalBERT) model. For taking into account the model calibration, we apply label smoothing. We test our proposed approaches in three publicly available datasets and demonstrate that the integration of linguistic features into transformer-based models presents a surge in performance. Also, the usage of label smoothing contributes to both the improvement of the model’s performance and the calibration of the model. We finally perform a linguistic analysis of the posts and show differences in language between stressful and nonstressful texts, as well as depressive and nondepressive posts.

LaMAR: Laplacian Pyramid for Multimodal Adaptive Super Resolution (Student Abstract)

Recent advances in image-to-image translation involve the integration of non-visual imagery in deep models. Non-visual sensors, although more costly, often produce low-resolution images. To combat this, methods using RGB images to enhance the resolution of these modalities have been introduced. Fusing these modalities to achieve high-resolution results demands models with millions of parameters and extended inference times. We present LaMAR, a lightweight model. It employs Laplacian image pyramids combined with a low-resolution thermal image for Guided Thermal Super Resolution. By decomposing the RGB image into a Laplacian pyramid, LaMAR preserves image details and avoids high-resolution feature map computations, ensuring efficiency. With faster inference times and fewer parameters, our model demonstrates state-of-the-art results.

Delving into Multimodal Prompting for Fine-Grained Visual Classification

Fine-grained visual classification (FGVC) involves categorizing fine subdivisions within a broader category, which poses challenges due to subtle inter-class discrepancies and large intra-class variations. However, prevailing approaches primarily focus on uni-modal visual concepts. Recent advancements in pre-trained vision-language models have demonstrated remarkable performance in various high-level vision tasks, yet the applicability of such models to FGVC tasks remains uncertain. In this paper, we aim to fully exploit the capabilities of cross-modal description to tackle FGVC tasks and propose a novel multimodal prompting solution, denoted as MP-FGVC, based on the contrastive language-image pertaining (CLIP) model. Our MP-FGVC comprises a multimodal prompts scheme and a multimodal adaptation scheme. The former includes Subcategory-specific Vision Prompt (SsVP) and Discrepancy-aware Text Prompt (DaTP), which explicitly highlights the subcategory-specific discrepancies from the perspectives of both vision and language. The latter aligns the vision and text prompting elements in a common semantic space, facilitating cross-modal collaborative reasoning through a Vision-Language Fusion Module (VLFM) for further improvement on FGVC. Moreover, we tailor a two-stage optimization strategy for MP-FGVC to fully leverage the pre-trained CLIP model and expedite efficient adaptation for FGVC. Extensive experiments conducted on four FGVC datasets demonstrate the effectiveness of our MP-FGVC.

Parallel wavelet networks incorporating modality adaptation for hyperspectral image super-resolution

Hyperspectral images (HSIs) have been applied to a wide range of areas thanks to their high spectral resolutions. However, spatial resolutions of HSIs are inevitably compromised due to hardware limits. While HSIs can be enhanced by the super resolution techniques, there may be still spectral distortions and spatial blurriness in the reconstructed high-resolution HSIs (HR-HSIs). Therefore, this paper proposes a Parallel Framework based on Modality-Adaptive Wavelet Networks (PF-MAWN) to reconstruct HR-HSISs by effectively fusing low-resolution HSIs (LR-HSIs) and high-resolution multispectral images (HR-MSIs). To avoid spectral artifacts, multiple MAWNs work in parallel and fuse HR-MSIs/LR-HSIs in groups of highly correlated adjacent bands. Each MAWN incorporates wavelet transform (WT) deeply into convolutional neural networks (CNN) by layers of Multidomain Preprocessing Blocks (MPBs), Multimodal Adaptation Blocks (MABs) and Multilevel Connection Blocks (MCBs). Within each MAB, there are two Feature Alignment Modules (FAMs) based on self-attention to radiometrically match HR-MSIs/LR-HSIs and an Information Control Module (ICM) based on Gaussian-weighted attention to control textural details in HR-HSIs. The framework exhibits good extensibility and generalizability on spectral images of arbitrary bands and diversified scales. Experimental results show that PF-MAWN can reconstruct high-quality HR-HSIs with different sizes of objects, ranging from microscopic to macroscopic. It outperforms several state-of-the-art methods on four public datasets of histopathology tissues, everyday stuff and remote sensing objects.

Multimodal brain tumor detection using multimodal deep transfer learning

MRI brain image analysis, including brain tumor detection, is a challenging task. MRI images are multimodal, and in recent years, multimodal medical image analysis has gotten more attention. Modes refer to data from multiple sources which are semantically correlated and sometimes provide complementary information to each other. In this paper, modalities in MRI brain images refer to the different planes of view (axial, sagittal, and coronal planes) in which MRI images are taken. In most cases, in the literature on multimodal data analysis, it is assumed that all modalities for all samples are available. While, in medical image analysis, this assumption is not valid, and only some modalities might be available for each sample. The knowledge transfer between and within modalities is considered to tackle this challenge in MRI brain image segmentation. For knowledge transfer, domain adaptation is an important step that deals with the problem of having different distributions between the training and test sets. These challenges have not been considered in recent multimodal brain image analysis studies. This paper proposed a new multimodal deep transfer learning for MRI brain image analysis. The main differences of the proposed approach, with respect to the other multimodal brain image analysis, are 1) proposing a new multimodal feature encoder and 2) proposing a new multimodal adaptation technique to handle the different distribution between the training and test sets. We applied it to IBSR and Figshre brain tumor datasets to evaluate the proposed approach. The results confirm that the proposed approach significantly outperforms the other comparable approaches.

Photoreceptor and Retinal Pigment Epithelium Relationships in Eyes With Vitelliform Macular Dystrophy Revealed by Multimodal Adaptive Optics Imaging.

PurposeTo assess the structure of cone photoreceptors and retinal pigment epithelial (RPE) cells in vitelliform macular dystrophy (VMD) arising from various genetic etiologies.MethodsMultimodal adaptive optics (AO) imaging was performed in 11 patients with VMD using a custom-assembled instrument. Non-confocal split detection and AO-enhanced indocyanine green were used to visualize the cone photoreceptor and RPE mosaics, respectively. Cone and RPE densities were measured and compared across BEST1-, PRPH2-, IMPG1-, and IMPG2-related VMD.ResultsWithin macular lesions associated with VMD, both cone and RPE densities were reduced below normal, to 37% of normal cone density (eccentricity 0.2 mm) and to 8.4% of normal RPE density (eccentricity 0.5 mm). Outside of lesions, cone and RPE densities were slightly reduced (both to 92% of normal values), but with high degree of variability in the individual measurements. Comparison of juxtalesional cone and RPE measurements (<1 mm from the lesion edge) revealed significant differences in RPE density across the four genes (P < 0.05). Overall, cones were affected to a greater extent than RPE in patients with IMPG1 and IMPG2 pathogenic variants, but RPE was affected more than cones in BEST1 and PRPH2 VMD. This trend was observed even in contralateral eyes from a subset of five patients who presented with macular lesions in only one eye.ConclusionsAssessment of cones and RPE in retinal locations outside of the macular lesions reveals a pattern of cone and RPE disruption that appears to be gene dependent in VMD. These findings provide insight into the cellular pathogenesis of disease in VMD.

SCA-MMA: Spatial and Channel-Aware Multi-Modal Adaptation for Robust RGB-T Object Tracking

The RGB and thermal (RGB-T) object tracking task is challenging, especially with various target changes caused by deformation, abrupt motion, background clutter and occlusion. It is critical to employ the complementary nature between visual RGB and thermal infrared data. In this work, we address the RGB-T object tracking task with a novel spatial- and channel-aware multi-modal adaptation (SCA-MMA) framework, which builds an adaptive feature learning process for better mining this object-aware information in a unified network. For each type of modality information, the spatial-aware adaptation mechanism is introduced to dynamically learn the location-based characteristics of specific tracking objects at multiple convolution layers. Further, the channel-aware multi-modal adaptation mechanism is proposed to adaptively learn the feature fusion/aggregation of different modalities. In order to perform object tracking, we employ a binary classification module with two fully connected layers to predict the bounding boxes of specific targets. Comprehensive evaluations on GTOT and RGBT234 datasets demonstrate the significant superiority of our proposed SCA-MMA for robust RGB-T object tracking tasks. In particular, the precision rate (PR) and success rate (SR) on GTOT and RGBT234 datasets can reach 90.5%/73.2% and 80.2%/56.9%, significantly higher than the state-of-the-art algorithms.

Persistent Dark Cones in Oligocone Trichromacy Revealed by Multimodal Adaptive Optics Ophthalmoscopy.

Dark cone photoreceptors, defined as those with diminished or absent reflectivity when observed with adaptive optics (AO) ophthalmoscopy, are increasingly reported in retinal disorders. However, their structural and functional impact remain unclear. Here, we report a 3-year longitudinal study on a patient with oligocone trichromacy (OT) who presented with persistent, widespread dark cones within and near the macula. Diminished electroretinogram (ERG) cone but normal ERG rod responses together with normal color vision confirmed the OT diagnosis. In addition, the patient had normal to near normal visual acuity and retinal sensitivity. Occasional dark gaps in the photoreceptor layer were observed on optical coherence tomography, in agreement with reflectance AO scanning light ophthalmoscopy, which revealed that over 50% of the cones in the fovea were dark, increasing to 74% at 10° eccentricity. In addition, the cone density was 78% lower than normal histologic value at the fovea, and 20–40% lower at eccentricities of 5–15°. Interestingly, color vision testing was near normal at locations where cones were predominantly dark. These findings illustrate how a retina with predominant dark cones that persist over at least 3 years can support near normal central retinal function. Furthermore, this study adds to the growing evidence that cones can continue to survive under non-ideal conditions.

Enhancement of High-Fidelity Haptic Feedback Through Multimodal Adaptive Robust Control for Teleoperated Systems

Achieving high-fidelity haptic feedback in teleoperated robotic systems is crucial, particularly in applications, such as medical, industrial, military, space, and underwater explorations. The quality of haptic feedback depends to a great extent on the controller design objectives, such as stability and transparency. In addition to the time delay, the most challenging issues that affect haptic feedback are model uncertainties, parametric variations, and external disturbances. In this article, such issues have been addressed by developing a novel multimodal adaptive robust control algorithm for teleoperated systems with time-varying delay. Here, parametric convergence is achieved swiftly with a milder condition than a stronger persistent excitation condition on the regressor signal, resulting in the enhancement of transient performance. The proposed method ensures the robustness of the system against bounded perturbations and high-frequency unmodeled dynamics using a modified terminal sliding-mode controller alongside the adaptive scheme. The proposed method promises high-fidelity haptic feedback to the operator by ensuring improved robust global asymptotic stability, synchronization, and transparency, addressing parametric variations and disturbances. The validity of the proposed adaptive robust control law is substantiated through simulations and comparisons with the existing methods for a 2-R teleoperation system. The experimental results for a 1-R teleoperation system are also presented.

Integrating Multimodal Information in Large Pretrained Transformers.

Recent Transformer-based contextual word representations, including BERT and XLNet, have shown state-of-the-art performance in multiple disciplines within NLP. Fine-tuning the trained contextual models on task-specific datasets has been the key to achieving superior performance downstream. While fine-tuning these pre-trained models is straight-forward for lexical applications (applications with only language modality), it is not trivial for multimodal language (a growing area in NLP focused on modeling face-to-face communication). Pre-trained models don't have the necessary components to accept two extra modalities of vision and acoustic. In this paper, we proposed an attachment to BERT and XLNet called Multimodal Adaptation Gate (MAG). MAG allows BERT and XLNet to accept multimodal nonverbal data during fine-tuning. It does so by generating a shift to internal representation of BERT and XLNet; a shift that is conditioned on the visual and acoustic modalities. In our experiments, we study the commonly used CMU-MOSI and CMU-MOSEI datasets for multimodal sentiment analysis. Fine-tuning MAG-BERT and MAG-XLNet significantly boosts the sentiment analysis performance over previous baselines as well as language-only fine-tuning of BERT and XLNet. On the CMU-MOSI dataset, MAG-XLNet achieves human-level multimodal sentiment analysis performance for the first time in the NLP community.

Finding Strength: Multimodal Assessment and Adaptation of Children’s Writing Into Theatre

This article examines the assessment terrains and processes employed formally and informally by a theatre-focused group of adults adapting and performing children’s original writing as plays for young audiences through a project called YOUR Stories on Stage. The journey of three anchor child-written exemplars from original form through screening into script then onto rehearsal and performance is chronicled and questioned to help articulate specific ways a theatre lens on children’s writing finds strengths and assets in the work, with particular attention to elements of multimodal literacy (Cope and Kalantzis 2009; Jewitt and Kress 2003). The same three pieces of writing are also analyzed with the well-established criteria of six-traits writing analysis (Spandel 2012) used widely in various forms by classroom teachers, school systems, and testing organizations. Complementary and contrasting patterns are identified and discussed with findings pointing to echoed connections between the two approaches, particularly in the areas of voice and word choice. A need for a wider range of assessments used in writing and literacy instruction is highlighted, including those assessments of artistic elements alone and in relation to narrow views of literacy as reading and writing.

The ‘Strange Alteration’ of Hamlet: Comic Books, Adaptation and Constructions of Adolescent literacy

This article examines two comic book adaptations of Shakespeare’s Hamlet produced for teenage readers and used in school classrooms. It seeks to understand the ways in which particular kinds of literacy are being implied and constructed through the textual practice of multimodal adaptation. It presents a close reading of sections of the texts and places them within the framework of multimodal theory and adaptation studies. The paper is part of a larger study of comic book adaptations of Shakespeare’s plays based on a series of semi-structured interviews with the producers (publishers, artists, textual editors) involved in the adaptation process. The data are analysed in order to raise questions concerning the assumptions about literacy inherent in educational texts, and to outline the importance of further qualitative research in the area.

Comics and multimodal adaptation plenary address

Abstract

An inclusive approach to develop web TV applications

PurposeThis paper aims to present a brief overview of the European “Gentle User Interfaces for Elderly People” (GUIDE) project, which is creating a software framework and design tools that allow developers to efficiently integrate accessibility and personalisation features into their applications, minimizing intervention with existing development process and tools. The GUIDE project aims to fill the accessibility, expertise, time, budget and framework gap between mainstream applications and accessibility systems through its user modelling and multimodal adaptation systems.Design/methodology/approachThe paper looks at the GUIDE project which involves users in early design process, understands their requirements, formulates them into user models and implements the model into a software framework to personalize interfaces with change of range of abilities and context of use. It proposes a research solution to industry and works closely with it so that the research can make a market impact.FindingsEarly evaluation of the system shows it can accommodate a wide range of users and enhance their quality of participation with digital media. The project also contributes to develop standards for hybrid TV and the interoperable user model.Originality/valueThe GUIDE project is a first of its kind to take an end‐to‐end approach to accessible computing through presentations on user‐centred design, accessibility solutions, a software framework for industrial developers and standardization issues for legislators and regulators.

Developing Multimodal Adaptation Algorithm for Mobility Impaired Users by Evaluating Their Hand Strength

Recent research on interactive electronic systems like computer, digital TV, smartphones can improve the quality of life of many disabled and elderly people by helping them to engage more fully to the world. Previously, a simulator was developed that reflects the effect of impairment on interaction with electronic devices and thus helps designers in developing accessible systems. In this article, the scope of the simulator has been extended to multiple pointing devices. The way that hand strength affects pointing performance of people with and without mobility impairment in graphical user interfaces was investigated for four different input modalities, and a set of linear equations to predict pointing time and average number of submovements for different devices was developed. These models were used to develop an adaptation algorithm to facilitate pointing in electronic interfaces by users with motor impairment using different pointing devices. The algorithm attracts a pointer when it is near a target and thus helps to reduce random movement during homing and clicking. The algorithm was optimized using the simulator and then tested on a real-life application with multiple distractors involving three different pointing devices. The algorithm significantly reduces pointing time for different input modalities.

Non-Retinotopic Motor-Visual Recalibration to Temporal Lag

Temporal order judgment (TOJ) between the voluntary motor action and its perceptual feedback is important in distinguishing between a sensory feedback which is caused by observer’s own action and other stimulus, which are irrelevant to that action. Prolonged exposure to fixed temporal lag between motor action and visual feedback recalibrates motor-visual temporal relationship, and consequently shifts the point of subjective simultaneity (PSS). Previous studies on the audio-visual temporal recalibration without voluntary action revealed that the low level processing is involved. However, it is not clear how the low and high level processings affect the recalibration to constant temporal lag between voluntary action and visual feedback. This study examined retinotopic specificity of the motor-visual temporal recalibration. During the adaptation phase, observers repeatedly pressed a key, and visual stimulus was presented in left or right visual field with a fixed temporal lag (0 or 200 ms). In the test phase, observers performed a TOJ for observer’s voluntary keypress and test stimulus, which was presented in the same as or opposite to the visual field in which the stimulus was presented in the adaptation phase. We found that the PSS was shifted toward the exposed lag in both visual fields. These results suggest that the low visual processing, which is retinotopically specific, has minor contribution to the motor-visual temporal lag adaptation, and that the adaptation to shift the PSS mainly depends upon the high level processing such as attention to specific properties of the stimulus.