Multimode Field Research Articles

A Multimodal Metaphorical Representation of Selected Political Cartoons in Nigeria

Complementing studies on the representation of political cartoons which apply metaphorical representation in the portrayal of societal ills, this study explores a multimodal metaphorical representation of selected political cartoons in Nigeria to investigate how cartoonists employed metaphors to represent the current political situation in the country. In the face of the ongoing election petitions and handover of office to the President, among other activities within the political domain in Nigeria, a number of political cartoonists have flooded the online platform with cartoons depicting visual metaphorical representations of happenings within the nation. Such metaphorical representations are semiotically showcased via such cartoons as Go to Court, the Scape Goat, Buhari’s Pet, Corruption, and Original Wahala Received by Me in the portrayal of reality that Nigeria faces today. Given the foregoing, this study combines linguistic and visual modes to form a multimodal representation of the metaphors used in the selected political cartoons. The study adopts a qualitative analysis of 10 political cartoons in Nigeria. It uses Kress and van Leeuwen’s (2006, 2020) approach to multimodality and Forceville’s (1996, 2016) view of visual realisations of conceptual metaphor in constructing meaning to investigate how visual images are constructed to cue conceptual metaphors. The results of the study suggest that Nigerian political cartoons rely mainly on visual metaphors as a means of communicating their messages. In addition, the metaphors used often rely on comic, exaggerated and simplistic depictions to convey their meaning effectively and to stir a strong emotional reaction from the readers. Furthermore, the findings of the study emphasise the importance of metaphorical representation in political cartoons and its implications for public discourse. The results reveal ways to improve the power of the metaphorical messages and the impact of the cartoons on public opinion. The study contributes to the multimodal metaphor research field and reveals the Nigerian public’s underlying beliefs and ideologies.

Intensive, Real-Time Data Collection of Psychological and Physiological Stress During a 96-Hour Field Training Exercise at a Senior Military College: Feasibility and Acceptability Cohort Study.

Poor physical fitness, stress, and fatigue are factors impacting military readiness, national security, and economic burden for the United States Department of Defense. Improved accuracy of wearable biosensors and remote field biologic sample collection strategies could make critical contributions to understanding how physical readiness and occupational stressors result in on-the-job and environment-related injury, sleep impairments, diagnosis of mental health disorders, and reductions in performance in war-fighters. This study aimed to evaluate the feasibility and acceptability of intensive biomarker and biometric data collection to understand physiological and psychological stress in Army Reserved Officer Training Corps cadets before, during, and after a 96-hour field training exercise (FTX). A prospective pilot study evaluated the feasibility and acceptability of multimodal field data collection using passive drool saliva sampling, sweat sensors, accelerometry, actigraphy, and photoplethysmography. In addition, physical fitness (Army Combat Fitness Test), self-reported injury, and psychological resilience (Brief Resilience Scale) were measured. A total of 22 cadets were included. Two were lost to follow-up due to injury during FTX, for a retention rate of 91%. Assessments of performance and psychological resilience were completed for all remaining participants, resulting in 100% testing adherence. All participants provided saliva samples before the FTX, with 98% adherence at the second time point and 91% at the third. For sweat, data collection was not possible. Average daily wear time for photoplethysmography devices was good to excellent, meeting a 70% threshold with data collected for ≥80% of person-days at all time points. Of the participants who completed the FTX and 12 completed a post-FTX acceptability survey for a response rate of 60%. Overall, participant acceptance was high (≥80%) for all metrics and devices. This study demonstrates that wearable biosensors and remote field biologic sample collection strategies during a military FTX have the potential to be used in higher stakes tactical environments in the future for some, but not all, of the strategies. Overall, real-time biometric and biomarker sampling is feasible and acceptable during field-based training and provides insights and strategies for future interventions on military cadet and active-duty readiness, environmental stress, and recovery.

Inverse and direct impedance eduction of liners in the MAINE Flow facility

The assessment of acoustic liners typically involves conducting experiments with airflow in a duct under various incident sound pressure levels and flow velocities. Impedance, as the most intrinsic parameter characterizing an acoustic wall (due to its independence from the duct section), is often the primary focus. Experimental techniques for its determination can be categorized into two classes: direct and inverse eduction methods. This paper contrasts these two methodological classes within the context of the MAINE Flow facility-a 150mm x 280mm rectangular duct capable of generating flow Mach numbers up to 0.63 and multimodal sound fields of Sound Pressure Levels up to 150 dB. Direct impedance determination necessitates a substantial degree of attenuation along the liner. However, in the current implementation, inverse impedance determination is constrained by the uniform flow hypothesis, which is invalid for large Mach numbers in such facilities. The findings illustrate that both methods can yield valuable impedance data, enabling the examination of the effects of flow and incident modal content on the measured impedance.

Collective Processes in Land Crafting

This article is based on the 3-day workshop, “Botanical Colour Laboratory”, which mapped the colours of the landscape of Osterøy, Norway, through land crafting with natural dyes. Land crafting builds an understanding of, and connection to, the land. This article discusses the multi-sensorial aspects and the relations between humans and non-humans in the process of collective plant dyeing. The terms: situated knowledge, orientations, the unruliness of things, production of locality and mujaawarah (neighbouring) are used to reflect on the workshop and its entangled relationships to matter, landscape, crafting and knowledge. These terms are woven together with the experiences of the “Botanical Colour Laboratory”. The research is conducted as a creative practice with an ethnographical approach. By combining written texts with video and drawings, I have made new multi-modal field notes. These field notes give a glimpse into the multifaceted processes of plant dyeing. The research shows that projects like “Botanical Colour Laboratory” are highly relevant and can influence orientations to our surroundings through collective land crafting processes with our living and non-living neighbours.

Polymer encapsulation impact on potential-induced degradation in PV modules revealed by a multi-modal field study

Potential-induced degradation is a significant issue affecting the performance and reliability of photovoltaic systems, particularly in large-scale installations. Understanding the mechanisms underlying the formation of potential-induced degradation is crucial for developing effective mitigation strategies and ensuring the long-term sustainability of solar energy technologies. During the inspection of a 1.2 MWp photovoltaic power plant, various imaging methods, electrical measurements, spectroscopic characterization methods, and monitoring data analyses were utilized and combined. We observed that two percent of the photovoltaic modules at the string ends exhibited the characteristic checkerboard pattern in infrared or electroluminescence imaging, implicating issues on twelve percent of all strings. This uneven distribution of potential-induced degradation-affected modules suggests additional key factors are at play. Investigations of the backsheet and ethylene-vinyl acetate using near-infrared reflectance analysis revealed that one backsheet type and two different ethylene-vinyl acetate types were present in the installed solar panels. Yet, only one type of ethylene-vinyl acetate correlated with potential-induced degradation. The same type also exhibited a higher rate of oxidative degradation. Scanning electron microscopy and energy-dispersive X-ray spectroscopy revealed a significantly increased sodium ion content in the ethylene-vinyl acetate and glass of the potential-induced degradation-affected modules. Our findings highlight the crucial role of polymer encapsulation in the development of potential-induced degradation and underscore the importance of careful polymer selection in the design of photovoltaic module technologies.

Thermodynamic sensing of quantum nonlinear noise correlations

Abstract We put forth the concept of quantum noise sensing in nonlinear two-mode interferometers coupled to mechanical oscillators. These autonomous machines are capable of sensing quantum nonlinear correlations of two-mode noisy fields via their thermodynamic variable of extractable work, alias work capacity (WC) or ergotropy. The fields are formed by thermal noise input via its interaction with multi-level systems inside the interferometer. Such interactions amount to the generation of two-mode quantum nonlinear gauge fields that may be partly unknown. We show that by monitoring a mechanical oscillator coupled to the interferometer, one can sense the WC of one of the output field modes and thereby reveal the quantum nonlinear correlations of the field. The proposed quantum sensing method can provide an alternative to quantum multiport interferometry where the output field is unraveled by tomography. This method may advance the simulation and control of multimode quantum nonlinear gauge fields.

Experimental photon addition and subtraction in multi-mode and entangled optical fields.

Multiple photon addition and subtraction applied to multi-mode thermal and sub-Poissonian fields as well as twin beams are mutually compared using one experimental setup. Twin beams (TWBs) with tight spatial correlations detected by an intensified CCD camera with high spatial resolution are used to prepare the initial fields. Up to three photons are added or subtracted to arrive at the nonclassical and non-Gaussian states. Only the photon-subtracted thermal states (TSs) remain classical. In general, the experimental photon-added states exhibit greater nonclassicality and non-Gaussianity than the comparable photon-subtracted states. Once photons are added or subtracted in twin beams, both processes result in comparable properties of the obtained states owing to twin-beam photon pairing.

MMCL-CPI: A multi-modal compound-protein interaction prediction model incorporating contrastive learning pre-training

MotivationCompound-protein interaction (CPI) prediction plays a crucial role in drug discovery and drug repositioning. Early researchers relied on time-consuming and labor-intensive wet laboratory experiments. However, the advent of deep learning has significantly accelerated this progress. Most existing deep learning methods utilize deep neural networks to extract compound features from sequences and graphs, either separately or in combination. Our team’s previous research has demonstrated that compound images contain valuable information that can be leveraged for CPI task. However, there is a scarcity of multimodal methods that effectively combine sequence and image representations of compounds in CPI. Currently, the use of text-image pairs for contrastive language-image pre-training is a popular approach in the multimodal field. Further research is needed to explore how the integration of sequence and image representations can enhance the accuracy of CPI task. ResultsThis paper presents a novel method called MMCL-CPI, which encompasses two key highlights: 1) Firstly, we propose extracting compound features from two modalities: one-dimensional SMILES and two-dimensional images. This approach enables us to capture both sequence and spatial features, enhancing the prediction accuracy for CPI. Based on this, we design a novel multimodal model. 2) Secondly, we introduce a multimodal pre-training strategy that leverages comparative learning on a large-scale unlabeled dataset to establish the correspondence between SMILES string and compound’s image. This pre-training approach significantly improves compound feature representations for downstream CPI task. Our method has shown competitive results on multiple datasets.

Gaussian quantum metrology for mode-encoded parameters

Quantum optical metrology aims to identify ultimate sensitivity bounds for the estimation of parameters encoded into quantum states of the electromagnetic field. In many practical applications, including imaging, microscopy, and remote sensing, the parameter of interest is not only encoded in the quantum state of the field, but also in its spatio-temporal distribution, i.e. in its mode structure. In this mode-encoded parameter estimation setting, we derive an analytical expression for the quantum Fisher information valid for arbitrary multimode Gaussian fields. To illustrate the power of our approach, we apply our results to the estimation of the transverse displacement of a beam and to the temporal separation between two pulses. For these examples, we show how the estimation sensitivity can be enhanced by adding squeezing into specific modes.

Question guided multimodal receptive field reasoning network for fact-based visual question answering

Question guided multimodal receptive field reasoning network for fact-based visual question answering

Can you hear the image speak? Primitivism, imagespeech, and audienceship

This paper proposes ‘imagespeech’ as a conceptual framework to discuss the political relationship between visual representation and distribution of voices that produce structures of (in)visibility and (in)audibility within the art historical legacies of colonialism, modernism, and nationalism. In particular, it explores the biography, materiality, and audienceship of Citra Irian (Image of Irian) – a series of artworks made between 1975 and 1985 by Sunaryo, who claims that his work resembles ‘primitive lines’ of Papuan material cultures and conveys the spiritual forces of Indonesian nationalism. In 1981, an art student named Semsar Siahaan burned down one of Sunaryo’s Citra Irian sculptures as an iconoclastic happening art act. Beyond a story of dispute between two Indonesian male artists, this paper employs imagespeech to listen to the various conflicting historical voices that resound from Citra Irian to identify by whom, how, and what Citra Irian was made to say. By asking ‘who has made an image speak?’ (a historical inquiry) and ‘what can an image say?’ (a speculative inquiry), this paper aims to understand primitive images, particularly those inflected with histories of silencing and speaking out, as a multimodal field that contains the possibility of practicing democratic audienceship and the ethics of representation in the way we engage with art history.

The preparation of supervisors through collaborative supervision: A narrative account

AbstractTeacher educators are typically involved in preparing preservice teachers (PSTs) in initial programs through teaching courses and supervising practicum experiences, including the culminating student teaching practicum. Using self‐study of teacher education practices (S‐STEPs) during supervision, the authors describe a scaffolded process of learning to supervise teacher candidates and engage in reflections about their experiences working collaboratively. This narrative account by a more experienced supervisor and a new supervisor uses samples of feedback to teacher candidates to demonstrate a collaborative process based on multimodal observational field notes, which included sample discourse from the classroom to exemplify interactions, teaching activities, and photographs of classroom activities. Both researchers participated in joint meetings with (1) PSTs to discuss feedback, (2) mentor teachers to discuss progress of PSTs, and (3) all PSTs to discuss themes that emerged from observations and assessments, something unique to this collaboration. The authors discuss how their collaboration led to curriculum changes to a TESOL methods, curriculum, and assessment course influenced by observations in the field. The article concludes with implications and recommendations for other teacher educators interested in collaborative supervision.

Optical phase encoding in a pulsed approach to reservoir computing.

The exploitation of the full structure of multimode light fields enables compelling capabilities in many fields including classical and quantum information science. We exploit data-encoding on the optical phase of the pulses of a femtosecond laser source for a photonic implementation of a reservoir computing protocol. Rather than intensity detection, data-reading is done via homodyne detection that accesses combinations of an amplitude and a phase of the field. Numerical and experimental results on nonlinear autoregressive moving average (NARMA) tasks and laser dynamic predictions are shown. We discuss perspectives for quantum-enhanced protocols.

A face retrieval technique combining large models and artificial neural networks

SummaryMultimodal sentiment analysis is a popular research direction in the field of affective computing. It extends unimodal‐based sentiment analysis to the environment based on multimodal information exchange. Face retrieval is the most critical technology in the current multimodal sentiment analysis field. Traditional methods for face retrieval rely on large amounts of data to train the matching relationship between text and face images. This approach suffers from several issues, including high data acquisition costs, insufficient or poor quality training data, and high computational costs for training models. To address the aforementioned issues, this paper proposes a face retrieval technique that merges a large language model with a visual base model. This technique explores the use of zero samples for training, eliminating the need to collect data and train the model to achieve face retrieval. As a result, it significantly reduces the cost requirements of data and computational power. First, an accurate natural text of face description is generated using a large language model based on cue engineering based on face‐independent features. Second, the text and images are encoded using a visual base model and mapped into the same multidimensional space for matching. The experimental results indicate that this method performs well on the CelebA dataset. Extracting 31 different sets of face feature descriptions from the CelebA dataset for retrieval achieves 72% on top1 accuracy and 93% on top3 accuracy. Compared to traditional methods, the method proposed in this paper achieves considerable face retrieval accuracy without collecting text image pairs of data and without training the model.

Multimodal subspace identification for modeling discrete-continuous spiking and field potential population activity

Objective. Learning dynamical latent state models for multimodal spiking and field potential activity can reveal their collective low-dimensional dynamics and enable better decoding of behavior through multimodal fusion. Toward this goal, developing unsupervised learning methods that are computationally efficient is important, especially for real-time learning applications such as brain–machine interfaces (BMIs). However, efficient learning remains elusive for multimodal spike-field data due to their heterogeneous discrete-continuous distributions and different timescales. Approach. Here, we develop a multiscale subspace identification (multiscale SID) algorithm that enables computationally efficient learning for modeling and dimensionality reduction for multimodal discrete-continuous spike-field data. We describe the spike-field activity as combined Poisson and Gaussian observations, for which we derive a new analytical SID method. Importantly, we also introduce a novel constrained optimization approach to learn valid noise statistics, which is critical for multimodal statistical inference of the latent state, neural activity, and behavior. We validate the method using numerical simulations and with spiking and local field potential population activity recorded during a naturalistic reach and grasp behavior. Main results. We find that multiscale SID accurately learned dynamical models of spike-field signals and extracted low-dimensional dynamics from these multimodal signals. Further, it fused multimodal information, thus better identifying the dynamical modes and predicting behavior compared to using a single modality. Finally, compared to existing multiscale expectation-maximization learning for Poisson–Gaussian observations, multiscale SID had a much lower training time while being better in identifying the dynamical modes and having a better or similar accuracy in predicting neural activity and behavior. Significance. Overall, multiscale SID is an accurate learning method that is particularly beneficial when efficient learning is of interest, such as for online adaptive BMIs to track non-stationary dynamics or for reducing offline training time in neuroscience investigations.

Mirror-based multimodal straightforward method for impedance eduction using a zigzag array

The Mirror-based Multimodal StraightForward Method (MM-SFM) is proposed herein to realize accurate and reliable liner impedance eduction covering the wide frequency range of aeroengine fan noises. It adopts a novel zigzag array in the flow duct, and unfolds them by a coordinate transformation into a mapping diagonal array in an extended duct, which can increase the array transverse spacing Δz considerably. Then, it invokes the M-SFM incorporating the trust-region and extended Prony's algorithms, to decompose the three-dimensional multimodal aeroacoustic field and educe the impedance. A parametric study based on the Monte-Carlo uncertainty analysis is conducted through finite-element numerical experiments on a 51 × 51 mm2 flow duct, to survey the method performance under random perturbations. It indicates that, the educing accuracy and reliability are promoted by increasing Δz or the axial spacing Δx properly, and the decomposed-mode number M×N=10 is appropriate herein. In our studied cases, using the recommended array of Δz=25.5mm and Δx=22mm locating at the nodes of transverse mode m = 2, the MM-SFM generally produces accurate averaged educed impedances in a considerably extended frequency range from 100 Hz up to nearly 10 kHz, and has small educing uncertainties at most absorptive frequencies and thus a good reliability.

Multimodal Machine Translation

In recent years, neural network machine translation, especially in the field of multimodality, has developed rapidly. It has been widely used in natural languages processing tasks such as event detection and sentiment classification. The existing multimodal neural network machine translation is mostly based on the autoencoder framework of the attention mechanism, which further integrates spatial-visual features. However, due to the ubiquitous lack of corpus and the semantic interaction between multimodalities, the quality of machine translation is difficult to guarantee. Therefore, this paper proposes a multi-modal machine translation model that integrates external linguistic knowledge. Specifically, on the encoder side, we adopt the pre-trained Bert model to be used as an additional encoder to integrate with the original text encoder and picture encoder. Under the cooperation of the three encoders, a better text representation and picture representation at the source end is generated. Besides, the decoder decodes and generates a translation based on the image and text representation of the source. To sum up, this paper studies the visual-text semantic interaction on the encoder side and the visual-text semantic interaction on the decoder side, and further improves the quality of translation by introducing external linguistic knowledge. We compared the performance of the multimodal neural network machine translation model with pre-trained Bert and other baseline models in English German translation tasks on the multi30k data sets. The results show that the model can significantly improve the quality of multimodal neural network machine translation, which also verifies the importance of integrating external knowledge and visual text semantic interaction.

Fusion-Embedding Siamese Network for Light Field Salient Object Detection

Light field salient object detection (SOD) has shown remarkable success and gained considerable attention from the computer vision community. Existing methods usually employ a single-/two-stream network to detect saliency. However, these methods can only handle up to two different modalities at a time, preventing them from being able to fully explore the rich information in multi-modal light field derived data. To address this, we propose the first joint multi-modal learning framework, called FES-Net, for light field SOD, which can take rich inputs not limited to two modalities. Specifically, we propose an attention-aware adaptation module to first transform the multi-modal inputs for use in our joint learning framework. The transformed inputs are then fed to a Siamese network along with multiple embedded feature fusion modules to extract informative multi-modal features. Finally, we predict saliency maps from the high-level extracted features using a saliency decoder module. Our joint multi-modal learning framework effectively resolves the limitations of existing methods, providing efficient and effective multi-modal learning that can fully explore the valuable information in light field data for accurate saliency detection. Furthermore, we improve the performance by introducing the Transformer as our backbone network. To the best of our knowledge, the improved version of our model, called FES-Trans, is the first attempt to address the challenging light field SOD with the powerful Transformer technique. Extensive experiments on benchmark datasets demonstrate that our models are superior light field SOD approaches and outperform cutting-edge models remarkably.

Seeking Remedy for Improving International College Students’ Reading Comprehension

This report reflects on the findings of one of the projects within a series of research studies in the field of multimodality conducted by the author of the current paper with different samples of participants in the US university settings. The principal investigator (PI) designed this qualitative case study and had it approved by the IRB committee. Vygotsky’s socio-cultural constructivism theory has been found most appropriate to lay the ground for this research as the sample was purposefully assembled of 14 International English language learning (ELL) students newly arrived in the United States from their significantly diverging social and academic backgrounds. The PI’s practical teaching experiences and the results of her previously conducted studies with the other groups of participants urged this researcher to further search for some efficient strategies and means of learning, which might help these specific student populations with their deeper reading comprehension of complicated academic texts they were otherwise struggling to comprehend. As claimed by this investigator, usage of multimodality might have a strong potential to assist the International ELL college students with their reading comprehension. Within this current project, data were collected via participant and site observations, surveys, artifacts in the form of the comprehension tests, and individual interviews with participants. All the data were coded, analyzed, and triangulated aiming to find recurring themes. This research found that participants recognized the visual mode of multimodality most helpful to them, followed by the auditory, and print-based ones. In the surveys and interviews, the participating students outlined a range of multimodal reading comprehension strategies, which they perceived as most efficient while their working on this project tasks. The comprehension test results served as the evidence in support of the interview- and survey-based data. High significance of the findings of this research is in their applicability to the wide populations of the International ELL college students in the US colleges and world-wide.

Nonlocal phase modulation of multimode, continuous-variable twin beams

We investigate experimentally the nonlocal phase modulation of multiple-frequency-mode, continuous-variable entangled twin beams. We use a pair of electro-optical phase modulators to modulate the entangled probe and conjugate light beams produced by four-wave mixing in hot rubidium vapor. A single phase modulator in either one of the twin beams reduces the two-mode squeezing signal. The overall quantum entanglement is preserved, however, as the modulator nonlocally distributes the beam correlations among frequency modes of the multimode fields. The two-mode squeezing can be recovered by reversing the mixing with an additional out-of-phase electro-optical phase modulator (EOM) in the other beam.