Visual Modality Research Articles

A Piezoelectric Nanogenerator-Driven Dual-Mode Platform for Visualization and Impedance Sensing.

Traditional visual biosensing platforms rely on color to display detection results, which can be influenced by individual visual abilities, equipment, parameters, and lighting conditions during photo capture. This limitation significantly impedes the advancement of next-generation portable electrochemical biosensors. Therefore, we propose a visual biosensing device that utilizes distance as an indicator, enabling the facile determination of the length of discoloration, which is inversely proportional to the concentration of the target analyte. The separation of the Signal Generation (SG) and Signal Output (SO) regions effectively mitigates potential interference from the sample color. Additionally, the SG region can be disassembled to facilitate electrochemical impedance spectroscopy (EIS) detection in laboratory settings, enabling dual-mode detection. Meanwhile, the utilization of piezoelectric nanogenerators (PENG) empowers the entire point-of-care testing (POCT) sensing device, effectively addressing the issue of a limited battery life. The biosensing device exhibited a satisfactory linear range (EIS mode, 5 pg/L to 5 mg/L; visual mode, 0.5 ng/L to 5 mg/L) and a low limit of detection (EIS mode, 2.3 pg/L; visual mode, 0.14 ng/L) with S/N = 3 for ochratoxin A (OTA) under optimized conditions. The self-powered and cost-effective dual-mode biosensing platform developed for OTA detection offers clear and easily interpretable results, demonstrating a high accuracy in laboratory settings.

Vision-Based Algorithm for Precise Traffic Sign and Lane Line Matching in Multi-Lane Scenarios

With the rapid development of intelligent transportation systems, lane detection and traffic sign recognition have become critical technologies for achieving full autonomous driving. These technologies offer crucial real-time insights into road conditions, with their precision and resilience being paramount to the safety and dependability of autonomous vehicles. This paper introduces an innovative method for detecting and recognizing multi-lane lines and intersection stop lines using computer vision technology, which is integrated with traffic signs. In the image preprocessing phase, the Sobel edge detection algorithm and weighted filtering are employed to eliminate noise and interference information in the image. For multi-lane lines and intersection stop lines, detection and recognition are implemented using a multi-directional and unilateral sliding window search, as well as polynomial fitting methods, from a bird’s-eye view. This approach enables the determination of both the lateral and longitudinal positioning on the current road, as well as the sequencing of the lane number for each lane. This paper utilizes convolutional neural networks to recognize multi-lane traffic signs. The required dataset of multi-lane traffic signs is created following specific experimental parameters, and the YOLO single-stage target detection algorithm is used for training the weights. In consideration of the impact of inadequate lighting conditions, the V channel within the HSV color space is employed to assess the intensity of light, and the SSR algorithm is utilized to process images that fail to meet the threshold criteria. In the detection and recognition stage, each lane sign on the traffic signal is identified and then matched with the corresponding lane on the ground. Finally, a visual module joint experiment is conducted to verify the effectiveness of the algorithm.

Automatic Code Generation for Android Applications Based on Improved Pix2code

With the expansion of the Internet market, the traditional software development method has been difficult to meet the market demand due to the problems of long development cycle, tedious work and difficult system maintenance. Therefore, to improve software development efficiency, this study uses residual networks and bidirectional long short-term memory networks to improve the Pix2code model. The experiment results show that after improving the visual module of the Pix2code model using residual networks, the accuracy of the training set improves from 0.92 to 0.96, and the convergence time is shortened from 3 hours to 2 hours. After using a bidirectional long short-term memory network to improve the language module and decoding layer, the accuracy and convergence speed of the model have also been improved. The accuracy of the training set grew from 0.88 to 0.92, and the convergence time was shortened by 0.5 hours. However, models improved by bidirectional long short-term memory networks might exhibit over-fitting, and thus this study uses Dropout and Xavier normal distribution to improve the memory network. The results validate that the training set accuracy of the optimized bidirectional long short-term memory network remains around 0.92, but the accuracy of the test set has improved to a maximum of 85%. Dropout and Xavier normal distributions can effectively improve the over-fitting problem of bidirectional long short-term memory networks. Although they can also decrease the model’s stability, their gain is higher. The training and testing accuracy of the Pix2code improved by residual network and bidirectional long short-term memory network are 0.95 and 0.82, respectively, while the code generation accuracy of the original Pix2code is only 0.77. The above data indicates that the improved Pix2code model has improved the accuracy and stability of code automatic generation.

Relational-branchformer: Novel framework for audio-visual speech recognition

This study embraced the state-of-the-art Branchformer series architecture within the realm of automatic speech recognition, supplanting the widely utilized Conformer architecture. This substitution offers an innovative remedy tailored to audio-visual speech recognition tasks. Building upon the Branchformer architecture, enhancements were made, culminating in the proposal of the Relational-Branchformer (R-Branchformer). The convolutional attention relation module was innovatively incorporated to augment the connectivity between the local and global branches by meticulously considering their interrelations and interplays. Consequently, this module facilitates the mutual embedding of local and global contextual information, ultimately leading to a substantial enhancement in model performance. Our model was grounded in the utilization of the connectionist temporal classification (CTC) loss, wherein intermediate CTC losses were incorporated between blocks. Moreover, through the reference and enhancement of the gated interlayer collaboration module, which superseded the inter CTC module, the conditional independence assumption intrinsic to the CTC model was effectively relaxed. As a consequence, this augmentation markedly bolstered the overall performance of our model. Furthermore, the audio-visual output enhancement module was proposed, which adeptly assimilates information from both audio and visual modalities to enrich the representation of audio-visual information. Consequently, the R-Branchformer model achieved remarkable word error rates of 1.7% and 1.5% on the LRS2 and LRS3 test sets, respectively, exemplifying its state-of-the-art performance in audio-visual speech recognition tasks.

Emotion recognition in user‐generated videos with long‐range correlation‐aware network

AbstractEmotion recognition in user‐generated videos plays an essential role in affective computing. In general, visual information directly affects human emotions, so the visual modality is significant for emotion recognition. Most classic approaches mainly focus on local temporal information of videos, which potentially restricts their capacity to encode the correlation of long‐range context. To address this issue, a novel network is proposed to recognize emotions in videos. To be specific, a spatio‐temporal correlation‐aware block is designed to depict the long‐range correlations between input tokens, where the convolutional layers are used to learn the local correlations and the inter‐image cross‐attention is designed to learn the long‐range and spatio‐temporal correlations between input tokens. To generate diverse and challenging samples, a dual‐augmentation fusion layer is devised, which fuses each frame with its corresponding frame in the temporal domain. To produce rich video clips, a long‐range sampling layer is designed, which generates clips in a wide range of spatial and temporal domains. Extensive experiments are conducted on two challenging video emotion datasets, namely VideoEmotion‐8 and Ekman‐6. The experimental results demonstrate that the proposed method obtains better performance than baseline methods. Moreover, the proposed method achieves state‐of‐the‐art results on the two datasets. The source code of the proposed network is available at: https://github.com/JinChow/LRCANet.

The Latest Research Progress on Bionic Artificial Hands: A Systematic Review.

Bionic prosthetic hands hold the potential to replicate the functionality of human hands. The use of bionic limbs can assist amputees in performing everyday activities. This article systematically reviews the research progress on bionic prostheses, with a focus on control mechanisms, sensory feedback integration, and mechanical design innovations. It emphasizes the use of bioelectrical signals, such as electromyography (EMG), for prosthetic control and discusses the application of machine learning algorithms to enhance the accuracy of gesture recognition. Additionally, the paper explores advancements in sensory feedback technologies, including tactile, visual, and auditory modalities, which enhance user interaction by providing essential environmental feedback. The mechanical design of prosthetic hands is also examined, with particular attention to achieving a balance between dexterity, weight, and durability. Our contribution consists of compiling current research trends and identifying key areas for future development, including the enhancement of control system integration and improving the aesthetic and functional resemblance of prostheses to natural limbs. This work aims to inform and inspire ongoing research that seeks to refine the utility and accessibility of prosthetic hands for amputees, emphasizing user-centric innovations.

HD-tDCS to the lateral occipital complex improves haptic object recognition.

High-definition transcranial direct current stimulation (HD-tDCS) is a non-invasive brain stimulation technique that has been shown to be safe and effective in modulating neuronal activity. The present study investigates the effect of anodal HD-tDCS on haptic object perception and memory through stimulation of the lateral occipital complex (LOC), a structure that has been shown to be involved in both visual and haptic object recognition. In this single-blind, sham-controlled, between-subjects study, blindfolded healthy, sighted participants used their right (dominant) hand to perform haptic discrimination and recognition tasks with 3D-printed, novel objects called "Greebles" while receiving 20min of 2 milliamp (mA) anodal stimulation (or sham) to the left or right LOC. Compared to sham, those who received left LOC stimulation (contralateral to the hand used) showed an improvement in haptic object recognition but not discrimination-a finding that was evident from the start of the behavioral tasks. A second experiment showed that this effect was not observed with right LOC stimulation (ipsilateral to the hand used). These results suggest that HD-tDCS to the left LOC can improve recognition of objects perceived via touch. Overall, this work sheds light on the LOC as a multimodal structure that plays a key role in object recognition in both the visual and haptic modalities.

Metalinguistic Discourse in an Emerging Sign Language

In this paper, I explore metalinguistic discourse in Zinacantec Family Homesign (‘Z sign’), an emergent sign language developed by three deaf siblings and their hearing family members. In particular, I examine how metalinguistic discourse unfolds between a hearing Z signer and various members of her family—including her deaf siblings, her elderly hearing father, and her young hearing son. I do so via a close examination of several snippets of conversation in which the Z signers talk about the “right” way to sign, paying close attention to how they mobilize various semiotic devices, including manual signs, eye gaze, facial expressions, and speech. I aim to understand not only the formal components of metalinguistic discourse in Z sign but also how it functions as a form of social action in this small linguistic community. How do members of this family position themselves and others as (in)competent, (non-)authoritative signers in light of existing social divisions among them? How do they reinforce or challenge those social divisions through metalinguistic discourse? How might metalinguistic discourse contribute to the propagation of emergent linguistic norms throughout the family? I find that a recurrent device for enacting metalinguistic critique among the Z signers is the partial re-production and transformation of others’ utterances and other visible actions, manifested in a way that exploits the availability of multiple, semi-independent manual and non-manual articulators in the visual modality.

Cultural background and input familiarity influence multisensory emotion perception.

During multisensory emotion perception, the attention devoted to the visual versus the auditory modality (i.e., modality dominance) varies depending on the cultural background of the perceiver. In the present study, we examined (a) how cultural familiarity influences multisensory emotion perception in Eastern and Western cultures and (b) the underlying processes accounting for the cultural difference in modality dominance. Native Mandarin speakers from China and native English speakers from the United States were presented with audiovisual emotional stimuli from their own culture (i.e., familiar) and from a different culture (i.e., unfamiliar) and asked to evaluate the emotion from one of the two modalities. Across modalities, the emotions were either the same (i.e., congruent, happy face, and happy voice) or different (i.e., incongruent, happy face, and sad voice). When the input was in a familiar cultural context, American participants were more influenced by the visual modality, while Chinese participants were more influenced by the auditory modality. While both groups integrated the incongruent emotion from the irrelevant modality, only the American group integrated the congruent emotion from the irrelevant modality. When the input was in a less familiar cultural context, both groups showed increased visual dominance, but only the Chinese group simultaneously showed decreased auditory dominance. We conclude that cultural background and input familiarity interact to influence modality dominance during multisensory emotion perception. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Dissecting attention: Rate modulation vs. phase locking

In this issue of Neuron, Spyropoulos etal.1 report direct top-down modulation of neuronal firing rates underlying selective visual attentional modulation.

L’apport du support audio-visuel sous-titré à la compréhension de l’oral et au vocabulaire en milieu universitaire

This article is part of a didactic perspective. He is interested in oral didactics in a university setting. Through this experimental research, we wanted to demonstrate that the interaction between subtitles and visual modality is an essential contribution to oral comprehension (global and detailed) and vocabulary learning through the subtitled audio-visual medium. Our method consists of using a video in two modes of presentation with two distinct groups: the first without subtitles and the second with subtitles.We also used an experimental post questionnaire to find out if students would have liked to have had the text transcribed.

Exploring Narcopolis as a Postmodern Narrative and Semiotic Text

Jeet Thayil’s Narcopolis (2012), inspired by his own experience as an opium and later heroin addict, paints a vivid picture of the seedy underbelly of 1970s Bombay. The novel features a multitude of characters from the opium dens of Bombay and threads multiple narratives, subversive and carnivalesque, to create a postmodern chronicle of the city. This article analyzes the narrative techniques in Jeet Thayil's novel Narcopolis (2012) to explore traits of postmodernity. It contends that the occurrence of multiple narratives, whose reliability is questionable, indicates a Barthian death of the author, encouraging the readers to give meaning to the text. Also, an inquiry into the novel’s commentary on its own narrative style indicates self-reflexivity. A study of the chronotope will be undertaken to decipher the temporality in the novel. Besides, the article will look out for possibilities of embedded narratives, plot digressions, and intertextuality. It also intends to scrutinize the novel using the tools of social semiotics. Linguistic analysis of discourse entails a monolithic approach, wherein it considers language as the only mode of interpretation of a text. Gunther Kress and Theo Van Leeuwen suggest that this limitation can be overcome by social semiotics, which studies a text in its multimodality. Signs, linguistic as well as nonlinguistic, are apparent through multiple modes, such as language, sound, visual, and so forth. The incorporation of certain pictures, changes in typography of certain pages as well as the book cover of Narcopolis poses a rich possibility of the semiotic understanding of the text through visual mode. Lastly, this article will aim to establish a connection between the narrative techniques and the semiotic interpretation of the text.

Compound eyes of Camponotus compressus (Fabricius, 1798) (Hymenoptera, Formicidae) reflects caste specific organisation and adaptation

Caste polymorphs of the ant Camponotus compressus (Fabricius, 1798) are distinct by their morphology and life styles; the two distinct castes are largely nocturnal and rely on their visual sensory system to interpret their temporal niches. The compound eyes of the castes were explored through Light and Scanning Electron Microscopy (SEM) in order to delineate cast specific organisation and adaptations of the compound eye. The findings reveal that major workers of C. compressus possess a more sophisticated visual system in terms of its morphological features along with optical properties that enhances a better vision, which includes a greater number of ommatidia and high ommatidial density, a higher ommatidial diameter, low inter ommatidial angle and a more efficient pupillary mechanism to counter conditions where ambient light levels are high. This underlines the dependence of scouts (major workers) on visual system and the foragers on olfactory system in the species.

Image and verbal text interfaces of internet memes: a cognitive analysis

The article focuses on image-text interface types of Internet memes mocking the same viral event – Melania Trump’s speech at the Republican Party convention. The future first lady’s address contained borrowings from Michelle Obama’s speech at the Democratic Party convention. The relevance of the study is accounted for both by the social significance of visual-verbal Internet memes as a tool influencing the formation of the Internet users’ views and by the need to elucidate the principles of interaction between the verbal and visual modes of memes in creating a humorous effect, which is the aim of the study. The application of cognitive linguistics methods, in particular, the tools of the theory of conceptual integration for multimodal analysis of communication, made it possible to establish four types of visual and verbal text interfaces. It has been concluded that simple visual-verbal metonymy underlies memes, in which the humorous effect is achieved through the interaction of an image that provides access to a viral event mocked by the meme and a verbal text that activates a certain metonymic association highlighting a feature of the event that is incompatible with the expectations provided by the generic space of the viral event. In memes built on the basis of visual metonymy and verbal allusion, incompatible conceptual structures are activated with the help of allusive connection. Radial visual-verbal metonymy occurs when an image engages a series of metonymic features linked by a radial semantic link, whereas the verbal text highlights the incompatibility of these features with the actual features of the viral event. The interaction of allusion and diffuse visual-verbal metonymy is the cognitive basis of memes, in which access to a viral event is provided by verbal allusion, while the image contains signs of absurdity and triggers metonymic associations that are not based on expected connections.

Heterogeneous Interactive Graph Network for Audio–Visual Question Answering

Audio–visual question answering (AVQA) is an emerging task that aims to provide answers by integrating visual contents, audio streams, and their associations within given videos. The major challenge lies in effectively fusing heterogeneous multi-modal data to comprehend complex scenes while capturing question-related clues to infer correct answers. Current AVQA models primarily employ attention mechanisms to extract question-related clues separately from visual and audio modalities before combining them. However, these approaches have two limitations: (1) They neglect the exploration of the association and complementary between audio and visual; (2) Encoding visual or audio holistically limits the capacity to capture the cross-modal and cross-temporal dynamic events. In this paper, we introduce the Heterogeneous Interactive Graph Network, a novel solution designed to address these limitations. Specifically, we construct heterogeneous multi-modal graphs that facilitate unified integration of multiple modalities, including visual, audio, and question. This approach effectively explores the associations and complementarity among multiple modalities, and it investigates local temporal interactions across visual and audio, enabling the effective capture of cross-modal and cross-temporal dynamic events. Additionally, we present a cross-modal feature alignment module, which acts as a bridge to overcome the semantic gap among heterogeneous multi-modal data. It promotes the convergence of multi-modal data distributions into a shared feature space, facilitating more effective and efficient processing. Extensive experimental results demonstrate the superiority of our method compared to state-of-the-art models across various question types on the challenging MUSIC-AVQA and AVQA benchmarks.

Double strategies for reproducing multimodal puns in interlinguistic translation: Experimental research

The research is dedicated to exposing the specifics of translating multimodal puns based on the retrospective experiment. A multimodal pun is defined as a creolized/ polycoded formation substantiated by a specific type of ambiguity consisting of two inhomogeneous semiotic modes. The material for our research was provided by verbal-visual puns functioning as separate texts. Each multimodal pun results from intersemiotic translation when the signs of one semiotic system are transformed into the signs of another. For the vast majority of multimodal puns, verbal signs are interpreted into pictures, but the opposite cannot be excluded either. The role of the visual component is twofold. It can be creative when the picture is part of the ambiguity mechanism, or it can be amplifying when the picture accentuates the verbal wordplay, not participating directly in the creation of ambiguity. Hence, our first hypothesis is that multimodal puns with an amplifying visual component are a lesser challenge for translators than those with a creative one due to the absence of the necessity to coordinate verbal and visual modes in the target text. The research aims to identify the strategies of interlinguistic translation of multimodal puns and the factors that determine them, particularly the impact of the visual mode on the translator’s decision-making. Hence, our second hypothesis is that multimodal puns require double strategies that would allow to correlate the reproduction of the elements of two different semiotic systems. The analysis of the subjects’ translations, as well as their reports received in the course of the delayed retrospective experiment, confirmed both hypotheses. Disclosure Statement The authors reported no potential conflicts of interest. * Corresponding author: Oleksandr Rebrii, 0000-0002-4912-7489o.v.rebrii@karazin.ua

Reliable object tracking by multimodal hybrid feature extraction and transformer-based fusion

Visual object tracking, which is primarily based on visible light image sequences, encounters numerous challenges in complicated scenarios, such as low light conditions, high dynamic ranges, and background clutter. To address these challenges, incorporating the advantages of multiple visual modalities is a promising solution for achieving reliable object tracking. However, the existing approaches usually integrate multimodal inputs through adaptive local feature interactions, which cannot leverage the full potential of visual cues, thus resulting in insufficient feature modeling. In this study, we propose a novel multimodal hybrid tracker (MMHT) that utilizes frame-event-based data for reliable single object tracking. The MMHT model employs a hybrid backbone consisting of an artificial neural network (ANN) and a spiking neural network (SNN) to extract dominant features from different visual modalities and then uses a unified encoder to align the features across different domains. Moreover, we propose an enhanced transformer-based module to fuse multimodal features using attention mechanisms. With these methods, the MMHT model can effectively construct a multiscale and multidimensional visual feature space and achieve discriminative feature modeling. Extensive experiments demonstrate that the MMHT model exhibits competitive performance in comparison with that of other state-of-the-art methods. Overall, our results highlight the effectiveness of the MMHT model in terms of addressing the challenges faced in visual object tracking tasks.

Evaluation of Selected Biometric Parameters in Cataract Patients-A Comparison between Argos® and IOLMaster 700®: Two Swept-Source Optical Coherence Tomography-Based Biometers.

Background and Objectives: To compare the biometry of eyes obtained with two swept-source optical coherence tomography-based biometers-Argos (A), using an individual refractive index, and IOLMaster 700 (IM), using an equivalent refractive index-for all structures. Materials and Methods: The biometry of 105 eyes of 105 patients before cataracts were analyzed in this study. Parameters such as axial length (AL), anterior chamber depth (ACD), and lens thickness (LT) were compared from both devices. According to the axial length measurements, patients were divided into three groups, as follows: group 1-short eyes (AL < 22.5 mm), group 2-average eyes (22.5 ≤ AL ≤ 26.0 mm), and group 3-long eyes (AL > 26.0 mm). Results: The correlation coefficiency among all compared parameters varies from R = 0.92 to R = 1.00, indicating excellent reliability of IM and A. A statistical significance in axial length was indicated in the group of short eyes (n = 26)-mean AL (A) 21.90 mm (±0.59 mm) vs. AL (IM) 21.8 mm ± (0.61 mm) (p < 0.001)-and in the group of long eyes (n = 5)-mean AL (A) 27.95 mm (±2.62 mm) vs. mean AL (IM) 28.10 mm (±2.64) (p < 0.05). In the group of average eyes (n = 74), outcomes were similar-mean AL (A) 23.56 mm (±0.70 mm) vs. mean AL (IM) 23,56 mm (±0.71 mm) (p > 0.05). The anterior chamber depth measurements were higher when obtained with Argos than with IOLMaster 700-mean ACD (A) 3.06 mm (±0.48 mm) vs. mean ACD (IM) 2.92 mm (±0.46) p < 0.001. There was no statistical significance in mean LT-mean LT (A) 4.75 mm (±0.46 mm) vs. mean LT (IM) 4.72 mm (±0.44 mm) (p = 0.054). The biometry of one eye with dense cataracts could be measured only with Argos, using the Enhanced Retinal Visualization mode. Conclusions: Axial length measurements from both devices were different in the groups of short and long eyes, but were comparable in the group of average eyes. The anterior chamber depth values obtained with Argos were higher than the measurements acquired with IOLMaster 700. These differences may be particularly important when selecting IOLs for patients with extreme AL values.

On the use of scale distortion for visual humour

In contrast to verbal humour, visual humour remains a relatively underdeveloped area of research. In this exploratory study, we investigate whether scale incongruity – i.e., discrepancy between the expected and actual experience of the size of an object – can serve as a source of humour in the visual modality. We adapt a pre-existing visual data set of mundane scenes by altering the size of an individual object in each scene and collecting humorousness ratings from human annotators on the original and scale-distorted versions. Our analysis of these annotations reveals that scenes with distorted objects are perceived to be significantly funnier than the original images.

Memory-aware continual learning with multi-modal social media streams for unsupervised disaster classification

Social media has emerged as a major hub for information dissemination during disasters. It serves as a central repository for massive real-time updates about disasters, facilitating the organization of humanitarian assistance. Disaster types classification, which identifies and categorizes various types of disasters on social media, has become a fundamental technique for rapid disaster response. However, current disaster classification methods suffer from the following two shortcomings: (1) They rely on manual annotation to train the models, which is expensive and time-consuming in rapidly updating social media; (2) During the disaster, reports on social media are progressively enriched from initial text to multi-modal mix containing image, audio, and video. Existing methods cannot handle these incremental multi-modal data. To address these challenges, we propose a Memory-aware Continual Learning (MaCL) method, which accomplishes unsupervised disaster classification by flexibly handling the multi-modal streams on social media in a continual learning manner. First, a modal-shared contrastive encoder is constructed to mine the correlation information among the textual and visual modalities on social media. We theoretically demonstrate that mining correlation information is approximately equivalent to preserving predictive information in the original data. Then, we design a Correlation Memory Bank (CMB) to store the features of current modalities. As new modalities emerge, the correlation information between the new modality and its global neighbors in the CMB is maximized. This strategy enables the correlation information to guide the classification of the newly arrived modalities. Finally, the new correlation information generated by the CMB continuously updates and accumulates, leading to improve disaster classification performance across all modalities. Extensive experiments on seven real-world datasets validate the effectiveness of the proposed MaCL.