Visual Counterpart Research Articles

Perceptual coupling in human dyads: Kinematics does not affect interpersonal synchronization

The minimal, essential condition for individuals to interact is that they exchange information via at least one sensory channel. Once informational coupling is established, it enables basic forms of coordinated behavior to spontaneously emerge from the interaction. Our previous study revealed different coordination dynamics in dyads engaged in a joint finger-tapping task based on visual versus auditory coupling. This observation led us to propose the ‘modality-dependent hypothesis', which posits that coordination dynamics are influenced by the sensory modality mediating informational coupling. However, recognizing that different modalities have inherent differences in accessing spatiotemporal features of perceived movement, we formulated the alternative ‘kinematic hypothesis'. This hypothesis posits that differences in dynamics would vanish given equivalent kinematic information across modalities. The study involved forty (N = 40) participants, grouped into twenty (N = 20) dyads, who engaged in a joint finger-tapping task. This task was conducted under varying conditions of visual and auditory coupling, with manipulations in the access to kinematic information, categorized as discrete and continuous. Contrary to our initial predictions, the results strongly supported the ‘modality-dependent hypothesis'. We observed that visual and auditory coupling consistently yielded distinct attractor dynamics, regardless of the access to kinematic information. Furthermore, all conditions of auditory coupling resulted in higher levels of synchronization than their visual counterparts. These findings suggest that the differences in interpersonal synchronization are predominantly influenced by the sensory modality, rather than the continuity of kinematic information. Our study highlights the significance of sensorimotor interactions in interpersonal synchronization and addresses the potential of sonification strategies in supporting motor training and rehabilitation.

The Good Shepherd. On Endre Bálint’s late photomontages

In this paper I analyse the photomontages of Endre Bálint (1914–1986) in the context of the cultural politics of the Iron Curtain period in Eastern Europe and Hungary. First, the study takes a diachronic, historical approach to photomontage as a creative method that maintained continuity between the avant-garde generations of artists, a way of thinking and a creative process accordingly, cultivated in the circles of persecuted, banned or silenced artists experimenting during the decades from the 1920s to 1970s. Second, by placing Endre Bálint’s late photomontages (made after his 1963 return from his “exodus” to Paris) in the context of contemporary artistic phenomena (e.g. Brecht’s photobook, Jean-Luc Godard’s work, David Hockney’s montages or bricoleur punk artists), I argue that, in contrast to the generally held view, they are not the withering signs of renunciation, but rather a way out of renunciation, that is, a solution. “We breathe in fragments,” Bálint wrote, and the pictorial form to fittingly reflect a fragmented way of existence (together with the psychological burdens, shadows of the past, and the social and cultural-political determinations) was the montage-technique that had flourished from the 1920s, and whose Eastern European, distinctively Hungarian variant found its guardian, its good shepherd in Bálint, in his creative practice. Bálint’s late photomontages also deserve attention from the point of view of a silent narrative of art history, which does not focus on middle-aged artists’ major works, but on the profuse production by the old masters, the masters of sprezzatura, which is characterised by an aesthetic lightness, a kind of aesthetic liberation and swiftness, and the ability to allow memories a free influx into the creative work. One of the conclusions of this study is that Hungarian photomontage, and especially the late work of Endre Bálint, can be instructively read in conjunction with the equally restrained psychoanalytic literature of the period, in which the splitting of Self as a traumatic consequence of shocking events and also a means to survive those events is a key concept. A critically productive artistic construction that is based on fragments can be seen and read as the visual counterpart of a psychological notion of Self-splitting.

Sub-visemic discrimination and the effect of visual resemblance on silent lip-reading

ABSTRACT Relevant visual information is available in speakers’ faces during face-to-face interactions that improve speech perception. There is an ongoing debate, however, about how phonemes and their visual counterparts, visemes are mapped. An influential hypothesis claims that several phonemes can be mapped into a single visemic category (many-to-one phoneme-viseme mapping). In contrast, recent findings have challenged this view, reporting evidence for sub-visemic syllable discrimination. We aim to investigate whether Spanish words from the same visemic category can be identified or not. We designed a lip-reading task in which participants had to identify target words presented in silent video clips among 3 distractors differing in their visual resemblance from the target. Target words were identified above chance and significantly more than distractors from the same visemic category. Moreover, the error rate for distractors significantly decreased with decreasing visemic resemblance to the target. These results challenge the many-to-one phoneme-viseme mapping hypothesis.

Tactile adaptation to orientation produces a robust tilt aftereffect and exhibits crossmodal transfer when tested in vision

Orientation processing is one of the most fundamental functions in both visual and somatosensory perception. Converging findings suggest that orientation processing in both modalities is closely linked: somatosensory neurons share a similar orientation organisation as visual neurons, and the visual cortex has been found to be heavily involved in tactile orientation perception. Hence, we hypothesized that somatosensation would exhibit a similar orientation adaptation effect, and this adaptation effect would be transferable between the two modalities, considering the above-mentioned connection. The tilt aftereffect (TAE) is a demonstration of orientation adaptation and is used widely in behavioural experiments to investigate orientation mechanisms in vision. By testing the classic TAE paradigm in both tactile and crossmodal orientation tasks between vision and touch, we were able to show that tactile perception of orientation shows a very robust TAE, similar to its visual counterpart. We further show that orientation adaptation in touch transfers to produce a TAE when tested in vision, but not vice versa. Additionally, when examining the test sequence following adaptation for serial effects, we observed another asymmetry between the two conditions where the visual test sequence displayed a repulsive intramodal serial dependence effect while the tactile test sequence exhibited an attractive serial dependence. These findings provide concrete evidence that vision and touch engage a similar orientation processing mechanism. However, the asymmetry in the crossmodal transfer of TAE and serial dependence points to a non-reciprocal connection between the two modalities, providing further insights into the underlying processing mechanism.

Walking With A Camera

In his work, the author primarily focuses on the urban landscape and its impacts on human life. His entire documentary work is linked by a unified authorial idea. In the “Jungle City” cycle, he works with the visual counterpart of the current form of the Bratislava metropolis, full of transportation routes, dense modern development, and concrete surfaces. The means to create unexpected views of the city for him are carefully selected locations on the urban peripheries surrounded by dense vegetation – trees and wild grasses. He thus achieves a completely new view of Bratislava, in which it returns to the wilderness. The project “Quiet Sunday Afternoons in the Town by the Refinery” is the author’s reflection on the impacts of industry in the rural surroundings. The focal point is the technical structure of an oil factory, which covers a significant part of an otherwise tranquil landscape of a small Austrian town. However, through austere visual expression, Korček delivers an urgent message about the adverse impact of the factory on the ecosystem. As part of the series “Here Is Our Paradise”, which unifies the theme of lawns and trees, Korček addresses the disruption of the stereotype of urban housing estates through the small self-help interventions of their residents. While walking with a camera, he supplements these plants with various found mixtures of folk interventions placed in front of panel buildings. In the “Searching” series, he adopts the position of an observer of urban movement, consisting of figures of pedestrians in narrow rays of the low-lying sun. This time he works in the environment of New York, but unlike the previous series, he does not specify this location, only utilizing its elemental potential. Korček does not work with photographic effects; he fully utilizes the poetics of the chosen locations. He skillfully works with perspective, multiple plans, or the selection of a location. Through their specific combination, he succeeds in creating authentic images of the location, in which we still find a lot of personal fantasy.

Grundmuster: Ausdrucksfunktionen des monochromen Hintergrundes im Renaissanceporträt

Abstract Monochrome backgrounds represent a consistent “pattern” in Renaissance painting. Until now, these backgrounds have mostly been understood as neutral, secondary, or arbitrary. This paper proposes different classification approaches to capture the diverse pictorial potentials and artistic expressive functions. The two-dimensional background forms a visual counterpart to the portrayed person by shaping the immediate surrounding space. It offers a field of appearance in which and with which the individual simultaneously figures and interferes. As a media-reflexive iconic pictorial device, the shapeless ground is based on multi-layered pictorial traditions, conventions, and connotations.

Grounded language acquisition through the eyes and ears of a single child.

Starting around 6 to 9 months of age, children begin acquiring their first words, linking spoken words to their visual counterparts. How much of this knowledge is learnable from sensory input with relatively generic learning mechanisms, and how much requires stronger inductive biases? Using longitudinal head-mounted camera recordings from one child aged 6 to 25 months, we trained a relatively generic neural network on 61 hours of correlated visual-linguistic data streams, learning feature-based representations and cross-modal associations. Our model acquires many word-referent mappings present in the child's everyday experience, enables zero-shot generalization to new visual referents, and aligns its visual and linguistic conceptual systems. These results show how critical aspects of grounded word meaning are learnable through joint representation and associative learning from one child's input.

A Visually Enhanced Neural Encoder for Synset Induction

The synset induction task is to automatically cluster semantically identical instances, which are often represented by texts and images. Previous works mainly consider textual parts, while ignoring the visual counterparts. However, how to effectively employ the visual information to enhance the semantic representation for the synset induction is challenging. In this paper, we propose a Visually Enhanced NeUral Encoder (i.e., VENUE) to learn a multimodal representation for the synset induction task. The key insight lies in how to construct multimodal representations through intra-modal and inter-modal interactions among images and text. Specifically, we first design the visual interaction module through the attention mechanism to capture the correlation among images. To obtain the multi-granularity textual representations, we fuse the pre-trained tags and word embeddings. Second, we design a masking module to filter out weakly relevant visual information. Third, we present a gating module to adaptively regulate the modalities’ contributions to semantics. A triplet loss is adopted to train the VENUE encoder for learning discriminative multimodal representations. Then, we perform clustering algorithms on the obtained representations to induce synsets. To verify our approach, we collect a multimodal dataset, i.e., MMAI-Synset, and conduct extensive experiments. The experimental results demonstrate that our method outperforms strong baselines on three groups of evaluation metrics.

A Comprehensive Survey on Video Saliency Detection With Auditory Information: The Audio-Visual Consistency Perceptual is the Key!

Video saliency detection (VSD) aims at fast locating the most attractive objects/things/patterns in a given video clip. Existing VSD-related works have mainly relied on the visual system but paid less attention to the audio aspect. In contrast, our audio system is the most vital complementary part of our visual system. Also, audio-visual saliency detection (AVSD), one of the most representative research topics for mimicking human perceptual mechanisms, is currently in its infancy, and none of the existing survey papers have touched on it, especially from the perspective of saliency detection. Thus, the ultimate goal of this paper is to provide an extensive review to bridge the gap between audio-visual fusion and saliency detection. In addition, as another highlight of this review, we have provided a deep insight into key factors that could directly determine AVSD deep models’ performances. We claim that the audio-visual consistency degree (AVC) — a long-overlooked issue, can directly influence the effectiveness of using audio to benefit its visual counterpart when performing saliency detection. Moreover, to make the AVC issue more practical and valuable for future followers, we have newly equipped almost all existing publicly available AVSD datasets with additional frame-wise AVC labels. Based on these upgraded datasets, we have conducted extensive quantitative evaluations to ground our claim on the importance of AVC in the AVSD task. In a word, our ideas and new sets serve as a convenient platform with preliminaries and guidelines, all of which can potentially facilitate future works in further promoting state-of-the-art (SOTA) performance.

Points2Sound: from mono to binaural audio using 3D point cloud scenes

For immersive applications, the generation of binaural sound that matches its visual counterpart is crucial to bring meaningful experiences to people in a virtual environment. Recent studies have shown the possibility of using neural networks for synthesizing binaural audio from mono audio by using 2D visual information as guidance. Extending this approach by guiding the audio with 3D visual information and operating in the waveform domain may allow for a more accurate auralization of a virtual audio scene. We propose Points2Sound, a multi-modal deep learning model which generates a binaural version from mono audio using 3D point cloud scenes. Specifically, Points2Sound consists of a vision network and an audio network. The vision network uses 3D sparse convolutions to extract a visual feature from the point cloud scene. Then, the visual feature conditions the audio network, which operates in the waveform domain, to synthesize the binaural version. Results show that 3D visual information can successfully guide multi-modal deep learning models for the task of binaural synthesis. We also investigate how 3D point cloud attributes, learning objectives, different reverberant conditions, and several types of mono mixture signals affect the binaural audio synthesis performance of Points2Sound for the different numbers of sound sources present in the scene.

Boxers, Cue Balls, and Comedians: E. E. Cummings’ Polytexts

Poetry and painting were the “the twin obsessions” of the American modernist E. E. Cummings. His engagement with more than one artistic practice makes his work suitable for a case study of polytextuality, which is the purpose of the present article. First, a brief theoretical introduction to polytextuality is offered. Then, four of Cummings’ poems are read against the background of their visual counterparts. Since the poem-picture pairs are treated as polytexts, the discussion focuses on the “intersemiotic transposition” involved in the process of transferring thematic and material data from one sign system to another.

Early blindness modulates haptic object recognition.

Haptic object recognition is usually an efficient process although slower and less accurate than its visual counterpart. The early loss of vision imposes a greater reliance on haptic perception for recognition compared to the sighted. Therefore, we may expect that congenitally blind persons could recognize objects through touch more quickly and accurately than late blind or sighted people. However, the literature provided mixed results. Furthermore, most of the studies on haptic object recognition focused on performance, devoting little attention to the exploration procedures that conducted to that performance. In this study, we used iCube, an instrumented cube recording its orientation in space as well as the location of the points of contact on its faces. Three groups of congenitally blind, late blind and age and gender-matched blindfolded sighted participants were asked to explore the cube faces where little pins were positioned in varying number. Participants were required to explore the cube twice, reporting whether the cube was the same or it differed in pins disposition. Results showed that recognition accuracy was not modulated by the level of visual ability. However, congenitally blind touched more cells simultaneously while exploring the faces and changed more the pattern of touched cells from one recording sample to the next than late blind and sighted. Furthermore, the number of simultaneously touched cells negatively correlated with exploration duration. These findings indicate that early blindness shapes haptic exploration of objects that can be held in hands.

A specialized channel for encoding auditory transients in the magnocellular division of the human medial geniculate nucleus.

We test the hypothesis that there exists a generalized magnocellular system in the brain optimized for temporal processing. In the visual system, it is well known that the magnocellular layers in the lateral geniculate nucleus (LGN) are strongly activated by transients and quickly habituate. However, little is known about the perhaps analogous magnocellular division of the medial geniculate nucleus (MGN), the auditory relay in the thalamus. We measured the functional responses of the MGN in 11 subjects who passively listened to sustained and transient nonlinguistic sounds, using functional MRI. We observed that voxels in the ventromedial portion of the MGN, corresponding to the magnocellular division, exhibited a robust preference to transient sounds, consistently across subjects, whereas the remainder of the MGN did not discriminate between sustained and transient sounds. We conclude that the magnocellular neurons in the MGN parallel the magnocellular neurons in its visual counterpart, LGN, and constitute an information stream specialized for encoding auditory dynamics.

Urban Sound Auralization and Visualization Framework—Case Study at IHTApark

In the context of acoustic urban planning, the use of noise mappings is a worldwide well-established practice. Therefore, the noise levels in an urban environment are calculated based on models of the sound sources, models of the physical sound propagation effects and the position of the receivers in the area of interest. However, the noise mapping method is limited to sound levels in frequency bands due to missing temporal and spectral information of the sound signals. This, in turn, leads to missing information about the qualitative sound properties, as they can be evaluated in psychoacoustic parameters. Beyond the scope of the classical noise mapping, auralization and physically-based simulation of sound fields can be applied to urban scenarios in the context of urban soundscape analysis. By supporting the auralization technology with a visual counterpart of the urban space, a plausible virtual representation of a real environment can be achieved. The presented framework combines the possibilities of the open-source auralization tool Virtual Acoustics with 3D visualization. In order to enable studies with natural human response or for public communication of urban design projects, those virtual scenes can be either reproduced with immersive technologies—such head-mounted displays (HMD)—or using online video platforms and traditional playback devices. The paper presents an overview of what physical principles can already be simulated, which technological considerations need to be taken into account, and how to set up such environment for auralization and visualization of urban scenes. We present the framework by the case study of IHTApark.

The Symbolist Conception of Illustration and Tyra Kleen’s Nevermore

The Symbolist Conception of Illustration and Tyra Kleen’s <i>Nevermore</i>

Supporting theatre education through resources and policy

Theater educators, like their music, dance, and visual arts counterparts, had to pivot quickly to provide educational opportunities in theater for students during the COVID-19 pandemic. This article outlines how two national associations, the Educational Theater Association, and the American Alliance for Theater in Education, responded to the crisis and provided support to theater educators in the field during the pandemic. In addition, the article includes information on a brief survey conducted with theater educators in Fall, 2020, speaking to how they were teaching during the pandemic, and what they were teaching, including administrative and resource supports. The online curricular and instructional resources created by EdTA are of particular interest as the organization was just ramping up online professional development at the start of the pandemic. During the pandemic, EdTA came to the forefront in providing resources for all educators on how to create instructional content in a variety of platforms.

Comparing verbal working memory load in auditory and visual modalities using functional near-infrared spectroscopy

The verbal identity n-back task is commonly used to assess verbal working memory (VWM) capacity. Only three studies have compared brain activation during the n-back when using auditory and visual stimuli. The earliest study, a positron emission tomography study of the 3-back, found no differences in VWM-related brain activation between n-back modalities. In contrast, two subsequent functional magnetic resonance imaging (fMRI) studies of the 2-back found that auditory VWM was associated with greater left dorsolateral prefrontal cortex (DL-PFC) activation than visual VWM, perhaps suggesting that auditory VWM requires more cognitive effort than its visual counterpart. The current study aimed to assess whether DL-PFC activation (i.e., cognitive effort) differs by VWM modality. To do this, 16 younger adults completed an auditory and visual n-back, both at four levels of VWM load. Concurrently, activation of the PFC was measured using functional near-infrared spectroscopy (fNIRS), a silent neuroimaging method. We found that DL-PFC activation increased with VWM load, but it was not affected by VWM modality or the interaction between load and modality. This supports the view that both VWM modalities require similar cognitive effort, and perhaps that previous fMRI results were an artefact of scanner noise. We also found that, across conditions, DL-PFC activation was positively correlated with reaction time. This may further support DL-PFC activation as an index of cognitive effort, and fNIRS as a method to measure it.

An Auditory Stroop Test to Implement in Cognitive Hearing Sciences: Development and Normative Data.

.Objective: This study developed and gained insight in an auditory Stroop test, implementable in cognitive hearing sciences. Methods: An auditory Stroop test was developed and performed in 178 participants, aged between 18 and 69 years. This Auditory Stroop test consisted of two tests: Stroop-tones and Stroop-words whereby the pitch of pure-tones and spoken words (i.e., the words high and low) had to be identified by high or low, respectively. An interference score was calculated as a measure of verbal executive functioning. Regression models were conducted to examine the effect of age, sex, education, awakeness, hearing, as well as visual and verbal working memory, and processing speed on the auditory Stroop scores. Normative data were obtained per age decade. Results: Compared to the visual counterparts, the auditory Stroop outcomes were better predicted by verbal working memory and processing speed. A trend was observed showing a decrease in performances with increasing age. No other participant-related variables had a significant relationship with the auditory Stroop test. Conclusion: This auditory Stroop test was considered a good test for measuring executive functioning using auditory stimuli. Implementing this auditory Stroop test within cognitive hearing sciences will contribute to unravel the auditory-cognitive perspective of speech understanding.

"A Monument to Blackness"

Murals have been present at pivotal moments during the Black freedom struggle, yet they remain on the margins of scholarship. Birthed during the Harlem Renaissance, cresting in the streets during Black Power, and resurrected today during #BlackLivesMatter, murals are radical, visual counterparts to the spoken politics of a movement. This article highlights the instrumental role murals played in fostering Black unity and empowerment in grassroots mobilizations since the 1920s. It introduces their complex, malleable roles as touchstones of communication, sites of ritual, visual tombstones, and talismans of a new Black consciousness, demonstrating how they became what a 1968 newspaper called “a monument to blackness.”

Trompe l’œil in the Decoration of Alba Amicorum from circa 1750 to circa 1815 from the Collection of the Wrocław University Library

The article is an attempt at systemizing and interpreting trompe l'œil motifs and devices – from single objects shown as if resting on a page, through those altering the appearance of the page, such as folded page corners, up to still lifes – that constitute elements of book-of-friendship entries in one particular collection. Examples are used to show their relations with the entry text: illusionistic elements could illustrate, but also extend or complement it. It was particularly the quodlibet-type compositions that could allude to the book’s owner and the shared memories, forming as if a visual counterpart of ‘memorabilia’. The researched motifs do not always comply with all the criteria of trompe l’œil; they do, however, aspire to being illusionistic, additionally fulfilling well the tasks that are associated with books of friendship.