High-level Cues Research Articles

Integration of semantic and coarticulation cues during spoken language comprehension in adults

Recent work examining cue integration across levels of linguistic representation has found that listeners can dynamically integrate some of the lower-level and higher-level cues during spoken language comprehension. However, it is still not well understood how the mechanism of cue integration works. This study investigated how adults (n = 52) process preceding higher-level semantic cues and later low-level coarticulation cues during spoken language comprehension using an eye-tracking paradigm. Participants were tested on sentences that contained a prime (semantically related or semantically unrelated to the target) and a target which had varying coarticulation cues (matching versus mismatching splicing cues). Participants were presented with two pictures (target and competitor) on a screen. Analyses looked at the proportion of looking to the target during the prime and target time windows. Results demonstrate that adults flexibly use both the preceding semantic cues and later coarticulatory cues once they are available. Our findings also indicate that adults flexibly weighed both the preceding higher-level and later lower-level cues, such that the processing of low-level coarticulatory cue varied depending on the semantic context. We have added an unstudied level of cue (semantic context) to the set of cues that our cognitive system can integrate during language comprehension.

The development of visual categorization based on high-level cues.

This study investigated the development of rapid visual object categorization. N = 20 adults (Experiment 1), N = 21 five to six-year-old children (Experiment 2), and N = 140 four-, seven-, and eleven-month-old infants (Experiment 3; all predominantly White, 81 females, data collected in 2013-2020) participated in a fast periodic visual stimulation electroencephalographic task. Similar categorization of animal and furniture stimuli emerged in children and adults, with responses much reduced by phase-scrambling (R2 = .34-.73). Categorization was observed from 4 months, but only at 11 months, high-level cues enhanced performance (R2 = .11). Thus, first signs of rapid categorization were evident from 4 months, but similar categorization patterns as in adults were recorded only from 11 months on.

Concept Parser With Multimodal Graph Learning for Video Captioning

Conventional video captioning methods are either stage-wise or simple end-to-end. While the former might introduce additional noise when exploiting off-the-shelf models to provide extra information, the latter suffers from lacking high-level cues. Therefore, a more desired framework should be able to capture multi-aspects of videos consistently. To this end, we present a concept-aware and task-specific model named CAT that accounts for both low-level visual and high-level concept cues, and incorporates them effectively in an end-to-end manner. Specifically, low-level visual and high-level concept features are obtained from the video transformer and concept parser of CAT. And a concept loss is further introduced to regularize the learning process of concept parser w.r.t. generated pseudo ground truth. To combine multi-level features, a caption transformer is later introduced in CAT, where visual and concept features are the inputs and caption is its output. In particular, we make critical design choices in the caption transformer to learn to exploit these cues with a multi-modal graph. This is achieved by a graph loss that enforces effective learning of intra and inter correlations between multi-level cues. Extensive experiments on three benchmark datasets demonstrate that CAT achieves 2.3 and 0.7 improvements in the CIDEr metric on MSVD and MSR-VTT compared to the state-of-the-art method SwinBERT [1], and also achieves a competitive result on VATEX.

Cognitive load does not increase reliance on speaker information in phonetic categorization.

Past research on speech perception has shown that speaker information, such as gender, affects phoneme categorization. Additionally, studies on listening under divided attention have argued that cognitive load decreases attention to phonetic detail and increases reliance on higher-level cues such as lexical information. This study examines the processing of speaker information under divided attention. The results of two perception experiments indicate that additional cognitive load does not increase listeners' reliance on the gender of the speaker during phoneme categorization tasks. This suggests that the processing of speaker information may pattern with lower-level acoustic rather than higher-level lexical information.

Second Language Experience Facilitates Sentence Recognition in Temporally-Modulated Noise for Non-native Listeners.

Non-native listeners deal with adverse listening conditions in their daily life much harder than native listeners. However, previous work in our laboratories found that native Chinese listeners with native English exposure may improve the use of temporal fluctuations of noise for English vowel identification. The purpose of this study was to investigate whether Chinese listeners can generalize the use of temporal cues for the English sentence recognition in noise. Institute of Electrical and Electronics Engineers (IEEE) sentence recognition in quiet condition, stationary noise, and temporally-modulated noise were measured for native American English listeners (EN), native Chinese listeners in the United States (CNU), and native Chinese listeners in China (CNC). Results showed that in general, EN listeners outperformed the two groups of CN listeners in quiet and noise, while CNU listeners had better scores of sentence recognition than CNC listeners. Moreover, the native English exposure helped CNU listeners use high-level linguistic cues more effectively and take more advantage of temporal fluctuations of noise to process English sentence in severely degraded listening conditions [i.e., the signal-to-noise ratio (SNR) of −12 dB] than CNC listeners. These results suggest a significant effect of language experience on the auditory processing of both speech and noise.

The interplay between gaze and consistency in scene viewing: Evidence from visual search by young and older adults

Searching for an object in a complex scene is influenced by high-level factors such as how much the item would be expected in that setting (semantic consistency). There is also evidence that a person gazing at an object directs our attention towards it. However, there has been little previous research that has helped to understand how we integrate top-down cues such as semantic consistency and gaze to direct attention when searching for an object. Also, there are separate lines of evidence to suggest that older adults may be more influenced by semantic factors and less by gaze cues compared to younger counterparts, but this has not been investigated before in an integrated task. In the current study we analysed eye-movements of 34 younger and 30 older adults as they searched for a target object in complex visual scenes. Younger adults were influenced by semantic consistency in their attention to objects, but were more influenced by gaze cues. In contrast, older adults were more guided by semantic consistency in directing their attention, and showed less influence from gaze cues. These age differences in use of high-level cues were apparent early in processing (time to first fixation and probability of immediate fixation) but not in later processing (total time looking at objects and time to make a response). Overall, this pattern of findings indicates that people are influenced by both social cues and prior expectations when processing a complex scene, and the relative importance of these factors depends on age.

What you hear first, is what you get: Initial metrical cue presentation modulates syllable detection in sentence processing.

Auditory rhythms create powerful expectations for the listener. Rhythmic cues with the same temporal structure as subsequent sentences enhance processing compared with irregular or mismatched cues. In the present study, we focus on syllable detection following matched rhythmic cues. Cues were aligned with subsequent sentences at the syllable (low-level cue) or the accented syllable (high-level cue) level. A different group of participants performed the task without cues to provide a baseline. We hypothesized that unaccented syllable detection would be faster after low-level cues, and accented syllable detection would be faster after high-level cues. There was no difference in syllable detection depending on whether the sentence was preceded by a high-level or low-level cue. However, the results revealed a priming effect of the cue that participants heard first. Participants who heard a high-level cue first were faster to detect accented than unaccented syllables, and faster to detect accented syllables than participants who heard a low-level cue first. The low-level-first participants showed no difference between detection of accented and unaccented syllables. The baseline experiment confirmed that hearing a low-level cue first removed the benefit of the high-level grouping structure for accented syllables. These results suggest that the initially perceived rhythmic structure influenced subsequent cue perception and its influence on syllable detection. Results are discussed in terms of dynamic attending, temporal context effects, and implications for context effects in neural entrainment.

Examining Whether Semantic Cues Can Affect Felt Heaviness When Lifting Novel Objects.

It is well established that manipulations of low-level stimulus properties unrelated to mass can impact perception of heaviness, the most famous example being the size-weight illusion whereby small objects feel heavier than equally-weighted larger objects. Interestingly, manipulations of high-level cues such as material have also induced weight illusions, highlighting that cognitive expectations alone are enough to create illusory weight differences. Less is known, however, about what type of cognitive expectations can influence perception of heaviness. As labels are often used to signify the heaviness of objects, this study examined whether semantic cues could induce a novel weight illusion. Participants lifted equally-sized and equally-weighted sets of objects labelled as ‘light’ and ‘heavy’ and reported their perceived heaviness both prior to and after lifting. Fingertip forces were also measured to understand how semantic cues may influence sensorimotor prediction. The labels clearly affected pre-lift-off expectations of heaviness. By contrast, we found no effect of these labels on the perceived heaviness of objects, nor on the forces used to grip and lift them on early trials. In other words, we find no evidence that semantic cues affect perception or action enough to induce a novel weight illusion. These findings suggest that the explicit expectations created by the labels did not dominate the implicit expectations created by the equal sizes of the objects, highlighting the segregated nature of cognitive expectations and their variable influences on perception and action.

Physical interaction makes invisible surfaces visible

Perhaps the most basic task faced by visual processing is to detect the presence of objects; before computing an object’s color, shape, or orientation, we must first register that something is there. Detection may be fairly straightforward when an object is fully visible, but in many realistic viewing conditions the visual system can only infer the presence of objects using cues such as continuity behind occluders, coincidental clipping of multiple figures, or unified motion against a background. All such cues, however, concern fairly low-level aspects of visual processing, encompassing only basic geometric and kinetic factors. Might more sophisticated forms of input cue an object’s presence? Here, we explore how a surprisingly high-level cue—physical interaction—can imply the presence of a hidden surface, in ways that directly facilitate later detection. Subjects saw an animation of a disk falling and then unexpectedly bouncing off of an ‘invisible’ surface. Sometimes, the disk bounced straight back up, implying a flat surface; other times, the disk’s bounce implied an angled surface. Afterwards, a visible line appeared where the disk had just bounced, whose orientation either matched or didn’t match the surface implied by the disk’s exit trajectory; subjects’ task was simply to report the orientation of this new, visible line, regardless of the physical events that came before it. Subjects were faster and more accurate at reporting the line’s orientation when it matched the orientation of the physically implied surface vs. when it conflicted. Follow-up experiments extended this work to attentive search in multi-event displays; again, detection of a specific oriented line was facilitated by seeing physical interactions that implied surfaces with that orientation. This work shows how a process as basic and seemingly low-level as detecting the presence of an object or contour can be influenced by a surprisingly high-level cue: otherwise-unexplained physical interactions.

Single Image Reflection Removal Using Non-Linearly Synthesized Glass Images and Semantic Context

An image captured through a glass plane usually contains both of a target transmitted scene behind the glass plane and a reflected scene in front of the glass plane. We propose a semantic context based network to remove reflection artifacts from a single glass image. We first investigate a non-linear intensity mapping relationship for glass images to synthesize more realistic training sets. Then we devise an efficient reflection removal network using multi-scale generators and an interpreter, where the semantic context of the transmission image is adopted as a high level cue for the interpreter to guide the generators. We also provide a new test data set of real glass images including the ground truth transmission and reflection images. Experiments are performed on four test data sets and we show that the proposed algorithm decomposes an input glass image into a transmission image and a reflection image more faithfully compared with the four existing state-of-the-art methods.

Occlusion-Aware Unsupervised Learning of Monocular Depth, Optical Flow and Camera Pose with Geometric Constraints

We present an occlusion-aware unsupervised neural network for jointly learning three low-level vision tasks from monocular videos: depth, optical flow, and camera motion. The system consists of three different predicting sub-networks simultaneously coupled by combined loss terms and is capable of computing each task independently on test samples. Geometric constraints extracted from scene geometry which have traditionally been used in bundle adjustment or pose-graph optimization are formed as various self-supervisory signals during our end-to-end learning approach. Different from prior works, our image reconstruction loss also takes account of optical flow. Moreover, we impose novel 3D flow consistency constraints over the predictions of all the three tasks. By explicitly modeling occlusion and taking utilization of both 2D and 3D geometry relationships, abundant geometric constraints are formed over estimated outputs, enabling the system to capture both low-level representations and high-level cues to infer thinner scene structures. Empirical evaluation on the KITTI dataset demonstrates the effectiveness and improvement of our approach: (1) monocular depth estimation outperforms state-of-the-art unsupervised methods and is comparable to stereo supervised ones; (2) optical flow prediction ranks top among prior works and even beats supervised and traditional ones especially in non-occluded regions; (3) pose estimation outperforms established SLAM systems under comparable input settings with a reasonable margin.

3D printing spatially varying color and translucency

We present an efficient and scalable pipeline for fabricating full-colored objects with spatially-varying translucency from practical and accessible input data via multi-material 3D printing. Observing that the costs associated with BSSRDF measurement and processing are high, the range of 3D printable BSSRDFs are severely limited, and that the human visual system relies only on simple high-level cues to perceive translucency, we propose a method based on reproducing perceptual translucency cues. The input to our pipeline is an RGBA signal defined on the surface of an object, making our approach accessible and practical for designers. We propose a framework for extending standard color management and profiling to combined color and translucency management using a gamut correspondence strategy we callopaque relative processing.We present an efficient streaming method to compute voxel-level material arrangements, achieving both realistic reproduction of measured translucent materials and artistic effects involving multiple fully or partially transparent geometries.

Attention region detection based on closure prior in layered bit Planes

An attention region usually possesses a closed contour and a significant color contrast against its surroundings. Inspired by this observation, a novel prior so called closure prior is introduced as an important high-level saliency cue in this paper. Specifically, firstly, the connectivity analysis is used in layered bit planes to extract closed regions based on closure prior. Secondly, the closed regions touching image boundaries are removed based on the assumption that most photographers will not crop attention regions along the view frame. Thirdly, based on the hypothesis that an image region should have a big contrast against its surroundings if it has more chances in all bit planes to be a closed region, the contrast contributions of a closed region in all bit planes are accumulated to obtain color contrast. Meanwhile, by taking account for the characteristics of human visual system according to the perception of small attention region, and the mechanism of human visual resource allocation between attention regions, several morphological filtering technologies are applied to the steps of color contrast calculating. Finally, the saliency map aiming for attention region detection is generated from the contrast map. The experimental results show the presented detection method achieves acceptable performance compared with twelve state-of-the-art models.

A learning-based visual saliency prediction model for stereoscopic 3D video (LBVS-3D)

Saliency prediction models provide a probabilistic map of relative likelihood of an image or video region to attract the attention of the human visual system. Over the past decade, many computational saliency prediction models have been proposed for 2D images and videos. Considering that the human visual system has evolved in a natural 3D environment, it is only natural to want to design visual attention models for 3D content. Existing monocular saliency models are not able to accurately predict the attentive regions when applied to 3D image/video content, as they do not incorporate depth information. This paper explores stereoscopic video saliency prediction by exploiting both low-level attributes such as brightness, color, texture, orientation, motion, and depth, as well as high-level cues such as face, person, vehicle, animal, text, and horizon. Our model starts with a rough segmentation and quantifies several intuitive observations such as the effects of visual discomfort level, depth abruptness, motion acceleration, elements of surprise, size and compactness of the salient regions, and emphasizing only a few salient objects in a scene. A new fovea-based model of spatial distance between the image regions is adopted for considering local and global feature calculations. To efficiently fuse the conspicuity maps generated by our method to one single saliency map that is highly correlated with the eye-fixation data, a random forest based algorithm is utilized. The performance of the proposed saliency model is evaluated against the results of an eye-tracking experiment, which involved 24 subjects and an in-house database of 61 captured stereoscopic videos. Our stereo video database as well as the eye-tracking data are publicly available along with this paper. Experiment results show that the proposed saliency prediction method achieves competitive performance compared to the state-of-the-art approaches.

Consistency of Border-Ownership Cells across Artificial Stimuli, Natural Stimuli, and Stimuli with Ambiguous Contours.

To distinguish different objects in a natural scene, the brain must segment the image into regions corresponding to objects. The so-called "border-ownership" cells appear to be dedicated to this task, as they signal for a given edge on which side the object is that owns it. Here, we report that individual border-ownership cells are unreliable when tested across a battery of artificial stimuli used previously but can signal border-ownership consistently as a population. We show that these border-ownership population signals are also suited for signaling border-ownership for natural objects and at longer latency, even for stimuli without local edge information. Our results suggest that border-ownership cells integrate both local, low-level and global, high-level cues to segment the scene.

Mechanisms of color perception and cognition covered by #thedress

Color is notoriously ambiguousmany color illusions existbut until now it has been thought that all people with normal color vision experience color illusions the same way. How does the visual system resolve color ambiguity? Here, we present work that addresses this question by quantifying peoples perception of a particularly ambiguous image, the dress photograph. The colors of the individual pixels in the photograph when viewed in isolation are light-blue or brown, but popular accounts suggest the dress appears either white/gold or blue/black. We tested more than 1400 people, both on-line and under controlled laboratory conditions. Subjects first completed the sentence: this is a ___and___dress. Then they performed a color-matching experiment that did not depend on language. Surprisingly, the results uncovered three groups of subjects: white/gold observers, blue/black observers and blue/brown observers. Our findings show that the brain resolves ambiguity in the dress into one of three stable states; a minority of people switched which colors they saw (~11%). It is clear that what we see depends on both retinal stimulation and internal knowledge about the world. Cases of multi-stability such as the dress provide a rare opportunity to investigate this interplay. In particular, we go on to demonstrate that the dress photograph can be used as a tool to discover that skin reflectance is a particularly important implicit cue used by the brain to estimate the color of the light source, to resolve color ambiguity, shedding light on the role of high-level cues in color perception. Meeting abstract presented at VSS 2016

Stereo reconstruction using top-down cues

We present a framework which allows standard stereo reconstruction to be unified with a wide range of classic top-down cues from urban scene understanding. The resulting algorithm is analogous to the human visual system where conflicting interpretations of the scene due to ambiguous data can be resolved based on a higher level understanding of urban environments. The cues which are reformulated within the framework include: recognising common arrangements of surface normals and semantic edges (e.g. concave, convex and occlusion boundaries), recognising connected or coplanar structures such as walls, and recognising collinear edges (which are common on repetitive structures such as windows). Recognition of these common configurations has only recently become feasible, thanks to the emergence of large-scale reconstruction datasets. To demonstrate the importance and generality of scene understanding during stereo-reconstruction, the proposed approach is integrated with 3 different state-of-the-art techniques for bottom-up stereo reconstruction. The use of high-level cues is shown to improve performance by up to 15% on the Middlebury 2014 and KITTI datasets. We further evaluate the technique using the recently proposed HCI stereo metrics, finding significant improvements in the quality of depth discontinuities, planar surfaces and thin structures.

A Biologically Inspired Automatic System for Media Quality Assessment

Photo aesthetic quality evaluation is a challenging task in artificial intelligence systems. In this paper, we propose a biologically inspired aesthetic descriptor that mimicks humans sequentially perceiving visually/semantically salient In general, visually salient regions are perceived by low-level visual features, such as the high contrast between the foreground and the background objects; while semantically salient regions are perceived by high-level visual features such as human faces. regions in a photo. In particular, a weakly supervised learning paradigm is developed to project the local image descriptors into a low-dimensional semantic space. Then, each graphlet can be described by multiple types of visual features, both in low-level and in high-level. Since humans usually perceive only a few salient regions in a photo, a sparsity-constrained graphlet ranking algorithm is proposed that seamlessly integrates both the low-level and the high-level visual cues. Top-ranked graphlets are those visually/semantically prominent local aesthetic descriptors in a photo. They are sequentially linked into a path that simulates humans actively viewing process. Finally, we learn a probabilistic aesthetic measure based on such actively viewing paths (AVPs) from the training photos. Experimental results show that: 1) the AVPs are 87.65% consistent with real human gaze shifting paths, as verified by the eye-tracking data and 2) our aesthetic measure outperforms many of its competitors.

Learning to Predict Sequences of Human Visual Fixations.

Most state-of-the-art visual attention models estimate the probability distribution of fixating the eyes in a location of the image, the so-called saliency maps. Yet, these models do not predict the temporal sequence of eye fixations, which may be valuable for better predicting the human eye fixations, as well as for understanding the role of the different cues during visual exploration. In this paper, we present a method for predicting the sequence of human eye fixations, which is learned from the recorded human eye-tracking data. We use least-squares policy iteration (LSPI) to learn a visual exploration policy that mimics the recorded eye-fixation examples. The model uses a different set of parameters for the different stages of visual exploration that capture the importance of the cues during the scanpath. In a series of experiments, we demonstrate the effectiveness of using LSPI for combining multiple cues at different stages of the scanpath. The learned parameters suggest that the low-level and high-level cues (semantics) are similarly important at the first eye fixation of the scanpath, and the contribution of high-level cues keeps increasing during the visual exploration. Results show that our approach obtains the state-of-the-art performances on two challenging data sets: 1) OSIE data set and 2) MIT data set.

Supplemental Material for Snack Intake Is Reduced Using an Implicit, High-Level Construal Cue

Supplemental Material for Snack Intake Is Reduced Using an Implicit, High-Level Construal Cue