Segmentation Cues Research Articles

Tap and Shoot Segmentation

We present a new segmentation method that leverages latent photographic information available at the moment of taking pictures. Photography on a portable device is often done by tapping to focus before shooting the picture. This tap-and-shoot interaction for photography not only specifies the region of interest but also yields useful focus/defocus cues for image segmentation. However, most of the previous interactive segmentation methods address the problem of image segmentation in a post-processing scenario without considering the action of taking pictures. We propose a learning-based approach to this new tap-and-shoot scenario of interactive segmentation. The experimental results on various datasets show that, by training a deep convolutional network to integrate the selection and focus/defocus cues, our method can achieve higher segmentation accuracy in comparison with existing interactive segmentation methods.

Does sound structure affect word learning? An eye-tracking study of Danish learning toddlers

Previous research has shown that Danish-learning children lag behind in early lexical acquisition compared with children learning a number of other languages. This delay has been ascribed to the opaque phonetic structure of Danish, which appears to have fewer reliable segmentation cues than other closely related languages. In support of this hypothesis, recent work has shown that the phonetic properties of Danish negatively affect online language processing in young Danish children. In this study, we used eye-tracking to investigate whether the challenges associated with processing Danish also affect how Danish-learning children between 24 and 35 months of age establish and learn novel label–object mappings. The children were presented with a series of novel mappings, either ostensively (one novel object presented alone on the screen) or ambiguously (one novel object presented together with a familiar one), through carrier phrases with different phonetic structures (more vs less opaque). Our results showed two main trends. First, Danish-learning children performed poorly on the task of mapping novel labels onto novel objects. Second, when learning did occur, accuracy was affected by the phonetic opacity of the speech stimuli. We suggest that this finding results from the interplay of a perceptually challenging speech input and a slower onset of early vocabulary experience, which in turn may delay the onset of word learning skills in Danish-learning children.

Prosodic and segmental cues to regional dialect variation in American English

Historically, most research on regional variation in American English (AE) examined lexical and segmental sources of variability. Focusing on prosodic variation, we have demonstrated significant effects of regional dialect on overall articulation rate, distributions of pauses, pitch accents, phrasal-boundary tone combinations, and syllable-to-syllable vowel and consonant duration variability (Clopper & Smiljanic, 2015, 2011). In this talk, we examine whether native AE listeners utilize prosodic and segmental cues to classify AE talkers by region and whether listener familiarity with different dialects affects classification. Using free classification with unmodified, monotone (flattened F0), and low-pass filtered (removed segmental information) stimuli, listeners from Ohio and Texas grouped 60 talkers based on perceived regional similarities. Listeners were more accurate in classifying talkers with the unmodified and monotone stimuli than with the low-pass filtered stimuli. There were no differences between the two listener groups in accuracy. Multidimensional scaling analyses indicated that listeners used gender and region as salient dimensions in classifying talkers in the unmodified and monotone conditions, but not in the low-pass filtered condition. The results suggest that segmental cues facilitated classification whereas intonation and global temporal cues on their own did not. The relationship between acoustic-phonetic features and classification patterns will be discussed.

Developmental Differences Between Children and Adults in the Use of Visual Cues for Segmentation.

Recent work asked if visual cues facilitate word segmentation in adults and infants (Thiessen, 2010). While adults showed better word segmentation when presented with a regular visual cue (consistent mapping between words and objects), infants did not. This difference was attributed to infants' lack of understanding that objects have labels. Alternatively, infants' performance could reflect their difficulty with tracking and integrating multiple multimodal cues. We contrasted these two accounts by looking at the effect of visual cues on word segmentation in adults and across childhood (6-12years). We found that older children (Mage 10;7) benefitted from the regular visual cues, but younger children (Mage 7;10), who already knew that objects have labels, did not. Knowing that objects have labels was not enough to use visual cues as an aid for segmentation. These findings show that the ability to integrate multimodal cues develops during childhood, and it is not yet adult-like in children.

Consistency functional map propagation for repetitive patterns

Repetitive patterns appear frequently in both man-made and natural environments. Automatically and robustly detecting such patterns from an image is a challenging problem. We study repetitive pattern alignment by embedding segmentation cue with a functional map model. However, this model cannot tackle the repetitive patterns directly due to the large photometric and geometric variations. Thus, a consistency functional map propagation (CFMP) algorithm that extends the functional map with dynamic propagation is proposed to address this issue. This propagation model is acquired in two steps. The first one aligns the patterns from a local region, transferring segmentation functions among patterns. It can be cast as an L 2,1 norm optimization problem. The latter step updates the template segmentation for the next round of pattern discovery by merging the transferred segmentation functions. Extensive experiments and comparative analyses have demonstrated an encouraging performance of the proposed algorithm in detection and segmentation of repetitive patterns.

Visual statistical learning provides scaffolding for emerging object representations

Although an abundance of studies demonstrated human's abilities for visual statistical learning (VSL), much fewer studies focused on the consequences of VSL. Recent papers reported that attention is biased toward detected statistical regularities, but this observation was restricted to spatial locations and provided no functional interpretation of the phenomenon. We tested the idea that statistical regularities identified by VSL constrain subsequent visual processing by coercing further processing to be compatible with those regularities. Our paradigm used the well-documented fact that within-object processing has an advantage over across-object processing. We combined the standard VSL paradigm with a visual search task in order to assess whether participants detect a target better within a statistical chunk than across chunks. Participants (N=11) viewed 4-4 alternating blocks of "observation" and "search" trials. In both blocks, complex multi-shape visual scenes were presented, which unbeknownst to the participants, were built from pairs of abstract shapes without any clear segmentation cues. Thus, the visual chunks (pairs of shapes) generating the scenes could only be extracted by tracking the statistical contingencies of shapes across scenes. During "observation", participants just passively observed the visual scenes, while during "search", they performed a 3-AFC task deciding whether T letters appearing in the middle of the shapes formed a horizontal or vertical pairs. Despite identical distance between the target letters, participants performed significantly better in trials in which targets appeared within a visual chunk than across two chunks or across a chunk and a single shape. These results suggest that similar to object-defined within/between relations, statistical contingencies learned implicitly by VSL facilitate visual processing of elements that belong to the same statistical chunk. This similarity between the effects of true objects and statistical chunks support the notion that VSL has a central role in the emergence of internal object representations. Meeting abstract presented at VSS 2017

Motion-Appearance Interactive Encoding for Object Segmentation in Unconstrained Videos

We present a novel method of integrating motion and appearance cues for foreground object segmentation in unconstrained videos. Unlike conventional methods encoding motion and appearance patterns individually, our method puts particular emphasis on their mutual assistance. Specifically, we propose using an interactively constrained encoding (ICE) scheme to incorporate motion and appearance patterns into a graph that leads to a spatiotemporal energy optimization. The reason of utilizing ICE is that both motion and appearance cues for the same target share underlying correlative structure, thus can be exploited in a deeply collaborative manner. We perform ICE not only in the initialization but also in the refinement stage of a two-layer framework for object segmentation. This scheme allows our method to consistently capture structural patterns about object perceptions throughout the whole framework. Our method can be operated on superpixels instead of raw pixels to reduce the number of graph nodes by two orders of magnitude. Moreover, we propose to partially explore the multi-object localization problem with inter-occlusion by weighted bipartite graph matching. Comprehensive experiments on three benchmark datasets (i.e., SegTrack, MOViCS, and GaTech) demonstrate the effectiveness of our approach compared with extensive state-of-the-art methods.

Experience with a second language affects the use of fundamental frequency in speech segmentation.

This study investigates whether listeners’ experience with a second language learned later in life affects their use of fundamental frequency (F0) as a cue to word boundaries in the segmentation of an artificial language (AL), particularly when the cues to word boundaries conflict between the first language (L1) and second language (L2). F0 signals phrase-final (and thus word-final) boundaries in French but word-initial boundaries in English. Participants were functionally monolingual French listeners, functionally monolingual English listeners, bilingual L1-English L2-French listeners, and bilingual L1-French L2-English listeners. They completed the AL-segmentation task with F0 signaling word-final boundaries or without prosodic cues to word boundaries (monolingual groups only). After listening to the AL, participants completed a forced-choice word-identification task in which the foils were either non-words or part-words. The results show that the monolingual French listeners, but not the monolingual English listeners, performed better in the presence of F0 cues than in the absence of such cues. Moreover, bilingual status modulated listeners’ use of F0 cues to word-final boundaries, with bilingual French listeners performing less accurately than monolingual French listeners on both word types but with bilingual English listeners performing more accurately than monolingual English listeners on non-words. These findings not only confirm that speech segmentation is modulated by the L1, but also newly demonstrate that listeners’ experience with the L2 (French or English) affects their use of F0 cues in speech segmentation. This suggests that listeners’ use of prosodic cues to word boundaries is adaptive and non-selective, and can change as a function of language experience.

The functional weight of a prosodic cue in the native language predicts the learning of speech segmentation in a second language

This study newly investigates whether the functional weight of a prosodic cue in the native language predicts listeners’ learning and use of that cue in second-language speech segmentation. It compares English and Dutch listeners’ use of fundamental-frequency (F0) rise as a cue to word-final boundaries in French. F0 rise signals word-initial boundaries in English and Dutch, but has a weaker functional weight in English than Dutch because it is more strongly correlated with vowel quality in English than Dutch. English- and Dutch-speaking learners of French matched in French proficiency and experience, and native French listeners completed a visual-world eye-tracking experiment in French where they monitored words ending with/out an F0 rise (replication of Tremblay, Broersma, Coughlin & Choi, 2016). Dutch listeners made earlier/greater use of the F0 rise than English listeners, and in one condition they made greater use of F0 rise than French listeners, extending the cue-weighting theory to speech segmentation.

Crowd Segmentation Using both Appearance and Stereo Information

Crowd segmentation is an important issue in video surveillance. With the decrease in their cost, stereo cameras can be used to help develop new algorithms to achieve better accuracy in crowd segmentation. This paper aims to develop a method to explore the depth cues for crowd segmentation in video surveillance. The contributions of this paper are twofold. First, a novel crowd segmentation method closely coupling appearance and stereo information has been developed. Instead of performing disparity calculation as a preprocessing step, stereo information is obtained concurrently with appearance-based crowd segmentation. Second, an object-level disparity algorithm is proposed for object segmentation in surveillance scenarios. Only one disparity value for each hypothetical object greatly reduces the computational complexity and simplifies the segmentation method. Experimental results and quantitative evaluations based on two surveillance scenarios are presented in this paper. The results consistently show the effectiveness of the algorithm in exploring depth cues for crowd segmentation.

Segmental cues to intonation of statements and polar questions in whispered, semi-whispered and normal speech modes

This paper examines how acoustic characteristics of vowels and consonants reflect intonational differences between polar questions and statements in Polish whispered, semi-whispered and normal speech modes, with particular focus on the spectral characteristics of voiceless consonants as a function of intonation, and across speech modes. The results reveal significant differences in spectral properties of both utterance-final vowels and consonants across statements and polar questions. Questions have higher vowel intensities and show differences in formant frequencies that vary with speech mode. Regarding the consonants, both fricatives and affricates are produced with higher intensity, spectral peaks at higher frequencies, and higher Centre of Gravity and Spectral Standard Deviation values in questions than in statements. Conversely, skewness and kurtosis are lower in questions than in statements. Some spectral features of sibilants, including spectral slopes, show greater question-statement differences in the whispered speech mode than in other speech modes. The finding that some cues are more pronounced in whispered speech suggests that they may compensate for the absence of fundamental frequency in this mode. Most generally, the study shows that speakers produce intended intonation patterns by varying the type and magnitude of cues depending on speech mode.

Stereoscopic Segmentation Cues Improve Visual Timing Performance in Spatiotemporally Cluttered Environments.

Recently, Cass and Van der Burg demonstrated that temporal order judgment (TOJ) precision could be profoundly impaired by the mere presence of dynamic visual clutter elsewhere in the visual field. This study examines whether presenting target and distractor objects in different depth planes might ameliorate this remote temporal camouflage (RTC) effect. TOJ thresholds were measured under static and dynamic (flickering) distractor conditions. In Experiment 1, targets were presented at zero, crossed, or uncrossed disparity, with distractors fixed at zero disparity. Thresholds were significantly elevated under dynamic compared with static contextual conditions, replicating the RTC effect. Crossed but not uncrossed disparity targets improved performance in dynamic distractor contexts, which otherwise produce substantial RTC. In Experiment 2, the assignment of disparity was reversed: targets fixed at zero disparity; distractors crossed, uncrossed, or zero. Under these conditions, thresholds improved significantly in the nonzero distractor disparity conditions. These results indicate that presenting target and distractor objects in different planes can significantly improve TOJ performance in dynamic conditions. In Experiment 3, targets were each presented with a different sign of disparity (e.g., one crossed and the other uncrossed), with no resulting performance benefits. Results suggest that disparity can be used to alleviate the performance-diminishing effects of RTC, but only if both targets constitute a single and unique disparity-defined surface.

Lexical Segmentation in Artificial Word Learning: The Effects of Converging Sublexical Cues.

This study examines how French listeners segment and learn new words of artificial languages varying in the presence of different combinations of sublexical segmentation cues. The first experiment investigated the contribution of three different types of sublexical cues (acoustic-phonetic, phonological and prosodic cues) to word learning. The second experiment explored how participants specifically exploited sublexical prosodic cues. Whereas complementary cues signaling word-initial and word-final boundaries had synergistic effects on word learning in the first experiment, the two manipulated prosodic cues redundantly signaling word-final boundaries in the second experiment were rank-ordered with final pitch variations being more weighted than final lengthening. These results are discussed in light of the notions of cue type, cue position and cue efficiency.

Cross-linguistic differences in the use of durational cues for the segmentation of a novel language.

It is widely accepted that duration can be exploited as phonological phrase final lengthening in the segmentation of a novel language, i.e., in extracting discrete constituents from continuous speech. The use of final lengthening for segmentation and its facilitatory effect has been claimed to be universal. However, lengthening in the world languages can also mark lexically stressed syllables. Stress-induced lengthening can potentially be in conflict with right edge phonological phrase boundary lengthening. Thus the processing of durational cues in segmentation can be dependent on the listener's linguistic background, e.g., on the specific correlates and unmarked location of lexical stress in the native language of the listener. We tested this prediction and found that segmentation by both German and Basque speakers is facilitated when lengthening is aligned with the word final syllable and is not affected by lengthening on either the penultimate or the antepenultimate syllables. Lengthening of the word final syllable, however, does not help Italian and Spanish speakers to segment continuous speech, and lengthening of the antepenultimate syllable impedes their performance. We have also found a facilitatory effect of penultimate lengthening on segmentation by Italians. These results confirm our hypothesis that processing of lengthening cues is not universal, and interpretation of lengthening as a phonological phrase final boundary marker in a novel language of exposure can be overridden by the phonology of lexical stress in the native language of the listener.

Saliency-Aware Video Object Segmentation.

Video saliency, aiming for estimation of a single dominant object in a sequence, offers strong object-level cues for unsupervised video object segmentation. In this paper, we present a geodesic distance based technique that provides reliable and temporally consistent saliency measurement of superpixels as a prior for pixel-wise labeling. Using undirected intra-frame and inter-frame graphs constructed from spatiotemporal edges or appearance and motion, and a skeleton abstraction step to further enhance saliency estimates, our method formulates the pixel-wise segmentation task as an energy minimization problem on a function that consists of unary terms of global foreground and background models, dynamic location models, and pairwise terms of label smoothness potentials. We perform extensive quantitative and qualitative experiments on benchmark datasets. Our method achieves superior performance in comparison to the current state-of-the-art in terms of accuracy and speed.

Speech segmentation by statistical learning is supported by domain-general processes within working memory.

The purpose of this study was to examine the extent to which working memory resources are recruited during statistical learning (SL). Participants were asked to identify novel words in an artificial speech stream where the transitional probabilities between syllables provided the only segmentation cue. Experiments 1 and 2 demonstrated that segmentation performance improved when the speech rate was slowed down, suggesting that SL is supported by some form of active processing or maintenance mechanism that operates more effectively under slower presentation rates. In Experiment 3 we investigated the nature of this mechanism by asking participants to perform a two-back task while listening to the speech stream. Half of the participants performed a two-back rhyme task designed to engage phonological processing, whereas the other half performed a comparable two-back task on un-nameable visual shapes. It was hypothesized that if SL is dependent only upon domain-specific processes (i.e., phonological rehearsal), the rhyme task should impair speech segmentation performance more than the shape task. However, the two loads were equally disruptive to learning, as they both eradicated the benefit provided by the slow rate. These results suggest that SL is supported by working-memory processes that rely on domain-general resources.

Listeners use temporal information to identify French- and English-accented speech

Which acoustic cues can be used by listeners to identify speakers’ linguistic origins in foreign-accented speech? We investigated accent identification performance in signal-manipulated speech, where (a) Swiss German listeners heard native German speech to which we transplanted segment durations of French-accented German and English-accented German, and (b) Swiss German listeners heard 6-band noise-vocoded French-accented and English-accented German speech to which we transplanted native German segment durations. Therefore, the foreign accent cues in the stimuli consisted of only temporal information (in a) and only strongly degraded spectral information (in b). Findings suggest that listeners were able to identify the linguistic origin of French and English speakers in their foreign-accented German speech based on temporal features alone, as well as based on strongly degraded spectral features alone. When comparing these results to previous research, we found an additive trend of temporal and spectral cues: identification performance tended to be higher when both cues were present in the signal. Acoustic measures of temporal variability could not easily explain the perceptual results. However, listeners were drawn towards some of the native German segmental cues in condition (a), which biased responses towards ‘French’ when stimuli featured uvular /r/s and towards ‘English’ when they contained vocalized /r/s or lacked /r/.

Using Predictability for Lexical Segmentation.

This study investigates a strategy based on predictability of consecutive sub-lexical units in learning to segment a continuous speech stream into lexical units using computational modeling and simulations. Lexical segmentation is one of the early challenges during language acquisition, and it has been studied extensively through psycholinguistic experiments as well as computational methods. However, despite strong empirical evidence, the explicit use of predictability of basic sub-lexical units in models of segmentation is underexplored. This paper presents an incremental computational model of lexical segmentation for exploring the usefulness of predictability for lexical segmentation. We show that the predictability cue is a strong cue for segmentation. Contrary to earlier reports in the literature, the strategy yields state-of-the-art segmentation performance with an incremental computational model that uses only this particular cue in a cognitively plausible setting. The paper also reports an in-depth analysis of the model, investigating the conditions affecting the usefulness of the strategy.

Functional load of fundamental frequency in the native language predicts learning and use of these cues in second-language speech segmentation

This study investigates whether second-language (L2) learners make greater use of prosodic cues to word boundaries if these cues have a higher functional load in the native language (L1). It examines the use of fundamental-frequency (F0) rise in the segmentation of French speech by English- and Dutch-speaking L2 learners of French matched in French proficiency and experience (and native French listeners). In both English and Dutch, an F0 rise tends to signal word-initial boundaries in pitch-accented words with initial stress, but English has more vowel reduction than Dutch, and thus full vowels also signal word-initial boundaries in English. By contrast, in French, an F0 rise tends to signal word-final boundaries. Participants completed an eye-tracking experiment: They heard sentences where the monosyllabic target ended or did not end with an F0 rise (duration being held constant), and saw four orthographic words in the display, including the target (bal “ball“) and a disyllabic competitor (balcon “balcon...

Cue combination encoding via contextual modulation of V1 and V2 neurons.

Neurons in early visual cortical areas encode the local properties of a stimulus in a number of different feature dimensions such as color, orientation, and motion. It has been shown, however, that stimuli presented well beyond the confines of the classical receptive field can augment these responses in a way that emphasizes these local attributes within the greater context of the visual scene. This mechanism imparts global information to cells that are otherwise considered local feature detectors and can potentially serve as an important foundation for surface segmentation, texture representation, and figure–ground segregation. The role of early visual cortex toward these functions remains somewhat of an enigma, as it is unclear how surface segmentation cues are integrated from multiple feature dimensions. We examined the impact of orientation- and motion-defined surface segmentation cues in V1 and V2 neurons using a stimulus in which the two features are completely separable. We find that, although some cells are modulated in a cue-invariant manner, many cells are influenced by only one cue or the other. Furthermore, cells that are modulated by both cues tend to be more strongly affected when both cues are presented together than when presented individually. These results demonstrate two mechanisms by which cue combinations can enhance salience. We find that feature-specific populations are more frequently encountered in V1, while cue additivity is more prominent in V2. These results highlight how two strongly interconnected areas at different stages in the cortical hierarchy can potentially contribute to scene segmentation.