Feature Cues Research Articles

Memory-enhanced hierarchical transformer for video paragraph captioning

Video paragraph captioning aims to describe a video that contains multiple events with a paragraph of generated coherent sentences. Such a captioning task is full of challenges since the high requirements for visual-textual relevance and semantic coherence across the captioning paragraph of a video. In this work, we introduce a memory-enhanced hierarchical transformer for video paragraph captioning. Our model adopts a hierarchical structure, where the outer layer transformer extracts visual information from a global perspective and captures the relevancy between event segments throughout the entire video, while the inner layer transformer further mines local details within each event segment. By thoroughly exploring both global and local visual information at the video and event levels, our model can provide comprehensive visual feature cues for promising paragraph caption generation. Additionally, we design a memory module to capture similar patterns among event segments within a video, which preserves contextual information across event segments and updates its memory state accordingly. Experimental results on two popular datasets, ActivityNet Captions and YouCook2, demonstrate that our proposed model can achieve superior performance, generating higher quality caption while maintaining consistency in the content of video.

Unravelling the object-based nature of visual working memory: insight from pointers.

Visual working memory (VWM) plays a crucial role in temporarily storing and processing visual information, but the nature of stored representations and their interaction with new inputs has long been unclear. The pointer system refers to how VWM links new sensory inputs to stored information using specific cues. This study aimed to investigate whether the pointer system is based on spatial, feature-based, or object-based cues by employing the repetition benefit effect, where memory performance improves with repeated memory items. Across three experiments, we manipulated spatial positions, shapes, and colors as pointer cues to determine how these features affect VWM consolidation and updating. The results showed that while spatial location serves as a strong pointer cue, shape and color features can also effectively reestablish object correspondence in VWM. These findings support the view that the pointer system in VWM is flexible and object-based, utilizing various feature cues to maintain memory continuity. This study provides new insights into how VWM connects new inputs with stored information through the pointer system.

A multitask tensor-based relation network for cloth-changing person re-identification

The Cloth-changing person re-identification (CC-ReID) is more challenging than person re-identification (ReID) because of the unreliability of cloth-relevant features. Existing CC-ReID methods output the unique latent vector feature for the same pedestrian image. However, on the one hand, the latent vector feature generated by the pooling layer will lose more spatial information. On the other hand, the features that they focus on are the same for the same pedestrian image under different comparison pedestrian images, which does not consider the pedestrian pair relation information. For this, we propose a Multitask Tensor-based Relation Network (MTTRN) for CC-ReID. In MTTRN, we utilize the augmented cloth-changing identity images, human parsing images and head images to guide the model learning more fine-grained cloth-irrelevant feature cues. We propose a novel channel subspace generator and use the tensor mode-n product to generate the subspace tensor features instead of the vector feature generated by the pooling layer, which can retain more spatial feature information. Furthermore, we assume that the model will focus on different features for the same pedestrian image under different comparison pedestrian images in the same feature subspace or the same pedestrian image pair in different feature subspaces, in which the tensor feature pair relation is considered fully to mine more robust features. Extensive experiments show that our method achieves the state-of-the-art or competitive performance on three CC-ReID benchmark datasets and demonstrate the robustness of our model.

Salience Beats Individual Cue Preferences

Animals can use different cues to memorise food resources, which can be broadly divided into spatial and feature cues. Differences in species-specific preferences for spatial or feature cues has been historically linked to their reliance on spatial information (food-caching and migratory species prefer spatial cues), but more recently, attention has shifted to procedural differences largely affecting the salience of cues. Here, we investigated cue preferences in the colour-polymorphic Gouldian Finch to test a range of hypotheses. Birds were trained to find food either in a white or coloured feeder always positioned in the same location. In experimental trials, feeders were swapped, dissociating the feature and the spatial cue from the feature cue rewarded. Similar numbers of birds preferred the feature or the spatial cue in the first experimental trial. Preferences were not affected by head colour morph, sex or age group. However, birds with a spatial preference changed their choices over time, resulting in a significant number of birds choosing the feature cue at the end of the experiment. These results indicate that Gouldian Finches have individual preferences for one cue type or the other but can change their preference flexibly depending on the salience of the cue.

Use of geometry and featural cues in landmark configurations to reorient DCD children to the VR space: A route-learning study

Background. Working memory-based spatial cognition has attracted the attention of the scientific community in navigation and reorientation projects. The dominant approach considers that spontaneous spatial navigation behavior is based merely on environmental geometry (built and natural environmental objects). In this domain, DCD (Developmental Coordination Disorder) motor skill orientation problems have been frequently associated with poor visuospatial cognition, while immersive VR environments encourage more repetition, allowing for faster motor skill development and recovery. Objective. This pilot study tested the functionality of an immersive VR environment with environmental geometry (rectangular arena rich in symmetry) and featural landmark cues (striped wall, flora) as a route-learning tool for children with motor skill disorders. Methods. Forty DCD children aged 5 to 8 years (20 boys and 20 girls); five (5) 3D reality modeling setups with orthogonality, symmetry, and striped walls as design parameters; and trial walk-through coordination exercises using a predefined visual pathway with different motor control conditions (daylight, darkness). Participants’ path completion rate, path completion time, and walk-through satisfaction were recorded as route-learning performance variables and analyzed statistically. Results/findings. DCD children’s spatial orientation was statistically shown to be more stable and robust (in path completion rates, termination time, and walk-through level of satisfaction) in a virtual 3D environment rich in orthogonality, symmetry, and featural cues as landmarks. In this compound environmental geometry setup, training functionality and immersive learning performance enjoyed an 8.16% better path completion rate, a 12.37% reduction in path completion time, and 32.10% more walk-through satisfaction than reality modeling setups poor in geometry and landmarks. The effectiveness and robustness were validated statistically. Conclusion. Children with motor skill difficulties train and learn better in virtual 3D environments that are rich in orthogonality, symmetry, and featural landmark cues.

MFTSC: A Semantically Constrained Method for Urban Building Height Estimation Using Multiple Source Images

The use of remote sensing imagery has significantly enhanced the efficiency of building extraction; however, the precise estimation of building height remains a formidable challenge. In light of ongoing advancements in computer vision, numerous techniques leveraging convolutional neural networks and Transformers have been applied to remote sensing imagery, yielding promising outcomes. Nevertheless, most existing approaches directly estimate height without considering the intrinsic relationship between semantic building segmentation and building height estimation. In this study, we present a unified architectural framework that integrates the tasks of building semantic segmentation and building height estimation. We introduce a Transformer model that systematically merges multi-level features with semantic constraints and leverages shallow spatial detail feature cues in the encoder. Our approach excels in both height estimation and semantic segmentation tasks. Specifically, the coefficient of determination (R2) in the height estimation task attains a remarkable 0.9671, with a root mean square error (RMSE) of 1.1733 m. The mean intersection over union (mIoU) for building semantic segmentation reaches 0.7855. These findings underscore the efficacy of multi-task learning by integrating semantic segmentation with height estimation, thereby enhancing the precision of height estimation.

CTG-Net: an efficient cascaded framework driven by terminal guidance mechanism for dilated pancreatic duct segmentation

Pancreatic duct dilation indicates a high risk of various pancreatic diseases. Segmentation for dilated pancreatic duct (DPD) on computed tomography (CT) image shows the potential to assist the early diagnosis, surgical planning and prognosis. Because of the DPD’s tiny size, slender tubular structure and the surrounding distractions, most current researches on DPD segmentation achieve low accuracy and always have segmentation errors on the terminal DPD regions. To address these problems, we propose a cascaded terminal guidance network to efficiently improve the DPD segmentation performance. Firstly, a basic cascaded segmentation architecture is established to get the pancreas and coarse DPD segmentation, a DPD graph structure is build on the coarse DPD segmentation to locate the terminal DPD regions. Then, a terminal anatomy attention module is introduced for jointly learning the local intensity from the CT images, feature cues from the coarse DPD segmentation and global anatomy information from the designed pancreas anatomy-aware maps. Finally, a terminal distraction attention module which explicitly learns the distribution of the terminal distraction regions is proposed to reduce the false positive and false negative predictions. We also propose a new metric called tDice to measure the terminal segmentation accuracy for targets with tubular structures and two other metrics for segmentation error evaluation. We collect our dilated pancreatic duct segmentation dataset with 150 CT scans from patients with five types of pancreatic tumors. Experimental results on our dataset show that our proposed approach boosts DPD segmentation accuracy by nearly 20% compared with the existing results, and achieves more than 9% improvement for the terminal segmentation accuracy compared with the state-of-the-art methods.

Orientation by environmental geometry and feature cues in the green and black poison frog (Dendrobates auratus).

The ability to use environmental geometry when orienting in space reflects an animal's ability to use a global, allocentric framework. Therefore, understanding when and how animal's use geometry relative to other types of cues in the environment has interested comparative cognition researchers for decades. Yet, only two amphibians have been tested to date. We trained the poison frog Dendrobates auratus to find goal shelters in a rectangular arena, in the presence and absence of a feature cue, and assessed the relative influence of the two types of cues using probe trials. We chose D. auratus because the species has complex interactions with their physical and social environments, including parental care that requires navigating to and from distant locations. We found that, like many vertebrates, D. auratus are capable of using geometric information to relocate goals. In addition, the frogs preferentially used the more reliable feature cue when the location of the feature conflicted with the geometry of the arena. The frogs were equally successful at using the feature cue when it was proximal or distal to the goal shelter, consistent with prior studies that found that D. auratus can use distal cues in a flexible manner. Our results provide further evidence that amphibians can use environmental geometry during orientation. Future studies that examine when and how amphibians use geometry relative to other types of cues will contribute to a more complete picture of spatial cognition in this important, yet understudied, group.

Investigating boundary-geometry use by whip spiders (Phrynus marginemaculatus) during goal-directed navigation.

Previous studies have shown that whip spiders (Amblypygi) can use a variety of cues to navigate to and recognize a home refuge. The current study aimed to determine whether whip spiders were capable of using the boundary geometry of an experimental space (geometric information) to guide goal-directed navigation and to investigate any preferential use of geometric or feature (visual) information. Animals were first trained to find a goal location situated in one corner of a rectangular arena (geometric information) fronting a dark-green-colored wall, which created a brightness contrast with the other three white walls (feature information). Various probe trials were then implemented to determine cue use. It was found that animals were capable of directing their choice behavior towards geometrically correct corners at a rate significantly higher than chance, even when the feature cue was removed. By contrast, choice behavior dropped to random chance when geometric information was removed (test in a square arena) and only feature information remained. Choice behavior was also reduced to chance when geometric and feature information were set in conflict (by moving the feature cue to one of the longer walls in the rectangular arena). The data thus suggest that whip spiders are capable of using geometric information to guide goal-directed navigation and that geometric information is preferred over feature guidance, although a feature cue may set the context for activating geometry-guided navigation. Experimental design limitations and future directions are discussed.

Semantic-Aligned Cross-Modal Visual Grounding Network with Transformers

Multi-modal deep learning methods have achieved great improvements in visual grounding; their objective is to localize text-specified objects in images. Most of the existing methods can localize and classify objects with significant appearance differences but suffer from the misclassification problem for extremely similar objects, due to inadequate exploration of multi-modal features. To address this problem, we propose a novel semantic-aligned cross-modal visual grounding network with transformers (SAC-VGNet). SAC-VGNet integrates visual and textual features with semantic alignment to highlight important feature cues for capturing tiny differences between similar objects. Technically, SAC-VGNet incorporates a multi-modal fusion module to effectively fuse visual and textual descriptions. It also introduces contrastive learning to align linguistic and visual features on the text-to-pixel level, enabling the capture of subtle differences between objects. The overall architecture is end-to-end without the need for extra parameter settings. To evaluate our approach, we manually annotate text descriptions for images in two fine-grained visual grounding datasets. The experimental results demonstrate that SAC-VGNet significantly improves performance in fine-grained visual grounding.

HSP-MFL: A High-level Semantic Property driven Multi-task Feature Learning Network for unsupervised person Re-ID

Unsupervised person re-identification aims to distinguish different pedestrians from discriminative representations on the basis of unlabeled data. Currently, most unsupervised Re-ID approaches explore visual representations to generate pseudo-labels for model’s training, which may suffer from background noise and semantic loss. To tackle this problem, this paper proposes a High-level Semantic Property driven Multi-task Feature Learning Network (HSP-MFL) to firstly introduce three high-level semantic properties for unsupervised person Re-ID. Technically, we design a novel Multiple Feature Fusion Module (MFFM) to deeply explore the complex correlation among multiple semantic and visual features to capture the discriminative feature cues, as well as a multi-task training scheme to generate robust fusion features. The architecture is quite simple and does not consume extra labeling costs. Extensive experiments on three datasets demonstrate that both high-level semantic properties and multi-task learning are effective in performance improvement, yielding SOTA mAPs for unsupervised person Re-ID.

Spatial learning overshadows learning novel odors and sounds in both predatory and frugivorous bats

Abstract To forage efficiently, animals should selectively attend to and remember the cues of food that best predict future meals. One hypothesis is that animals with different foraging strategies should vary in their reliance on spatial versus feature cues. Specifically, animals that store food in dispersed caches or that feed on spatially stable food, such as fruits or flowers, should be relatively biased towards learning a meal’s location, whereas predators that hunt mobile prey should instead be relatively biased towards learning feature cues such as odor or sound. Several authors have predicted that nectar-feeding and fruit-feeding bats would rely relatively more on spatial cues, whereas closely related predatory bats would rely more on feature cues, yet no experiment has compared these two foraging strategies under the same conditions. To test this hypothesis, we compared learning in the frugivorous bat, Artibeus jamaicensis, and the predatory bat, Lophostoma silvicolum, which hunts katydids using acoustic cues. We trained bats to find food paired with a unique and novel odor, sound, and location. To assess which cues each bat had learned, we then dissociated these cues to create conflicting information. Rather than finding that the frugivore and predator clearly differ in their relative reliance on spatial versus feature cues, we found that both species used spatial cues over sounds or odors in subsequent foraging decisions. We interpret these results alongside past findings on how foraging animals use spatial cues versus feature cues, and explore why spatial cues may be fundamentally more rich, salient, or memorable.

Space- and feature-based attention operate both independently and interactively within latent components of perceptual decision making.

Top-down visual attention filters undesired stimuli while selected information is afforded the lion's share of limited cognitive resources. Multiple selection mechanisms can be deployed simultaneously, but how unique influences of each combine to facilitate behavior remains unclear. Previously, we failed to observe an additive perceptual benefit when both space-based attention (SBA) and feature-based attention (FBA) were cued in a sparse display (Liang & Scolari, 2020): FBA was restricted to higher order decision-making processes when combined with a valid spatial cue, whereas SBA additionally facilitated target enhancement. Here, we introduced a series of design modifications across three experiments to elicit both attention mechanisms within signal enhancement while also investigating the impacts on decision making. First, we found that when highly reliable spatial and feature cues made unique contributions to search (experiment 1), or when each cue component was moderately reliable (experiments 2a and 2b), both mechanisms were deployed independently to resolve the target. However, the same manipulations produced interactive attention effects within other latent decision-making components that depended on the probability of the integrated cueing object. Time spent before evidence accumulation was reduced and responses were more conservative for the most likely pre-cue combination-even when it included an invalid component. These data indicate that selection mechanisms operate on sensory signals invariably in an independent manner, whereas a higher-order dependency occurs outside of signal enhancement.

A 3D Face Modeling and Recognition Method Based on Binocular Stereo Vision and Depth-Sensing Detection

The human face is an important channel for human interaction and is the most expressive part of the human body with personalized and diverse characteristics. To improve the modeling speed as well as recognition speed and accuracy of 3D faces, this paper proposes a laser scanning binocular stereo vision imaging method based on binocular stereo vision and depth-sensing detection method. Existing matching methods have a weak anti-interference capability, cannot identify the spacing of objects quickly and efficiently, and have too large errors. Using the technique mentioned in this paper can remedy these shortcomings by mainly using the laser lines scanned onto the object as strong feature cues for left and right views for binocular vision matching and thus for depth measurement perception of the object. The experiments in this paper first explore the effect of parameter variation of the laser scanning binocular vision imaging system on the accuracy of object measurement depth values in an indoor environment; the experimental data are selected from 68 face feature points for modeling, cameras that photograph faces, stereo calibration of the cameras using calibration methods to obtain the parameters of the binocular vision imaging system, and after stereo correction of the left and right camera images, the laser line scan light is used as a pixel matching strong features for binocular camera matching, which can achieve fast recognition and high accuracy and then select the system parameters with the highest accuracy to perform 3D reconstruction experiments on actual target objects in the indoor environment to achieve faster recognition.

Preparing for the unknown: How working memory provides a link between perception and anticipated action

What mechanisms underlie the transfer of a working memory representation into a higher-level code for guiding future actions? Electrophysiological correlates of attentional selection and motor preparation processes within working memory were investigated in two retrospective cuing tasks. In the first experiment, participants stored the orientation and location of a grating. Subsequent feature cues (selective vs. neutral) indicated which feature would be the target for later report. The oscillatory response in the mu and beta frequency range with an estimated source in the sensorimotor cortex contralateral to the responding hand was used as correlate of motor preparation. Mu/beta suppression was stronger following the selective feature cues compared to the neutral cue, demonstrating that purely feature-based selection is sufficient to form a prospective motor plan. In the second experiment, another retrospective cue was included to study whether knowledge of the task at hand is necessary to initiate motor preparation. Following the feature cue, participants were cued to either compare the stored feature(s) to a probe stimulus (recognition task) or to adjust the memory probe to match the target feature (continuous report task). An analogous suppression of mu oscillations was observed following a selective feature cue, even ahead of task specification. Further, a subsequent selective task cue again elicited a mu/beta suppression, which was stronger after a continuous report task cue. This indicates that working memory is able to flexibly store different types of information in higher-level mental codes to provide optimal prerequisites for all required action possibilities.

The medial pallium and the spatial encoding of geometric and visual cues in the terrestrial toad, Rhinella arenarum

The medial pallium (MP) of amphibians is the homologue of the mammalian hippocampus, and previous research has implicated MP for locating a using the boundary geometry of an environment. MP-lesioned, sham-operated and intact control terrestrial toads, Rhinella arenarum, were trained to locate a goal in a rectangular arena with a visual feature cue placed on one of the short walls. Whereas the sham-operated and intact subjects successfully learned to locate the goal, the MP-lesioned toads showed no evidence of learning. The data support the hypothesis that the amphibian MP is involved when the boundary geometry of an environment is used to locate a goal, which is consistent with evidence from other vertebrate groups. Curious, however, is that the MP lesions also resulted in the toads’ inability to locate the goal based on the visual feature cue. This result supports previous research and suggests that, in contrast to the hippocampal homologue of amniotes, the amphibian medial pallium plays a broader role in spatial learning processes.

Selective neural coding of object, feature, and geometry spatial cues in humans.

Orienting in space requires the processing of visual spatial cues. The dominant hypothesis about the brain structures mediating the coding of spatial cues stipulates the existence of a hippocampal-dependent system for the representation of geometry and a striatal-dependent system for the representation of landmarks. However, this dual-system hypothesis is based on paradigms that presented spatial cues conveying either conflicting or ambiguous spatial information and that used the term landmark to refer to both discrete three-dimensional objects and wall features. Here, we test the hypothesis of complex activation patterns in the hippocampus and the striatum during visual coding. We also postulate that object-based and feature-based navigation are not equivalent instances of landmark-based navigation. We examined how the neural networks associated with geometry-, object-, and feature-based spatial navigation compared with a control condition in a two-choice behavioral paradigm using fMRI. We showed that the hippocampus was involved in all three types of cue-based navigation, whereas the striatum was more strongly recruited in the presence of geometric cues than object or feature cues. We also found that unique, specific neural signatures were associated with each spatial cue. Object-based navigation elicited a widespread pattern of activity in temporal and occipital regions relative to feature-based navigation. These findings extend the current view of a dual, juxtaposed hippocampal-striatal system for visual spatial coding in humans. They also provide novel insights into the neural networks mediating object versus feature spatial coding, suggesting a need to distinguish these two types of landmarks in the context of human navigation.

A Novel Automatic Classification Method Based on the Hybrid Lightweight Shunt Network for Sintered Surfaces

Accurate control of sintering temperature is crucial for synthesizing high-quality iron ores. In practice, the temperature is usually regulated by experienced operators through adjusting igniters according to the class of the sintered surface. However, the manual method is difficult to meet the demand for real-time classification in a continuously running sintering process. In this article, to solve the above problem, we design a convolutional neural network with the hybrid lightweight shunt (HLWS-Net) for the online classification of sintered surfaces. Specifically, we introduce a lightweight shunt (LWS) module that guides the network to focus on feature cues at different scales with low computational complexity, avoiding the neglect of vital features in small regions. In addition, to reduce the information loss due to the lightweight structure, the Res-LWS module is proposed by combining the LWS module with the residual connection. Finally, the hybrid use of LWS and Res-LWS improves the classification performance of the network with little additional computational burden. To the best of our knowledge, this is the first method for sintering quality assessment from sintered surface images, which provides a new perspective to automate the sintering process. The experiments demonstrate that the proposed algorithm outperforms state-of-the-art deep learning classification methods with 92.1% classification accuracy, and the entire classification process takes only 0.038 seconds for a single image, which achieves a reasonable accuracy/speed trade-off.

Deep Residual Architecture Using Pixel and Feature Cues for View Synthesis and Temporal Interpolation

In this paper, we propose a deep residual architecture that can be used both for synthesizing high quality angular views in light fields and temporal frames in classical videos. The proposed framework consists of an optical flow estimator optimized for view synthesis, a trainable feature extractor and a residual convolutional network for pixel and feature-based view reconstruction. Among these modules, the fine-tuning of the optical flow estimator specifically for the view synthesis task yields scene depth or motion information that is well optimized for the targeted problem. In cooperation with the end-to-end trainable encoder, the synthesis block employs both pixel-based and feature-based synthesis with residual connection blocks, and the two synthesized views are fused with the help of a learned soft mask to obtain the final reconstructed view. Experimental results with various datasets show that our method performs favorably against other state-of-the-art (SOTA) methods with a large gain for light field view synthesis. Furthermore, with a little modification, our method can also be used for video frame interpolation, generating high quality frames compared with SOTA interpolation methods.

Separation of geometric and featural information in children’s spatial representation: Evidence from a model selection task

Previous studies in spatial reorientation have found that young children rely mainly on geometric shapes for reorientation and sometimes ignore features in the environment. Theoretical interpretations of children’s reorientation performance are usually attributed to children’s spatial representation of their surrounding environments. The geometric module theory states that featural information is represented separately from geometric shape in young children’s reorientation, whereas the adaptive combination model depicts an integral representation. Reorientation tasks, however, require the recognition of a specific location, and thus how the whole environment is represented remains unknown. The current study, using a model selection task, explored young children’s representation of the whole surrounding environment. A total of 75 children aged 3–5 years participated in the study. In each trial, children observed a large enclosure and were then asked to choose the corresponding model from two small models. The geometric shapes of the enclosure (rectangle vs. rhombus) and the types of distractors (shape distraction vs. feature position distraction) varied. Results showed that all three age groups performed above the chance level in the shape distraction conditions. Children had more difficulty with the feature position distraction conditions than with the shape distraction conditions. When the distractor shared the feature but at an inappropriate position, children’s performance was significantly poorer, especially in the rhombic enclosure. The results provide evidence that young children may represent featural cues separately from geometric shapes.