Scene Perception Research Articles

Perceived similarity as a window into representations of integrated sentence meaning.

When perceiving the world around us, we are constantly integrating pieces of information. The integrated experience consists of more than just the sum of its parts. For example, visual scenes are defined by a collection of objects as well as the spatial relations amongst them and sentence meaning is computed based on individual word semantic but also syntactic configuration. Having quantitative models of such integrated representations can help evaluate cognitive models of both language and scene perception. Here, we focus on language, and use a behavioral measure of perceived similarity as an approximation of integrated meaning representations. We collected similarity judgments of 200 subjects rating nouns or transitive sentences through an online multiple arrangement task. We find that perceived similarity between sentences is most strongly modulated by the semantic action category of the main verb. In addition, we show how non-negative matrix factorization of similarity judgment data can reveal multiple underlying dimensions reflecting both semantic as well as relational role information. Finally, we provide an example of how similarity judgments on sentence stimuli can serve as a point of comparison for artificial neural networks models (ANNs) by comparing our behavioral data against sentence similarity extracted from three state-of-the-art ANNs. Overall, our method combining the multiple arrangement task on sentence stimuli with matrix factorization can capture relational information emerging from integration of multiple words in a sentence even in the presence of strong focus on the verb.

Robot trajectory planning for autonomous 3D reconstruction of cockpit in aircraft final assembly testing

The trend towards automation and intelligence in aircraft final assembly testing has led to a new demand for autonomous perception of unknown cockpit operation scenes in robotic collaborative airborne system testing. To address this demand, a robotic automated 3D reconstruction cell which enables to autonomously plan the robot end-camera's trajectory is developed for image acquisition and 3D modeling of the cockpit operation scene. A continuous viewpoint path planning algorithm is proposed that incorporates both 3D reconstruction quality and robot path quality into optimization process. Smoothness metrics for viewpoint position paths and orientation paths are introduced together for the first time in 3D reconstruction. To ensure safe and effective movement, two spatial constraints, Domain of View Admissible Position (DVAP) and Domain of View Admissible Orientation (DVAO), are implemented to account for robot reachability and collision avoidance. By using diffeomorphism mapping, the orientation path is transformed into 3D, consistent with the position path. Both orientation and position paths can be optimized in a unified framework to maximize the gain of reconstruction quality and path smoothness within DVAP and DVAO. The reconstruction cell is capable of automatic data acquisition and fine scene modeling, using the generated robot C-space trajectory. Simulation and physical scene experiments have confirmed the effectiveness of the proposed method to achieve high-precision 3D reconstruction while optimizing robot motion quality.

Preferential signal pathways during the perception and imagery of familiar scenes: An effective connectivity study.

The perception and imagery of landmarks activate similar content-dependent brain areas, including occipital and temporo-medial brain regions. However, how these areas interact during visual perception and imagery of scenes, especially when recollecting their spatial location, remains unknown. Here, we combined functional magnetic resonance imaging (fMRI), resting-state functional connectivity (rs-fc), and effective connectivity to assess spontaneous fluctuations and task-induced modulation of signals among regions entailing scene-processing, the primary visual area and the hippocampus (HC), responsible for the retrieval of stored information. First, we functionally defined the scene-selective regions, that is, the occipital place area (OPA), the retrosplenial complex (RSC) and the parahippocampal place area (PPA), by using the face/scene localizer, observing that two portions of the PPA-anterior and posterior PPA-were consistently activated in all subjects. Second, the rs-fc analysis (n = 77) revealed a connectivity pathway similar to the one described in macaques, showing separate connectivity routes linking the anterior PPA with RSC and HC, and the posterior PPA with OPA. Third, we used dynamic causal modelling to evaluate whether the dynamic couplings among these regions differ between perception and imagery of familiar landmarks during a fMRI task (n = 16). We found a positive effect of HC on RSC during the retrieval of imagined places and an effect of occipital regions on both RSC and pPPA during the perception of scenes. Overall, we propose that under similar functional architecture at rest, different neural interactions take place between regions in the occipito-temporal higher-level visual cortex and the HC, subserving scene perception and imagery.

Outdoor mobility aid for people with visual impairment: Obstacle detection and responsive framework for the scene perception during the outdoor mobility of people with visual impairment

Outdoor mobility of individuals with visual impairment is challenging particularly where collision with obstacles can have significant impact on both physical and mental health. A variety of technological mobility aids for visually impaired people (VIP) have been studied and proposed in the literature which mainly utilise machine intelligence and deep learning (DL) approaches for object detection. However, object detection via the existing approaches suffers from reliability challenge due to real-time dynamics or the lack of available domain knowledge for specific obstacles identified by the VIP as potential hazards. In the present study, an object detection model (ObDtM) based on deep transfer learning techniques was developed for a custom-built dataset comprising of specific obstacles identified by the VIP as potential hazards. A custom dataset was compiled and manually annotated from various publicly available sources to train the ObDtM. Experiments were conducted to evaluate the proposed ObDtM for unseen obstacles kept as the test set. Results showed that ObDtM outperformed the state-of-the-art with 97% mean Average Precision (mAP), indicating a robust and generalizable DL approach. The compiled dataset and the ObDtM is useful for several potential use cases, particularly highlighting the use of DL in IoT and smart city applications. Additionally, a smart synergetic outdoor mobility framework was proposed for VIP (SOMAVIP) allowing comprehensive and accurate semantic representation of the surroundings by utilising the proposed ObDtM, cloud services, internet of things (IoT), and digital environment in the context of emerging smart city infrastructure. The proposed SOMAVIP can be highly impactful for improving VIPs’ quality of life mainly for safer, cost-effective, and reliable independent outdoor mobility enriched with real-time perception and interpretations of the surroundings.

GeoSynth: A Photorealistic Synthetic Indoor Dataset for Scene Understanding.

Deep learning has revolutionized many scene perception tasks over the past decade. Some of these improvements can be attributed to the development of large labeled datasets. The creation of such datasets can be an expensive, time-consuming, and imperfect process. To address these issues, we introduce GeoSynth, a diverse photorealistic synthetic dataset for indoor scene understanding tasks. Each GeoSynth exemplar contains rich labels including segmentation, geometry, camera parameters, surface material, lighting, and more. We demonstrate that supplementing real training data with GeoSynth can significantly improve network performance on perception tasks, like semantic segmentation. A subset of our dataset will be made publicly available at https://github.com/geomagical/GeoSynth.

Hybrid deep learning with improved Salp swarm optimization based multi-class grape disease classification model

The recent revolutions in Computer Vision (CV) and Artificial Intelligence (AI) techniques have found many applications in grapevine and smart agriculture processes. Recently, Deep Learning (DL) techniques like Convolutional Neural Networks (CNN) have been broadly applied in smart agriculture, leaf disease recognition, and scene perception. In this background, the current study develops a Hybrid Deep Learning with Improved Salp Swarm Optimization-based Multi-class Grape Disease Classification (HDLISSA-MGDC) model. The proposed HDLISSA-MGDC model focuses on the classification of grape leaf images into four distinct classes such as black measles, black rot, Isariopsis leaf spot and healthy. Initially, the Median Filtering (MF) technique is applied for image pre-processing, which eliminates the noise present in the images. In addition, the HDLISSA-MGDC model designs a feature extractor with the help of Dilated Residual Network (DRN) and Adam optimizer. For grape disease classification, the Convolutional Neural Network-Gated Recurrent Unit (CNN-GRU) model is employed in this study. Finally, the ISSA is exploited to adjust the hyperparameter values of the CNN-GRU method. The proposed HDLISSA-MGDC method was simulated using the plant leaf disease dataset. The simulation results show the significant performance of the proposed HDLISSA-MGDC model.

Examining whether adults with autism spectrum disorder encounter multiple problems in theory of mind: a study based on meta-analysis.

Theory of mind (ToM) represents a complex ability, while persons with autism spectrum disorder (ASD) encounter difficulties in the processing of ToM. The present ToM-focused studies on adults with ASD report inconsistent results, possibly owing to the differences between tasks. For instance, different ToM-related tasks involve different cognitive abilities, but the development of these cognitive abilities is different among adults with ASD, thereby leading to different behaviors by the same individual with ASD in different tasks. Therefore, it is of vital significance to explore the potential reasons for inconsistencies in the existing studies based on the task classification perspective. Hence, this study primarily reviews the existing ToM tasks used in studies on adults with ASD; afterward, based on the forms and characteristics of the task, the current ToM tasks are classified into four categories-reading comprehension, perceptual scene comprehension, comprehensive scene comprehension , and self-other processing. Subsequently, a meta-analysis is undertaken to determine the difference in each ToM task category between the ASD group and the typically developing (TD) group. As a result, 110 research papers (including 3,205 adults with ASD and 3,675 TD adults) that fulfilled the stated criteria are examined in this study. The study findings suggest that adults with ASD demonstrate worse performance in terms of all four ToM task categories as compared to TD adults. Furthermore, compared with tasks of self-other processing and perceptual scene comprehension, adults with ASD perform worse in reading comprehension and comprehensive scene comprehension. This shows that the differences between tasks may exert a potential influence on the study results. Future studies should focus on different abilities involved in ToM processing and the choice of ToM tasks, in order to elucidate the critical problems of ToM in adults with ASD.

Time Courses of Attended and Ignored Object Representations.

Selective attention prioritizes information that is relevant to behavioral goals. Previous studies have shown that attended visual information is processed and represented more efficiently, but distracting visual information is not fully suppressed, and may also continue to be represented in the brain. In natural vision, to-be-attended and to-be-ignored objects may be present simultaneously in the scene. Understanding precisely how each is represented in the visual system, and how these neural representations evolve over time, remains a key goal in cognitive neuroscience. In this study, we recorded EEG while participants performed a cued object-based attention task that involved attending to target objects and ignoring simultaneously presented and spatially overlapping distractor objects. We performed support vector machine classification on the stimulus-evoked EEG data to separately track the temporal dynamics of target and distractor representations. We found that (1) both target and distractor objects were decodable during the early phase of object processing (∼100 msec to ∼200 msec after target onset), and (2) the representations of both objects were sustained over time, remaining decodable above chance until ∼1000-msec latency. However, (3) the distractor object information faded significantly beginning after about 300-msec latency. These findings provide information about the fate of attended and ignored visual information in complex scene perception.

A dynamical scan-path model for task-dependence during scene viewing.

In real-world scene perception, human observers generate sequences of fixations to move image patches into the high-acuity center of the visual field. Models of visual attention developed over the last 25 years aim to predict two-dimensional probabilities of gaze positions for a given image via saliency maps. Recently, progress has been made on models for the generation of scan paths under the constraints of saliency as well as attentional and oculomotor restrictions. Experimental research demonstrated that task constraints can have a strong impact on viewing behavior. Here, we propose a scan-path model for both fixation positions and fixation durations, which include influences of task instructions and interindividual differences. Based on an eye-movement experiment with four different task conditions, we estimated model parameters for each individual observer and task condition using a fully Bayesian dynamical modeling framework using a joint spatial-temporal likelihood approach with sequential estimation. Resulting parameter values demonstrate that model properties such as the attentional span are adjusted to task requirements. Posterior predictive checks indicate that our dynamical model can reproduce task differences in scan-path statistics across individual observers. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Pedestrian reported activity and information preference while waiting at a red light

AbstractThe development of pedestrian signal lights made it a promising information center to distribute information to assist safe road crossing and provide “controlled engagement” to improve waiting experience. To determine what types of information to display, this study surveyed 555 pedestrians on frequency of their existing activities while waiting at a red light and preference of several types of information based on results of a structured interview. Pedestrian reported 14 types of activities, from which the four categories with decreasing frequency were extracted based on factor analysis: scene perception, recreation, mental activity, and anxiety activity. The participants reported more mental and anxiety activity have higher need for external displayed information on signal lights. Overall, among the four information categories extracted from 24 specific information types with factor analysis, pedestrians preferred road crossing and contextual information the most, with less preference on learning and appreciation information. The least preferred information is entertainment information, within which they even disliked entertainment news and general commercial ads ( vs. local ones). There are also individual differences in preference for these information categories. The findings have implications on information design for the roadside signal lights as well as advanced traveller information system in general.

Knowledge and topology: A two layer spatially dependent graph neural networks to identify urban functions with time-series street view image

With the rise of GeoAI research, streetscape imagery has received extensive attention due to its comprehensive coverage, abundant information, and accessibility. However, obtaining a holistic spatial–temporal scene representation is difficult because places are often composed of multiple images from different angles, times and locations. This problem also exists in other types of geo-tagged imagery. To solve it, we propose a purely visual, robust, and reliable method for urban function identification at the street scale. We introduce a method based on a two-layer spatially dependent graph neural network structure, which handles sequential street view imagery as input (typically available in services such as Google Street View, Baidu Maps, and Mapillary), with full consideration of the spatial dependencies among road networks. In this paper, we construct an urban topological map network using OpenStreetMap data in Wuhan, China, and compute a semantic representation of the scene as a whole at the street scale using a large-scale pre-trained model. We construct the graph network with streets as nodes based on 28,693 mapping relationships constructed from 75,628 street view images and 5,458 streets. Only 5.3% of the node labels were required to obtain 10 categories of functions for all nodes in the study area. The results demonstrate that by using appropriate spatial weights, street encoder, and graph structure, our novel method achieves high accuracy of P@1 46.2%, P@3 73.0%, P@5 82.4%, and P@10 89.9%, fully demonstrating the effectiveness of the introduced approach. We also use the model to sense urban spatial–temporal renewal by computing time series street images. The model is also applicable to the prediction of other attributes, where only a small number of labels are required to obtain valid and reliable scene perception results. The example data and code is shared at: https://github.com/yemanzhongting/Knowledge-and-Topology.

Bi-RRNet: Bi-level recurrent refinement network for camouflaged object detection

In this paper, we present a lightweight Bi-level Recurrent Refinement Network (Bi-RRNet) for Camouflaged Object Detection (COD) that consists of a Lower-level RRNet (L-RRN) and an Up-level RRNet (U-RRN) to progressively refine the multi-level context features for precise dense prediction. In particular, the L-RRN recursively refines the deeper layer high-level semantic features with the high-resolution low-level features from the earlier layers in a top-down manner, and the U-RRN progressively polishes the refined features from the L-RRN in a recurrent manner, producing the high-resolution semantic features that are essential to accurate COD. Moreover, we develop a Multi-scale Scene Perception Module (MSPM) that, in order to deal with target appearance variation, first compresses the global scene context information at each layer into a learnable weight vector and then modulates the multi-scale context features produced by a filter bank with various local receptive fields using the learned weights. Meanwhile, we design a Region-Consistency Enhancement Module (RCEM) that makes use of high-level semantic features to direct filtering out the cluttered information in the lower-layer features. This module can highlight the regions of camouflaged objects, maximizing the inter-class contrast between the objects and their surroundings. Extensive experiments on four challenging benchmark datasets, including CHAMELEON, CAMO, COD10K, and NC4K, show that our Bi-RRNet outperforms a variety of state-of-the-art methods in terms of accuracy and model parameters. Our Bi-RRNet, in particular, is lightweight, with 14.95M parameters that are only half the size of the state-of-the-art BSA-Net.

Perception of global properties, objects, and settings in natural auditory scenes

Theories of auditory scene analysis suggest our perception of scenes relies on identifying and segregating objects within it. However, a more global process may occur while analyzing scenes, which has been evidenced in the visual domain. In our first experiment, we studied perception of eight global properties (e.g., openness), using a collection of 200 high-quality auditory scenes. Participants showed high agreement on their ratings of global properties. The global properties were explained by a two-factor model. Acoustic features of scenes were explained by a seven-factor model, and linearly predicted the global ratings by different amounts (R-squared = 0.33–0.87), although we also observed nonlinear relationships between acoustical and global variables. A multi-layer neural network trained to recognize auditory objects in everyday soundscapes from YouTube shows high-level embeddings of our 200 scenes are correlated with some global variables at earlier stages of processing than others. In a second experiment, we evaluated participants’ accuracy in identifying the setting of and objects within scenes across three durations (1, 2, and 4 s). Overall, participants performed better on the object identification task, but needed longer duration stimuli to do so. These results suggest object identification may require more processing time and/or attention switching than setting identification.

Research on Scene Perception of Mobile Robot based on SLAM

Research on Scene Perception of Mobile Robot based on SLAM

Object Sub-Categorization and Common Framework<br /> Method using Iterative AdaBoost for Rapid Detection of Multiple Objects

Object detection and tracking in real time has numerous applications and benefits in various fields like survey, crime detection etc. The idea of gaining useful information from real time scenes on the roads is called as Traffic Scene Perception (TSP). TSP actually consists of three subtasks namely, detecting things of interest, recognizing the discovered objects and tracking of the moving objects. Normally the results obtained could be of value in object recognition and tracking, however the detection of a particular object of interest is of higher value in any real time scenario. The prevalent systems focus on developing unique detectors for each of the above-mentioned subtasks and they work upon utilizing different features. This obviously is time consuming and involves multiple redundant operations. Hence in this paper a common framework using the enhanced AdaBoost algorithm is proposed which will examine all dense characteristics only once thereby increasing the detection speed substantially. An object sub-categorization strategy is proposed to capture the intra-class variance of objects in order to boost generalisation performance even more. We use three detection applications to demonstrate the efficiency of the proposed framework: traffic sign detection, car detection, and bike detection. On numerous benchmark data sets, the proposed framework delivers competitive performance using state-of-the-art techniques.

Drivers or Pedestrians, Whose Dynamic Perceptions Are More Effective to Explain Street Vitality? A Case Study in Guangzhou

As an important indicator of urban development capacity, vitality can be affected by the human perception of street views, which is a dynamic sensory process that can differ greatly according to different transportation modes, due to their different travel speeds, distances, and routes. However, few studies have evaluated how the dynamic spatial perceptions differ between different travel modes and how these differences can affect vitality differently, due to the limitation of city-scale quantitative data on the dynamic perception of urban scenes. To fill the gap, we propose a “dynamic through-movement perception” (DTMP) measure which integrates a streetscape quality evaluation model with a network-based movement potential model. We measure the streetscape qualities from Baidu street-view images (SVI) and compare the spatial perceptions of drivers and pedestrians in central Guangzhou, China. First, more than twenty visual elements were classified from SVIs to predict human perceptions collected from visual surveys. Second, the through-movement probability of driving and walking were calculated based on classic natural movement theory in space syntax and measured as the angular betweenness for the two travel modes. Third, we accumulate the multipliers of visual perception and through-movement probability of driving and walking as the DTMP for both modes. Lastly, the DTMPs of both modes were fitted into linear regression models to explain street vitality, which is measured using Baidu mobile phone check-in data, when other control variables such as functional density, functional diversity and amenity clustering reachability are accounted for. The results show that the dynamic perception of driving overall shows a stronger correlation with street vitality, while perceived richness is significantly positive in both travel modes. This study provides the first quantitative evidence to reveal how the movement probability of different travel modes can significantly influence people’s sense of place, while in turn increasing street vitality. Our results can explain how different types of street commerce (i.e., pedestrian-oriented, and auto-oriented) aggregate spontaneously due to the dynamic movement potential, which provides an important reference for urban planners and decision makers for improving street vitality when making urban revitalization policies.

Mechanisms of speed encoding in the human middle temporal cortex measured by 7T fMRI.

Perception of dynamic scenes in our environment results from the evaluation of visual features such as the fundamental spatial and temporal frequency components of a moving object. The ratio between these two components represents the object's speed of motion. The human middle temporal cortex hMT+ has a crucial biological role in the direct encoding of object speed. However, the link between hMT+ speed encoding and the spatiotemporal frequency components of a moving object is still under explored. Here, we recorded high resolution 7T blood oxygen level-dependent BOLD responses to different visual motion stimuli as a function of their fundamental spatial and temporal frequency components. We fitted each hMT+ BOLD response with a 2D Gaussian model allowing for two different speed encoding mechanisms: (1) distinct and independent selectivity for the spatial and temporal frequencies of the visual motion stimuli; (2) pure tuning for the speed of motion. We show that both mechanisms occur but in different neuronal groups within hMT+, with the largest subregion of the complex showing separable tuning for the spatial and temporal frequency of the visual stimuli. Both mechanisms were highly reproducible within participants, reconciling single cell recordings from MT in animals that have showed both encoding mechanisms. Our findings confirm that a more complex process is involved in the perception of speed than initially thought and suggest that hMT+ plays a primary role in the evaluation of the spatial features of the moving visual input.

Interactionally Embedded Gestalt Principles of Multimodal Human Communication

Natural human interaction requires us to produce and process many different signals, including speech, hand and head gestures, and facial expressions. These communicative signals, which occur in a variety of temporal relations with each other (e.g., parallel or temporally misaligned), must be rapidly processed as a coherent message by the receiver. In this contribution, we introduce the notion of interactionally embedded, affordance-driven gestalt perception as a framework that can explain how this rapid processing of multimodal signals is achieved as efficiently as it is. We discuss empirical evidence showing how basic principles of gestalt perception can explain some aspects of unimodal phenomena such as verbal language processing and visual scene perception but require additional features to explain multimodal human communication. We propose a framework in which high-level gestalt predictions are continuously updated by incoming sensory input, such as unfolding speech and visual signals. We outline the constituent processes that shape high-level gestalt perception and their role in perceiving relevance and prägnanz. Finally, we provide testable predictions that arise from this multimodal interactionally embedded gestalt-perception framework. This review and framework therefore provide a theoretically motivated account of how we may understand the highly complex, multimodal behaviors inherent in natural social interaction.

Predictive Processing of Scene Layout Depends on Naturalistic Depth of Field

Boundary extension is a classic memory illusion in which observers remember more of a scene than was presented. According to predictive-processing accounts, boundary extension reflects the integration of visual input and expectations of what is beyond a scene’s boundaries. According to normalization accounts, boundary extension rather reflects one end of a normalization process toward a scene’s typically experienced viewing distance, such that close-up views give boundary extension but distant views give boundary contraction. Here, across four experiments (N = 125 adults), we found that boundary extension strongly depends on depth of field, as determined by the aperture settings on a camera. Photographs with naturalistic depth of field led to larger boundary extension than photographs with unnaturalistic depth of field, even when distant views were shown. We propose that boundary extension reflects a predictive mechanism with adaptive value that is strongest for naturalistic views of scenes. The current findings indicate that depth of field is an important variable to consider in the study of scene perception and memory.

Awareness survey on drug crime scene investigation and drug detection kits among drug-related police officers

This questionnaire-based study aimed to investigate the drug crime scene experienced by drug-related police officers and the perceptions of drug test kits by them before initiating the development of drug test kits to detect 16 types of drugs. The subjects were 57 drug-related police officers. Most of the respondents (96.5%) had <10 years of experience in drug-related work. Respondents were questioned about the drug scene investigation and perceptions of drug test kits. The questionnaire about drug test kits included the question on ‘simple/rapid drug test kit’ and ‘electronic portable drug analyzer’ regarding the disadvantages of existing kits and expecting features when a new kit is developed. First, in the on-site survey, the drug-related crime occurred at the suspect's house (47.8%), and methamphetamine (35.0%) and γ-hydroxybutyric acid (19.5%) were mainly found. In the awareness survey on drug test kits, most respondents (67.2%) had an experience of using ‘simple/rapid drug test kits’, whereas 17.5% for the ‘electronic portable drug analyzer’. In the case of ‘simple/rapid drug test kit’, the false-positive rate reached 53.8% by a misinterpretation due to ambiguous color change (47.6%). The inaccuracy of the result (33.3%) was the most concern in ‘electronic portable drug analyzer’. Respondents most favored pipette type for sample collector when a new kit is developed. In addition, they preferred the smaller kit with short detection times in both kit types. This survey could be applied to the development of efficient and practical kits for police officers working in drug-related fields.