Objects In Scene Research Articles

Remote intelligent perception system for multi-object detection.

During the last few years, a heightened interest has been shown in classifying scene images depicting diverse robotic environments. The surge in interest can be attributed to significant improvements in visual sensor technology, which has enhanced image analysis capabilities. Advances in vision technology have a major impact on the areas of multiple object detection and scene understanding. These tasks are an integral part of a variety of technologies, including integrating scenes in augmented reality, facilitating robot navigation, enabling autonomous driving systems, and improving applications in tourist information. Despite significant strides in visual interpretation, numerous challenges persist, encompassing semantic understanding, occlusion, orientation, insufficient availability of labeled data, uneven illumination including shadows and lighting, variation in direction, and object size and changing background. To overcome these challenges, we proposed an innovative scene recognition framework, which proved to be highly effective and yielded remarkable results. First, we perform preprocessing using kernel convolution on scene data. Second, we perform semantic segmentation using UNet segmentation. Then, we extract features from these segmented data using discrete wavelet transform (DWT), Sobel and Laplacian, and textual (local binary pattern analysis). To recognize the object, we have used deep belief network and then find the object-to-object relation. Finally, AlexNet is used to assign the relevant labels to the scene based on recognized objects in the image. The performance of the proposed system was validated using three standard datasets: PASCALVOC-12, Cityscapes, and Caltech 101. The accuracy attained on the PASCALVOC-12 dataset exceeds 96% while achieving a rate of 95.90% on the Cityscapes dataset. Furthermore, the model demonstrates a commendable accuracy of 92.2% on the Caltech 101 dataset. This model showcases noteworthy advancements beyond the capabilities of current models.

Counting dense object of multiple types based on feature enhancement.

Accurately counting the number of dense objects in an image, such as pedestrians or vehicles, is a challenging and practical task. The existing density map regression methods based on CNN are mainly used to count a class of dense objects in a single scene. However, in complex traffic scenes, objects such as vehicles and pedestrians usually exist at the same time, and multiple classes of dense objects need to be counted simultaneously. To solve the above issues, we propose a new multiple types of dense object counting method based on feature enhancement, which can enhance the features of dense counting objects in complex traffic scenes to realize the classification and regression counting of dense vehicles and people. The counting model consists of the regression subnet and the classification subnet. The regression subnet is primarily used to generate two-channel predicted density maps, mainly including the initial feature layer and the feature enhancement layer, in which the feature enhancement layer can enhance the classification features and regression counting features of dense objects in complex traffic scenes. The classification subnet mainly supervises classifying dense vehicles and people into two feature channels to assist the regression counting task of the regression subnets. Our method is compared on VisDrone+ datasets, ApolloScape+ datasets, and UAVDT+ datasets. The experimental results show that the method counts two kinds of dense objects simultaneously and outputs a high-quality two-channel predicted density map. The counting performance is better than the state-of-the-art counting network in dense people and vehicle counting. In future work, we will further improve the feature extraction ability of the model in complex traffic scenes to classify and count a variety of dense objects such as cars, pedestrians, and non-motor vehicles.

FocusDet: an efficient object detector for small object

The object scale of a small object scene changes greatly, and the object is easily disturbed by a complex background. Generic object detectors do not perform well on small object detection tasks. In this paper, we focus on small object detection based on FocusDet. FocusDet refers to the small object detector proposed in this paper. It consists of three parts: backbone, feature fusion structure, and detection head. STCF-EANet was used as the backbone for feature extraction, the Bottom Focus-PAN for feature fusion, and the detection head for object localization and recognition.To maintain sufficient global context information and extract multi-scale features, the STCF-EANet network backbone is used as the feature extraction network.PAN is a feature fusion module used in general object detectors. It is used to perform feature fusion on the extracted feature maps to supplement feature information.In the feature fusion network, FocusDet uses Bottom Focus-PAN to capture a wider range of locations and lower-level feature information of small objects.SIOU-SoftNMS is the proposed algorithm for removing redundant prediction boxes in the post-processing stage. SIOU multi-dimension accurately locates the prediction box, and SoftNMS uses the Gaussian algorithm to remove redundant prediction boxes. FocusDet uses SIOU-SoftNMS to address the missed detection problem common in dense tiny objects.The VisDrone2021-DET and CCTSDB2021 object detection datasets are used as benchmarks, and tests are carried out on VisDrone2021-det-test-dev and CCTSDB-val datasets. Experimental results show that FocusDet improves mAP@.5% from 33.6% to 46.7% on the VisDrone dataset. mAP@.5% on the CCTSDB2021 dataset is improved from 81.6% to 87.8%. It is shown that the model has good performance for small object detection, and the research is innovative.

A Multi-Feature Fusion Method for Urban Functional Regions Identification: A Case Study of Xi’an, China

Research on the identification of urban functional regions is of great significance for the understanding of urban structure, spatial planning, resource allocation, and promoting sustainable urban development. However, achieving high-precision urban functional region recognition has always been a research challenge in this field. For this purpose, this paper proposes an urban functional region identification method called ASOE (activity–scene–object–economy), which integrates the features from multi-source data to perceive the spatial differentiation of urban human and geographic elements. First, we utilize VGG16 (Visual Geometry Group 16) to extract high-level semantic features from the remote sensing images with 1.2 m spatial resolution. Then, using scraped building footprints, we extract building object features such as area, perimeter, and structural ratios. Socioeconomic features and population activity features are extracted from Point of Interest (POI) and Weibo data, respectively. Finally, integrating the aforementioned features and using the Random Forest method for classification, the identification results of urban functional regions in the main urban area of Xi’an are obtained. After comparing with the actual land use map, our method achieves an identification accuracy of 91.74%, which is higher than other comparative methods, making it effectively identify four typical urban functional regions in the main urban area of Xi’an (e.g., residential regions, industrial regions, commercial regions, and public regions). The research indicates that the method of fusing multi-source data can fully leverage the advantages of big data, achieving high-precision identification of urban functional regions.

Oculomotor routines for perceptual judgments.

In everyday life we frequently make simple visual judgments about object properties, for example, how big or wide is a certain object? Our goal is to test whether there are also task-specific oculomotor routines that support perceptual judgments, similar to the well-established exploratory routines for haptic perception. In a first study, observers saw different scenes with two objects presented in a photorealistic virtual reality environment. Observers were asked to judge which of two objects was taller or wider while gaze was tracked. All tasks were performed with the same set of virtual objects in the same scenes, so that we can compare spatial characteristics of exploratory gaze behavior to quantify oculomotor routines for each task. Width judgments showed fixations around the center of the objects with larger horizontal spread. In contrast, for height judgments, gaze was shifted toward the top of the objects with larger vertical spread. These results suggest specific strategies in gaze behavior that presumably are used for perceptual judgments. To test the causal link between oculomotor behavior and perception, in a second study, observers could freely gaze at the object or we introduced a gaze-contingent setup forcing observers to fixate specific positions on the object. Discrimination performance was similar between free-gaze and the gaze-contingent conditions for width and height judgments. These results suggest that although gaze is adapted for different tasks, performance seems to be based on a perceptual strategy, independent of potential cues that can be provided by the oculomotor system.

Category-based depth incorporation for salient object ranking

Salient object ranking is a visual task that aims to mimic the human visual system’s ability to prioritize various salient objects in complex scenes. It integrates various computer vision tasks, including object detection and instance segmentation, and draws insights from psychology and biology. Despite recent advancements in salient object ranking research, there is still a need for a more in-depth exploration of certain underlying factors that influence the ranking. Previous studies have primarily focused on factors such as the scale and position of objects, as well as interactions between objects and their context. However, they often overlooked the crucial interaction between objects and observers. Unlike other visual tasks, salient object ranking is highly subjective and influenced by the observer. Therefore, establishing a meaningful connection between the observed object and the observer becomes crucial. Our model addresses this gap by placing emphasis on the distance between objects and observers. It investigates the implicit relationship between object categories and distance through the Category-Aware Attention (CAA) module. This innovative approach incorporates depth into salient object ranking, resulting in an improvement in ranking performance.

Insights into Image Understanding: Segmentation Methods for Object Recognition and Scene Classification

Image understanding plays a pivotal role in various computer vision tasks, such as extraction of essential features from images, object detection, and segmentation. At a higher level of granularity, both semantic and instance segmentation are necessary for fully grasping a scene. In recent times, the concept of panoptic segmentation has emerged as a field of study that unifies semantic and instance segmentation. This article sheds light on the pivotal role of panoptic segmentation as a visualization tool for understanding scene components, including object detection, categorization, and precise localization of scene elements. Advancements in achieving panoptic segmentation and suggested improvements to the predicted outputs through a top-down approach are discussed. Furthermore, datasets relevant to both scene recognition and panoptic segmentation are explored to facilitate a comparative analysis. Finally, the article outlines certain promising directions in image recognition and analysis by underlining the ongoing evolution in image understanding methodologies.

Exploring the Semantic-Inconsistency Effect in Scenes Using a Continuous Measure of Linguistic-Semantic Similarity

Viewers use contextual information to visually explore complex scenes. Object recognition is facilitated by exploiting object–scene relations (which objects are expected in a given scene) and object–object relations (which objects are expected because of the occurrence of other objects). Semantically inconsistent objects deviate from these expectations, so they tend to capture viewers’ attention (the semantic-inconsistency effect). Some objects fit the identity of a scene more or less than others, yet semantic inconsistencies have hitherto been operationalized as binary (consistent vs. inconsistent). In an eye-tracking experiment (N = 21 adults), we study the semantic-inconsistency effect in a continuous manner by using the linguistic-semantic similarity of an object to the scene category and to other objects in the scene. We found that both highly consistent and highly inconsistent objects are viewed more than other objects (U-shaped relationship), revealing that the (in)consistency effect is more than a simple binary classification.

Limited information-processing capacity in vision explains number psychophysics.

Humans and other animals are able to perceive and represent a number of objects present in a scene, a core cognitive ability thought to underlie the development of mathematics. However, the perceptual mechanisms that underpin this capacity remain poorly understood. Here, we show that our visual sense of number derives from a visual system designed to efficiently encode the location of objects in scenes. Using a mathematical model, we demonstrate that an efficient but information-limited encoding of objects' locations can explain many key aspects of number psychophysics, including subitizing, Weber's law, underestimation, and effects of exposure time. In two experiments (N = 100 each), we find that this model of visual encoding captures human performance in both a change-localization task and a number estimation task. In a third experiment (N = 100), we find that individual differences in change-localization performance are highly predictive of differences in number estimation, both in terms of overall performance and inferred model parameters, with participants having numerically indistinguishable inferred information capacities across tasks. Our results therefore indicate that key psychophysical features of numerical cognition do not arise from separate modules or capacities specific to number, but rather as by-products of lower level constraints on perception. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

On investigating drivers’ attention allocation during partially-automated driving

The use of partially-automated systems require drivers to supervise the system functioning and resume manual control whenever necessary. Yet literature on vehicle automation show that drivers may spend more time looking away from the road when the partially-automated system is operational. In this study we answer the question of whether this pattern is a manifestation of inattentional blindness or, more dangerously, it is also accompanied by a greater attentional processing of the driving scene. Participants drove a simulated vehicle in manual or partially-automated mode. Fixations were recorded by means of a head-mounted eye-tracker. A surprise two-alternative forced-choice recognition task was administered at the end of the data collection whereby participants were quizzed on the presence of roadside billboards that they encountered during the two drives. Data showed that participants were more likely to fixate and recognize billboards when the automated system was operational. Furthermore, whereas fixations toward billboards decreased toward the end of the automated drive, the performance in the recognition task did not suffer. Based on these findings, we hypothesize that the use of the partially-automated driving system may result in an increase in attention allocation toward peripheral objects in the road scene which is detrimental to the drivers’ ability to supervise the automated system and resume manual control of the vehicle.

A Study of YOLO-Based Object Detection for Visually Impaired Individuals

Abstract: An essential area of investigation involves recognizing and tracking objects, given the frequent changes in object motion, scene dimensions, occlusions, appearance, ego-motion, and lighting variations. Effective object tracking heavily relies on feature selection due to its critical role. This process is integral to numerous real-time applications like vehicle detection and video surveillance. To address detection challenges, tracking methods need to be adept at handling object movement and appearance changes. Among these methods, tracking algorithms play a pivotal role in smoothing video sequences and receive significant attention. However, only a few approaches leverage previously gathered data on object attributes such as shape and color. This study delves into a tracking algorithm that encompasses all these object properties. Its aim is to investigate and assess past methodologies for tracking and detecting moving objects across various stages of video sequences. Additionally, it aims to identify existing challenges and propose innovative strategies to enhance object tracking throughout video frames

Saccades to partially occluded objects: Perceptual completion mediates oculomotor control.

Oculomotor behavior typically consists of directing gaze to objects in complex scenes for the purpose of extracting detailed perceptual information. Here, we probed the nature of the visual representations over which saccades to objects are computed. We contrasted an image-based oculomotor control hypothesis, holding that saccades are computed solely over information explicit in the retinal image, and an object-based oculomotor control hypothesis, holding that saccades are computed over object representations reflecting the three-dimensional structure of the scene. We recorded saccade landing positions to partially occluded objects in a naturalistic search task. In Experiment 1, saccade landing positions were biased toward the center of the perceptually completed object. Experiment 2 demonstrated that the bias held even when it would have been strategically advantageous to avoid it. Experiment 3 demonstrated that the bias was not due to image-level differences generated by the presence of occluders. The results indicate that saccade motor programs are computed, at least in part, over object-level representations reflecting the completion of occluded surfaces.

IDet3D: Towards Efficient Interactive Object Detection for LiDAR Point Clouds

Accurately annotating multiple 3D objects in LiDAR scenes is laborious and challenging. While a few previous studies have attempted to leverage semi-automatic methods for cost-effective bounding box annotation, such methods have limitations in efficiently handling numerous multi-class objects. To effectively accelerate 3D annotation pipelines, we propose iDet3D, an efficient interactive 3D object detector. Supporting a user-friendly 2D interface, which can ease the cognitive burden of exploring 3D space to provide click interactions, iDet3D enables users to annotate the entire objects in each scene with minimal interactions. Taking the sparse nature of 3D point clouds into account, we design a negative click simulation (NCS) to improve accuracy by reducing false-positive predictions. In addition, iDet3D incorporates two click propagation techniques to take full advantage of user interactions: (1) dense click guidance (DCG) for keeping user-provided information throughout the network and (2) spatial click propagation (SCP) for detecting other instances of the same class based on the user-specified objects. Through our extensive experiments, we present that our method can construct precise annotations in a few clicks, which shows the practicality as an efficient annotation tool for 3D object detection.

SeqRank: Sequential Ranking of Salient Objects

Salient Object Ranking (SOR) is the process of predicting the order of an observer's attention to objects when viewing a complex scene. Existing SOR methods primarily focus on ranking various scene objects simultaneously by exploring their spatial and semantic properties. However, their solutions of simultaneously ranking all salient objects do not align with human viewing behavior, and may result in incorrect attention shift predictions. We observe that humans view a scene through a sequential and continuous process involving a cycle of foveating to objects of interest with our foveal vision while using peripheral vision to prepare for the next fixation location. For instance, when we see a flying kite, our foveal vision captures the kite itself, while our peripheral vision can help us locate the person controlling it such that we can smoothly divert our attention to it next. By repeatedly carrying out this cycle, we can gain a thorough understanding of the entire scene. Based on this observation, we propose to model the dynamic interplay between foveal and peripheral vision to predict human attention shifts sequentially. To this end, we propose a novel SOR model, SeqRank, which reproduces foveal vision to extract high-acuity visual features for accurate salient instance segmentation while also modeling peripheral vision to select the object that is likely to grab the viewer’s attention next. By incorporating both types of vision, our model can mimic human viewing behavior better and provide a more faithful ranking among various scene objects. Most notably, our model improves the SA-SOR/MAE scores by +6.1%/-13.0% on IRSR, compared with the state-of-the-art. Extensive experiments show the superior performance of our model on the SOR benchmarks. Code is available at https://github.com/guanhuankang/SeqRank.

Exploiting Polarized Material Cues for Robust Car Detection

Car detection is an important task that serves as a crucial prerequisite for many automated driving functions. The large variations in lighting/weather conditions and vehicle densities of the scenes pose significant challenges to existing car detection algorithms to meet the highly accurate perception demand for safety, due to the unstable/limited color information, which impedes the extraction of meaningful/discriminative features of cars. In this work, we present a novel learning-based car detection method that leverages trichromatic linear polarization as an additional cue to disambiguate such challenging cases. A key observation is that polarization, characteristic of the light wave, can robustly describe intrinsic physical properties of the scene objects in various imaging conditions and is strongly linked to the nature of materials for cars (e.g., metal and glass) and their surrounding environment (e.g., soil and trees), thereby providing reliable and discriminative features for robust car detection in challenging scenes. To exploit polarization cues, we first construct a pixel-aligned RGB-Polarization car detection dataset, which we subsequently employ to train a novel multimodal fusion network. Our car detection network dynamically integrates RGB and polarization features in a request-and-complement manner and can explore the intrinsic material properties of cars across all learning samples. We extensively validate our method and demonstrate that it outperforms state-of-the-art detection methods. Experimental results show that polarization is a powerful cue for car detection. Our code is available at https://github.com/wind1117/AAAI24-PCDNet.

Learning Sliding Policy of Flat Multi-target Objects in Clutter Scenes

In clutter scenes, one or several targets need to be obtained, which is hard for robot manipulation task. Especially, when the targets are flat objects like book, plates, due to limitation of common robot end-effectors, it will be more challenging. By employing pre-grasp operation like sliding, it becomes feasible to rearrange objects and shift the target towards table edge, enabling the robot to grasp it from a lateral perspective. In this paper, the proposed method transfers the task into a Parameterized Action Markov Decision Process to solve the problem, which is based on deep reinforcement learning. The mask images are taken as one of observations to the network for avoiding the impact of noise of original image. In order to improve data utilization, the policynetwork predicts the parameters for the sliding primitive of each object, which is weight-sharing, and then the Q-network selects the optimal execution target. Meanwhile, extra reward mechanism is adopted for improving the efficiency of task actions to cope with multiple targets. In addition, an adaptive policy scaling algorithm is proposed to improve the speed and adaptability of policy training. In both simulation and real system, our method achieves a higher task success rate and requires fewer actions to accomplish the flat multi-target sliding manipulation task within clutter scene, which verifies the effectiveness of ours.

A number sense as an emergent property of the manipulating brain

The ability to understand and manipulate numbers and quantities emerges during childhood, but the mechanism through which humans acquire and develop this ability is still poorly understood. We explore this question through a model, assuming that the learner is able to pick up and place small objects from, and to, locations of its choosing, and will spontaneously engage in such undirected manipulation. We further assume that the learner’s visual system will monitor the changing arrangements of objects in the scene and will learn to predict the effects of each action by comparing perception with a supervisory signal from the motor system. We model perception using standard deep networks for feature extraction and classification. Our main finding is that, from learning the task of action prediction, an unexpected image representation emerges exhibiting regularities that foreshadow the perception and representation of numbers and quantity. These include distinct categories for zero and the first few natural numbers, a strict ordering of the numbers, and a one-dimensional signal that correlates with numerical quantity. As a result, our model acquires the ability to estimate numerosity, i.e. the number of objects in the scene, as well as subitization, i.e. the ability to recognize at a glance the exact number of objects in small scenes. Remarkably, subitization and numerosity estimation extrapolate to scenes containing many objects, far beyond the three objects used during training. We conclude that important aspects of a facility with numbers and quantities may be learned with supervision from a simple pre-training task. Our observations suggest that cross-modal learning is a powerful learning mechanism that may be harnessed in artificial intelligence.

Frequency and spatial based multi-layer context network (FSCNet) for remote sensing scene classification

Remote Sensing Scene Classification (RSSC) is an important and challenging research topic due to the variety of land cover sizes and spatial combinations, as well as significant interclass similarity and intraclass variability. Currently, convolutional neural network (CNN)-based methods have been widely used in RSSC tasks with significant results. However, CNNs lack the ability to obtain long-term correlations. Transformer addressed this problem, thanks to the global receptive field of multi-head self-attention (MSA). Nevertheless, the vanilla transformer also needs further improvement to accommodate the diverse in type and scale of objects in RS scenes. In addition, the existing RSSC methods either use the last layer features, which is not conducive to process multi-scale remote sensing images, or directly fuse the multi-layer features, which will bring redundant or mutually exclusive information. To address the above issues, a novel RSSC framework, named frequency and spatial based multi-layer attention network (FSCNet) for remote sensing scene classification is proposed in this article. First, to fully extract the pyramid multi resolution features of CNN, a cross resolution injection model (CRIM) is proposed. Second, to generate better understand of the multilevel features, a frequency and spatial MLP (FS-MLP) is designed. Third, in order to aggregate contextual relations among multi-layer features, a multi-layer context align attention (MCAA) is adopted. The final classification is integration of top-layer feature and aggregated multi-layer feature. The experiment results on three well-known RS scene classification datasets (UCM, AID, and NWPU) prove the effectiveness of FSCNet and it outperforms many state-of-the-art methods.

A new framework for improving semantic segmentation in aerial imagery

High spatial resolution (HSR) remote sensing imagery presents a rich tapestry of foreground-background intricacies, rendering semantic segmentation in aerial contexts a formidable and vital undertaking. At its core, this challenge revolves around two pivotal questions: 1) Mitigating Background Interference and Enhancing Foreground Clarity. 2) Accurate Segmentation in Dense Small Object Cluster. Conventional semantic segmentation methods primarily cater to the segmentation of large-scale objects in natural scenes, yet they often falter when confronted with aerial imagery’s characteristic traits such as vast background areas, diminutive foreground objects, and densely clustered targets. In response, we propose a novel semantic segmentation framework tailored to overcome these obstacles. To address the first challenge, we leverage PointFlow modules in tandem with the Foreground-Scene (F-S) module. PointFlow modules act as a barrier against extraneous background information, while the F-S module fosters a symbiotic relationship between the scene and foreground, enhancing clarity. For the second challenge, we adopt a dual-branch structure termed disentangled learning, comprising Foreground Precedence Estimation and Small Object Edge Alignment (SOEA). Our foreground saliency guided loss optimally directs the training process by prioritizing foreground examples and challenging background instances. Extensive experimentation on the iSAID and Vaihingen datasets validates the efficacy of our approach. Not only does our method surpass prevailing generic semantic segmentation techniques, but it also outperforms state-of-the-art remote sensing segmentation methods.

Constructing a game engine: A proposed game engine architecture course for undergraduate students

This paper presents a structured series of lessons for teaching undergraduate computer science or game development students how to construct a game engine using C++ and standard graphics APIs such as OpenGL or DirectX. Our proposed course content discusses rendering topics, game object management, scene management, and rigid body physics where learners are tasked to incorporate them into a fully functional prototype scene editor. By incorporating actual coding into the course, students can gain a better understanding of low-level engine features and also hone their programming skills. In addition to the learning content, we provide recommended assessment methods. We include discussions about our teaching experience in handling a game engine architecture course, titled GDENG03, which 36 students have passed. Our paper stands out from related works by providing a framework for interested educators on how to write a 3D game engine and how to teach the same in class. Based on our actual delivery of GDENG03, all 36 students achieved high scores of >80% out of 100 which demonstrates the effectiveness of our learning content. We received positive student feedback, such as a better appreciation of game engines, while suggestions for course improvement were raised, such as using simpler terminologies and better workload management. Based on student scores and feedback, we can attribute the success of our students’ learning to our inclusion of practical coding examples and discussions, highlighting their relevance to mainstream game engines. The learning materials and accompanying source code can be accessed in the following link: .