Cluttered Scenes Research Articles

Robot Grasp in Cluttered Scene Using a Multi-Stage Deep Learning Model

Robot Grasp in Cluttered Scene Using a Multi-Stage Deep Learning Model

CEPB dataset: a photorealistic dataset to foster the research on bin picking in cluttered environments.

Several datasets have been proposed in the literature, focusing on object detection and pose estimation. The majority of them are interested in recognizing isolated objects or the pose of objects in well-organized scenarios. This work introduces a novel dataset that aims to stress vision algorithms in the difficult task of object detection and pose estimation in highly cluttered scenes concerning the specific case of bin picking for the Cluttered Environment Picking Benchmark (CEPB). The dataset provides about 1.5M virtually generated photo-realistic images (RGB + depth + normals + segmentation) of 50K annotated cluttered scenes mixing rigid, soft, and deformable objects of varying sizes used in existing robotic picking benchmarks together with their 3D models (40 objects). Such images include three different camera positions, three light conditions, and multiple High Dynamic Range Imaging (HDRI) maps for domain randomization purposes. The annotations contain the 2D and 3D bounding boxes of the involved objects, the centroids' poses (translation + quaternion), and the visibility percentage of the objects' surfaces. Nearly 10K separated object images are presented to perform simple tests and compare them with more complex cluttered scenarios tests. A baseline performed with the DOPE neural network is reported to highlight the challenges introduced by the novel dataset.

PCSformer: Pair-wise Cross-scale Sub-prototypes mining with CNN-transformers for weakly supervised semantic segmentation

Generating initial seeds is an important step in weakly supervised semantic segmentation (WSSS). Our approach concentrates on generating and refining initial seeds. The convolutional neural networks (CNNs)–based initial seeds focus only on the most discriminative regions and lack global information about the target. The Vision Transformer (ViT)–based approach can capture long-range feature dependencies due to the unique advantage of the self-attention mechanism. Still, we find that it suffers from distractor object leakage and background leakage problems. Based on these observations, we propose PCSformer, which improves the model’s ability to extract features through a Pair-wise Cross-scale (PC) strategy and solves the problem of distractor object leakage by further extracting potential target features through Sub-Prototypes (SP) mining. In addition, the proposed Conflict Self-Elimination (CSE) module further alleviates the background leakage problem. We validate our approach on the widely adopted Pascal VOC 2012 and MS COCO 2014, and extensive experiments demonstrate our superior performance. Furthermore, our method proves to be competitive for WSSS in medical images and challenging scenarios involving deformable and cluttered scenes. Additionally, we extend the PCSformer to weakly supervised object localization tasks, further highlighting its scalability and versatility. The code is available at https://github.com/ChunmengLiu1/PCSformer.

Neural Correlates of Crowding in Macaque Area V4.

Visual crowding refers to the phenomenon where a target object that is easily identifiable in isolation becomes difficult to recognize when surrounded by other stimuli (distractors). Many psychophysical studies have investigated this phenomenon and proposed alternative models for the underlying mechanisms. One prominent hypothesis, albeit with mixed psychophysical support, posits that crowding arises from the loss of information due to pooled encoding of features from target and distractor stimuli in the early stages of cortical visual processing. However, neurophysiological studies have not rigorously tested this hypothesis. We studied the responses of single neurons in macaque (one male, one female) area V4, an intermediate stage of the object-processing pathway, to parametrically designed crowded displays and texture statistics-matched metameric counterparts. Our investigations reveal striking parallels between how crowding parameters-number, distance, and position of distractors-influence human psychophysical performance and V4 shape selectivity. Importantly, we also found that enhancing the salience of a target stimulus could alleviate crowding effects in highly cluttered scenes, and this could be temporally protracted reflecting a dynamical process. Thus, a pooled encoding of nearby stimuli cannot explain the observed responses, and we propose an alternative model where V4 neurons preferentially encode salient stimuli in crowded displays. Overall, we conclude that the magnitude of crowding effects is determined not just by the number of distractors and target-distractor separation but also by the relative salience of targets versus distractors based on their feature attributes-the similarity of distractors and the contrast between target and distractor stimuli.

Learning Sliding Policy of Flat Multi-target Objects in Clutter Scenes

In clutter scenes, one or several targets need to be obtained, which is hard for robot manipulation task. Especially, when the targets are flat objects like book, plates, due to limitation of common robot end-effectors, it will be more challenging. By employing pre-grasp operation like sliding, it becomes feasible to rearrange objects and shift the target towards table edge, enabling the robot to grasp it from a lateral perspective. In this paper, the proposed method transfers the task into a Parameterized Action Markov Decision Process to solve the problem, which is based on deep reinforcement learning. The mask images are taken as one of observations to the network for avoiding the impact of noise of original image. In order to improve data utilization, the policynetwork predicts the parameters for the sliding primitive of each object, which is weight-sharing, and then the Q-network selects the optimal execution target. Meanwhile, extra reward mechanism is adopted for improving the efficiency of task actions to cope with multiple targets. In addition, an adaptive policy scaling algorithm is proposed to improve the speed and adaptability of policy training. In both simulation and real system, our method achieves a higher task success rate and requires fewer actions to accomplish the flat multi-target sliding manipulation task within clutter scene, which verifies the effectiveness of ours.

A bat biomimetic model for scenario recognition using echo Doppler information

The flying bat can detect the difference in Doppler frequency between its echolocation transmission signal and the echoes in its surroundings, enabling it to distinguish between various scenarios effectively. By examining the bio-sonar biomimetic model of a flying bat that uses echo Doppler information for environmental recognition, it may enhance the scene recognition capability of human ultrasound sonar during movement. The paper establishes a three-dimensional clutter model of the flying state of bat bio-sonar for bats emitting constant frequency signals. It proposes a scene recognition method that combines multi-scale time-frequency feature analysis with a convolutional neural network (CNN). The short-time Fourier transform of different scales extract the Doppler and range dimensions, which are then fused to create a multi-scale feature plane containing both Doppler and range information. Combined with CNN’s powerful image classification and recognition capabilities, extract features from multi-scale feature planes of different clutter scenes to achieve environment recognition based on the differences in Doppler and range dimensions of echoes in various directions. Through computer simulations, this study provides a numerical interpretation of the environmental classification and perception capabilities of bats in flight. The algorithm significantly improves scenario classification and recognition performance according to simulation results, with accuracy exceeding 98% in varied clutter scenarios at 30 dB signal noise ratio. Based on computer simulations, an experimental scene was constructed and actual echo signals were collected and analyzed. The experiments demonstrate that utilizing Doppler information enables the classification and recognition of cluttered environments. The effectiveness of the proposed algorithm was also verified. Ultrasonic sonar systems, such as navigation robots and helicopter obstacle avoidance, can apply this biomimetic model and algorithm for environmental recognition during motion.

Landscape Characteristics in Mountain Parks across Different Urban Gradients and Their Relationship with Public Response

Numerous researchers have demonstrated the positive impacts of urban green spaces on human physiology and psychology. In mountainous urban regions, mountains have often been preserved as green spaces during urban sprawl, owing to the limited costs associated with development. While the landscape elements of these mountain parks exhibit differences depending on their locations, the nature and effects of such differences on the public’s physiological and psychological perceptions remain unclear. Therefore, we employed panoramic cameras and semantic segmentation (PSPNet-based training algorithm) to analyze the composition of landscape elements in mountain parks along an urban gradient (i.e., urban areas [UA], suburban areas [SA], and exurban areas [EA]). Concurrently, open-ended questionnaires and portable physiological monitors (ErgoLAB 3.0 Portable physiological monitoring equipment) were utilized to examine relationships between specific landscape elements and the public’s physiological and psychological responses. Our findings revealed that: (1) Urban park landscapes possessed high proportions of paved areas, humanistic vibe, vegetation hierarchy, and vegetation color richness, alongside lower scene clutter; suburban mountain park landscapes were characterized by heightened contemporary ambiance and wide viewshed area; and exurban mountain park landscapes exhibited high green view indices, expansive water surfaces, broad view area, and low scene clutter. (2) HRV and EMG differed significantly between mountain parks situated across the urban gradient. EMG also significantly varied across landscape types. All four psychological perception metrics showed significant distinctions across the three urban gradients and three green space categories. It further highlighted the importance of naturalness perception in urban mountain parks. (3) Viewshed area, average sight distance, architecture, enclosure, humanistic vibe, contemporary elements, vegetation color richness, trees and shrubs, distant hills, and scene clutter showed significant effects on both physiological and psychological outcomes. However, the application of these findings needs additional refinement tailored to the typology of the landscape. (4) To provide practical insights for constructing diverse green space typologies, we employed partial correlation modeling to eliminate covarying factors and developed a perception feedback model for public physiological and psychological indicators. Our findings elucidate relationships between landscape elements and the benefits of urban forests for public physiology and psychology. By shedding light on these connections, we further understand how landscape elements shape human perceptions of mountainous urban forests. These results offer valuable insights for shaping policies that promote favorable urban forest landscapes while also advancing landscape perception research through the use of semantic segmentation and portable physiological monitoring.

Semantic-Layout-Guided Image Synthesis for High-Quality Synthetic-Aperature Radar Detection Sample Generation

With the widespread application and functional complexity of deep neural networks (DNNs), the demand for training samples is increasing. This elevated requirement also extends to DNN-based SAR object detection. Most public SAR object detection datasets are oriented to marine targets such as ships, while data sets oriented to land targets are relatively rare, though they are an effective way to improve the land object detection capability of deep models through SAR sample generation. In this paper, a synthesis generation collaborative SAR sample augmentation framework is proposed to achieve flexible and diverse high-quality sample augmentation. First, a semantic-layout-guided image synthesis strategy is proposed to generate diverse detection samples. The issues of object location rationality and object layout diversity are also addressed. Meanwhile, a pix2pixGAN network guided by layout maps is utilized to achieve diverse background augmentation. Second, a progressive training strategy of diffusion models is proposed to achieve semantically controllable SAR sample generation to further improve the diversity of scene clutter. Finally, a sample cleaning method considering distribution migration and network filtering is employed to further improve the quality of detection samples. The experimental results show that this semantic synthesis generation method can outperform existing sample augmentation methods, leading to a comprehensive improvement in the accuracy metrics of classical detection networks.

Surgical navigation for guidewire placement from intraoperative fluoroscopy in orthopaedic surgery.

Objective. Surgical guidewires are commonly used in placing fixation implants to stabilize fractures. Accurate positioning of these instruments is challenged by difficulties in 3D reckoning from 2D fluoroscopy. This work aims to enhance the accuracy and reduce exposure times by providing 3D navigation for guidewire placement from as little as two fluoroscopic images.Approach. Our approach combines machine learning-based segmentation with the geometric model of the imager to determine the 3D poses of guidewires. Instrument tips are encoded as individual keypoints, and the segmentation masks are processed to estimate the trajectory. Correspondence between detections in multiple views is established using the pre-calibrated system geometry, and the corresponding features are backprojected to obtain the 3D pose. Guidewire 3D directions were computed using both an analytical and an optimization-based method. The complete approach was evaluated in cadaveric specimens with respect to potential confounding effects from the imaging geometry and radiographic scene clutter due to other instruments.Main results. The detection network identified the guidewire tips within 2.2 mm and guidewire directions within 1.1°, in 2D detector coordinates. Feature correspondence rejected false detections, particularly in images with other instruments, to achieve 83% precision and 90% recall. Estimating the 3D direction via numerical optimization showed added robustness to guidewires aligned with the gantry rotation plane. Guidewire tips and directions were localized in 3D world coordinates with a median accuracy of 1.8 mm and 2.7°, respectively.Significance. The paper reports a new method for automatic 2D detection and 3D localization of guidewires from pairs of fluoroscopic images. Localized guidewires can be virtually overlaid on the patient's pre-operative 3D scan during the intervention. Accurate pose determination for multiple guidewires from two images offers to reduce radiation dose by minimizing the need for repeated imaging and provides quantitative feedback prior to implant placement.

Scene-adaptive pattern coding-based fringe projection profilometry: diffuse surfaces identification and 3-D reconstruction in cluttered scenes.

Fringe projection profilometry (FPP) is one of the most widely used optical three-dimensional (3-D) perceiving techniques. However, when applied to cluttered scenes, acquiring accurate 3-D shapes is difficult because of the influences of indirect light caused by non-diffuse surfaces. In this paper, we first theoretically analyze and model the influences of indirect light in FPP, and then propose a scene-adaptive pattern coding-based method, which can design projection patterns based on the reflective properties of the scene's surfaces, to achieve accurate 3-D perceiving in cluttered scenes. Specifically, the scene confidence analysis method is first proposed to identify the reflective properties of various surfaces and localize the camera pixels of the diffuse surface. The illumination status (i.e., "0" or "1") of each projector pixel can be determined according to the camera-projection coordinate mapping and spatial pattern coding, where only diffuse surfaces can be illuminated, thus fundamentally preventing the influences of indirect light from the point of view of the light source. The 3-D shapes of diffuse surfaces can be accurately reconstructed in cluttered scenes. Different from traditional reflective properties change or light separation solutions, the proposed method can achieve accurate 3-D perceiving of cluttered scenes without additional hardware or expensive calculation. Extensive experiments verify that the proposed method outperforms the traditional methods in terms of accuracy and robustness.

Antipodal-points-aware dual-decoding network for robotic visual grasp detection oriented to multi-object clutter scenes

It is challenging for robots to detect grasps with high accuracy and efficiency-oriented to multi-object clutter scenes, especially scenes with objects of large-scale differences. Effective grasping representation, full utilization of data, and formulation of grasping strategies are critical to solving the problem. To this end, this paper proposes an antipodal-points grasping representation model. Based on this, the Antipodal-Points-aware Dual-decoding Network (APDNet) is presented for grasping detection in multi-object scenes. APDNet employs an encoding–decoding architecture. The shared encoding strategy based on an Adaptive Gated Fusion Module (AGFM) is proposed in the encoder to fuse RGB-D multimodal data. Two decoding branches, namely StartpointNet and EndpointNet, are presented to detect antipodal points. To better focus on objects at different scales in multi-object scenes, a global multi-view cumulative attention mechanism, called Global Accumulative Attention Mechanism (GAAM), is also designed in this paper for StartpointNet. The proposed method is comprehensively validated and compared using a public dataset and real robot platform. On the GraspNet-1Billion dataset, the proposed method achieves 30.7%, 26.4%, and 12.7% accuracy at a speed of 88.4 FPS for seen, unseen, and novel objects, respectively. On the AUBO robot platform, the detection and grasp success rates are 100.0% and 95.0% on single-object scenes and 97.0% and 90.3% on multi-object scenes, respectively. It is demonstrated that the proposed method exhibits state-of-the-art performance with well-balanced accuracy and efficiency.

Head and body cues guide eye movements and facilitate target search in real-world videos.

Static gaze cues presented in central vision result in observer shifts of covert attention and eye movements, and benefits in perceptual performance in the detection of simple targets. Less is known about how dynamic gazer behaviors with head and body motion influence search eye movements and performance in perceptual tasks in real-world scenes. Participants searched for a target person (yes/no task, 50% presence), whereas watching videos of one to three gazers looking at a designated person (50% valid gaze cue, looking at the target). To assess the contributions of different body parts, we digitally erase parts of the gazers in the videos to create three different body parts/whole conditions for gazers: floating heads (only head movements), headless bodies (only lower body movements), and the baseline condition with intact head and body. We show that valid dynamic gaze cues guided participants' eye movements (up to 3 fixations) closer to the target, speeded the time to foveate the target, reduced fixations to the gazers, and improved target detection. The effect of gaze cues in guiding eye movements to the search target was the smallest when the gazer's head was removed from the videos. To assess the inherent information about gaze goal location for each body parts/whole condition, we collected perceptual judgments estimating gaze goals by a separate group of observers with unlimited time. Observers' perceptual judgments showed larger estimate errors when the gazer's head was removed. This suggests that the reduced eye movement guidance from lower body cueing is related to observers' difficulty extracting gaze information without the presence of the head. Together, the study extends previous work by evaluating the impact of dynamic gazer behaviors on search with videos of real-world cluttered scenes.

Object-agnostic Affordance Categorization via Unsupervised Learning of Graph Embeddings

Acquiring knowledge about object interactions and affordances can facilitate scene understanding and human-robot collaboration tasks. As humans tend to use objects in many different ways depending on the scene and the objects’ availability, learning object affordances in everyday-life scenarios is a challenging task, particularly in the presence of an open set of interactions and objects. We address the problem of affordance categorization for class-agnostic objects with an open set of interactions; we achieve this by learning similarities between object interactions in an unsupervised way and thus inducing clusters of object affordances. A novel depth-informed qualitative spatial representation is proposed for the construction of Activity Graphs (AGs), which abstract from the continuous representation of spatio-temporal interactions in RGB-D videos. These AGs are clustered to obtain groups of objects with similar affordances. Our experiments in a real-world scenario demonstrate that our method learns to create object affordance clusters with a high V-measure even in cluttered scenes. The proposed approach handles object occlusions by capturing effectively possible interactions and without imposing any object or scene constraints.

Comparison of Terahertz Transceiver Array Topologies in Multiobject Imaging

We investigate the effect of side lobe level (SLL) and multipath propagation in millimeter-wave(mmWave) and terahertz (THz) imaging systems for various transceiver array topologies. This work considers practical limitations in high-resolution active imaging systems and focuses on electrically large, planar antenna array topologies. As such, we investigate the imaging performance of sparse transmitting/receiving (Tx/Rx) array topologies that enable high angular/spatial resolution imaging with minimum hardware complexity. We show that 1) properly tapering the sparse array excitation can mitigate image clutter caused by high side lobes and 2) sparse arrays are more vulnerable to multipath reflections and clutter in dense scenes with highly reflective surfaces. We present both full-wave numerical analysis and laboratory experiments (250-300 GHz) to verify and compare the imaging array performance for several exemplary scenes.

A model-free 6-DOF grasp detection method based on point clouds of local sphere area

ABSTRACT Grasping object is one of the basic tasks of robots in many scenarios. The main challenge is how to generate grasping poses for unknown objects in cluttered scenes. This paper proposes a model-free 6-DOF grasp detection framework based on single-view local point clouds. The whole process includes three stages: Candidate Generation Network(CGN), Reliable Adjustment Module(RAM), and Quality Assessment Network(QAN). CGN predicts the graspability and the initial grasp pose of the sampled points by the features extracted from the local sphere region based on the improved Pointnet. In order to better learn local area point clouds, we propose a progressive local region data learning mechanism, which can extract features from small to large scales efficiently. Candidate grasps are then consisting of graspable points and their grasping poses. RAM adjusts the position and width of the generated candidate grasps by using reliable heuristic rules. QAN uses a simpled-pointnet to evaluate the quality of the grasp candidates and filters out grasps with high confidence to execute. The proposed method not only achieves state-of-the-art results on GraspNet-1Billion but also shows high grasping success rates in real cluttered scenes.

Transformer Memory for Interactive Visual Navigation in Cluttered Environments

Substantial progress has been achieved in embodied visual navigation based on reinforcement learning (RL). These studies presume that the environment is stationary where all the obstacles are static. However, in real cluttered scenes, interactable objects (e.g. shoes and boxes) blocking the way of robots makes the environment non-stationary. Accordingly, the ego-centric visual agent will easily get stuck in the dilemma of finding the next waypoint as it struggles to decide whether to push the obstacles ahead. To handle the predicament, we formulate this interactive visual navigation as Partial Observed Markov Decision Process (POMDP). As the transformer encoder has demonstrated its superior ability to capture the spatial-temporal dependencies in natural language processing. We propose a transformer-based memory to empower the agents utilizing the historical interactive information. However, directly leveraging the transformer architecture in the RL settings is highly unstable. We further propose a surrogate objective to predict the next waypoint as the auxiliary task, which facilitates the representation learning and bootstraps the RL. We demonstrate our method in the iGibson environment and experimental results show a significant improvement over the interactive Gibson benchmark and the related recurrent RL policy both in the validation seen scenes and the test unseen scenes.

PP-GraspNet: 6-DoF grasp generation in clutter using a new grasp representation method

PurposeThe grasping task of robots in dense cluttered scenes from a single-view has not been solved perfectly, and there is still a problem of low grasping success rate. This study aims to propose an end-to-end grasp generation method to solve this problem.Design/methodology/approachA new grasp representation method is proposed, which cleverly uses the normal vector of the table surface to derive the grasp baseline vectors, and maps the grasps to the pointed points (PP), so that there is no need to add orthogonal constraints between vectors when using a neural network to predict rotation matrixes of grasps.FindingsExperimental results show that the proposed method is beneficial to the training of the neural network, and the model trained on synthetic data set can also have high grasping success rate and completion rate in real-world tasks.Originality/valueThe main contribution of this paper is that the authors propose a new grasp representation method, which maps the 6-DoF grasps to a PP and an angle related to the tabletop normal vector, thereby eliminating the need to add orthogonal constraints between vectors when directly predicting grasps using neural networks. The proposed method can generate hundreds of grasps covering the whole surface in about 0.3 s. The experimental results show that the proposed method has obvious superiority compared with other methods.

Frequency-Adaptive Learning for SAR Ship Detection in Clutter Scenes

Convolutional neural networks (CNNs) have been widely applied in the context of ship detection in synthetic aperture radar (SAR) images, but the detection performance is still not ideal in scenarios with clutter interference. Mining frequency-domain information to suppress the sea clutter in SAR ship detection has attracted wide attention. However, existing frequency-domain ship detection methods do not process frequency-domain information adaptively, which results in the degradation of ship detection performance. To overcome this problem, this article proposes a novel deep learning network called YOLO-FA. YOLO-FA contains the proposed Frequency Attention Module (FAM), which can process frequency-domain information of SAR images adaptively. The proposed method can suppress the sea clutter in the SAR images with the help of frequency-domain information. We evaluate the proposed method YOLO-FA on two datasets, i.e., the High-Resolution SAR images dataset (HRSID) and SAR Ship Detection Dataset (SSDD). Compared with the baseline method YOLOv5 and existing commonly used methods, YOLO-FA achieves state-of-the-art detection performance on both datasets.

Dim Moving Point Target Detection in Cloud Clutter Scenes Based on Temporal Profile Learning

This letter presents an approach for detecting dim moving point targets in cloud clutter scenes based on temporal profile learning. The main idea is that a weak transient disturbance will appear in the temporal profiles of target-present pixels, changing the temporal profile’s characteristics. We propose a novel signal-to-signal network to learn the temporal characteristics of the background and the clutter, in which the transient disturbance is extracted by the residual between the input signal and the reconstructed background and clutter signal. The structure of the ConvBlock-1D is designed to enhance the flow and propagation of features in layers. A loss function is proposed to solve the imbalance problem. We provided a comparison to other widely used methods by using simulated datasets and real-world image sequences. The experimental results demonstrate that our method has the best performance in terms of the qualitative and quantitative assessments.

Infrared Small Target Detection Based on Gradient Correlation Filtering and Contrast Measurement

Infrared small target detection under complex backgrounds, especially in dense cloud and changeable clutter scenes, has always been a challenging research task. In order to improve the detection ability of small targets under complex backgrounds, an infrared small target detection method based on gradient correlation filtering and gradient contrast measurement (GCF-CM) is proposed in this paper. The infrared gradient vector field (IGVF) of the original image is first constructed through the facet model. Then, considering the unique gradient characteristics of small targets, a gradient correlation filtering (GCF) method is proposed to filter small targets and background clutters. Meanwhile, a gradient contrast measurement (GCM) method is designed to further enhance the intensity of the small target. Finally, after fusing the two response maps, an adaptive threshold is adopted to extract small targets. Experimental results demonstrate that the proposed method can improve the intensity of the small target and suppress clutter sufficiently. In comparison with other excellent methods, the proposed method exhibits a robust detection performance.