Occluded Objects Research Articles

Detection-Free Object Tracking for Multiple Occluded Targets in Plenoptic Video

Multiple object tracking (MOT) is a fundamental task in vision, but MOT techniques for plenoptic video are scarce. Almost all 2D MOT algorithms that show high performance mostly use the detection-based method which has the disadvantage of operating only for a specific object. To enable tracking of arbitrary desired objects, this paper introduces a groundbreaking detection-free tracking method for MOT in plenoptic videos. The proposed method deviates from traditional detection-based tracking methods, emphasizing the challenges of tracking targets with occlusions. The paper presents specialized algorithms that exploit the multifocal information of plenoptic video, including the focal range restriction and dynamic focal range adjustment schemes to secure robustness for occluded object tracking. To the improvement of the spatial searching capability, the anchor ensemble and the dynamic change of spatial search region algorithms are also proposed. Additionally, in terms of MOT, to reduce the computation time involved, the motion-adaptive time scheduling technique is proposed, which improves computation speed while guaranteeing a certain level of accuracy. Experimental results show a significant improvement in tracking performance, with a 77% success rate based on intersection over union for occluded targets in plenoptic videos, marking a substantial advancement in the field of plenoptic object tracking.

Energy Demand in AR Applications—A Reverse Ablation Study of the HoloLens 2 Device

This study presents a comprehensive analysis of power consumption in the context of AR applications on the HoloLens 2. As an ablation study, the research systematically dissects various aspects, shedding light on both hardware and software components. The power consumption was examined in diverse scenarios, ranging from the device at idle to running AR applications of increasing complexity. Overall, the study provides insights into the power management aspects associated with Hololens 2 AR devices and applications, offering cues for optimizing energy efficiency and enhancing the user experience. The findings confirm that display-related factors, such as the screen brightness and content complexity, significantly impact power consumption. However, the results reveal that power usage does not scale linearly with the number of objects in an AR scene due to overhead factors and object occlusion. Furthermore, usage scenarios such as lighting conditions and large/minor user motions are analyzed. Moreover, the influence of different development platforms, Unity and Unreal, on power consumption is discussed, highlighting their respective contributions to energy usage. These findings might be useful for researchers, developers, and users of AR technology.

Fusion detection and ReID embedding with hybrid attention for multi-object tracking

Multi-object tracking (MOT) involves the prediction of object identities and their corresponding bounding boxes within video or image sequences. While numerous models have been proposed for MOT, there is still a lack of discrimination of object features and severe ID switches during the tracking stage. This paper presents a novel fusion detection and re-identification (ReID) embedding with hybrid attention for multi-object tracking to address this issue. It incorporates two major cores: a hybrid attention module (HAM) and an embedding association module (EAM). Firstly, the HAM comprises spatial-aware attention, scale-aware attention, and task-aware attention, aiming to obtain more informative features. By integrating these mechanisms, the proposed model can effectively handle variations in object scales and spatial relationships to promote discrimination and balance two tasks (detection and ReID). Secondly, we introduce an embedding association module to address the unreliable similarity matching during the tracking. Specifically, the EAM not only considers the appearance similarity but also ponders on geometric attributes to improve the ability to track in the presence of object occlusions and brief disappearances. Extensive experiments are conducted on the public MOT Challenge datasets, demonstrating that our method performs better than other advanced methods.

Rlm-tracking: online multi-pedestrian tracking supported by relative location mapping

The challenge of multi-object tracking stands as a fundamental focus in computer vision research, finding widespread applications in areas such as public safety, transportation, autonomous vehicles, robotics, and other domains involving artificial intelligence. Given the intricate nature of natural scenes, the occurrence of object occlusion and semi-occlusion is commonplace in basic tracking tasks. These factors often result in challenges such as ID switching, object loss, detection errors, and misaligned bounding boxes, thereby significantly impacting the precision of multi-object tracking.This paper aims to address the aforementioned issues and proposes a novel multi-object tracker, incorporating Relative location mapping (RLM) and Target region density (TRD) modeling. The new tracker is more sensitive to differences in the spatial relationships between targets, allowing it to dynamically introduce low-scoring detection boxes into different regions based on the density of target regions in the image. This improves the accuracy of target tracking while avoiding the consumption of a significant amount of computational resources.Our research results indicate that when applying this method to state-of-the-art multi-object tracking approaches, the proposed model achieves improvements of 0.4 to 0.8 points in the HOTA and IDF1 metrics on the MOT17 and MOT20 datasets. This demonstrates the effectiveness of the proposed method in enhancing multi-object tracking performance.

Surveillance video synopsis framework base on tube set

Video synopsis technology can shorten the length of the video, which has attracted wide attention. However, due to the limitation of object extraction technology and the difficulty of preserving interactivity, synopsis videos will lose the semantic information of the original video. To address the above problems, we propose a video synopsis framework based on tube sets. Firstly, we propose a video tracking algorithm based on Yolov4 and Kalman Filter, which can effectively alleviate the problem that it is difficult for object tracking technology to extract occluded objects. Secondly, we propose a method that combines moving direction and dynamic threshold to judge interactivity. Accurately judging interactivity can ensure that the interactive objects are not separated in the synopsis video. Finally, we propose a tube set mapping model (TSMM) that can rearrange with the tube set as the basic unit. Experimental results demonstrate that our method can achieve superior performance than other state-of-the-art methods.

Transparency-Aware Segmentation of Glass Objects to Train RGB-Based Pose Estimators.

Robotic manipulation requires object pose knowledge for the objects of interest. In order to perform typical household chores, a robot needs to be able to estimate 6D poses for objects such as water glasses or salad bowls. This is especially difficult for glass objects, as for these, depth data are mostly disturbed, and in RGB images, occluded objects are still visible. Thus, in this paper, we propose to redefine the ground-truth for training RGB-based pose estimators in two ways: (a) we apply a transparency-aware multisegmentation, in which an image pixel can belong to more than one object, and (b) we use transparency-aware bounding boxes, which always enclose whole objects, even if parts of an object are formally occluded by another object. The latter approach ensures that the size and scale of an object remain more consistent across different images. We train our pose estimator, which was originally designed for opaque objects, with three different ground-truth types on the ClearPose dataset. Just by changing the training data to our transparency-aware segmentation, with no additional glass-specific feature changes in the estimator, the ADD-S AUC value increases by 4.3%. Such a multisegmentation can be created for every dataset that provides a 3D model of the object and its ground-truth pose.

DA-Net: Density-Aware 3D Object Detection Network for Point Clouds

3D object detection is an important but demanding task, which has become an active research topic in the field of multimedia. Much recent research has been devoted to exploiting end-to-end trainable object detection networks with point clouds. However, most state-of-the-art methods have bottlenecks in detecting occluded objects and small objects, because the sparseness of point clouds is exacerbated on these objects. In this paper, a Density-Aware 3D object detection network (DA-Net) is proposed to improve the perception performance for detecting occluded and small objects, which contains four components: a backbone module with an inverse density scoring module (IDM) and a point-wise attention module (PAM), a 3D intersection over union Estimation Module (3DEM), a Consistent Label Assignment (CLA) method and an Adaptive-Soft-NMS method. The proposed backbone module makes the network concentrate on low-density points of occluded objects, and suppresses outliers and background points. Then, the 3DEM is introduced to evaluate the localization quality of the prediction boxes. Furthermore, the proposed CLA method can more accurately select positive and negative samples for small objects. Finally, Adaptive-Soft-NMS is proposed in our method to reduce the number of false detections during inference and thereby improve detection performance substantially. Extensive experiments demonstrated that the proposed method achieves state-of-the-art performance on two large-scale datasets, SUN RGB-D (62.1% in terms of mAP@0.25) and ScanNetV2 (67.1% in terms of mAP@0.25), and in particular, the detection accuracy of small objects and occluded objects are extremely improved.

TSFF: a two-stage fusion framework for 3D object detection.

Point clouds are highly regarded in the field of 3D object detection for their superior geometric properties and versatility. However, object occlusion and defects in scanning equipment frequently result in sparse and missing data within point clouds, adversely affecting the final prediction. Recognizing the synergistic potential between the rich semantic information present in images and the geometric data in point clouds for scene representation, we introduce a two-stage fusion framework (TSFF) for 3D object detection. To address the issue of corrupted geometric information in point clouds caused by object occlusion, we augment point features with image features, thereby enhancing the reference factor of the point cloud during the voting bias phase. Furthermore, we implement a constrained fusion module to selectively sample voting points using a 2D bounding box, integrating valuable image features while reducing the impact of background points in sparse scenes. Our methodology was evaluated on the SUNRGB-D dataset, where it achieved a 3.6 mean average percent (mAP) improvement in the mAP@0.25 evaluation criterion over the baseline. In comparison to other great 3D object detection methods, our method had excellent performance in the detection of some objects.

Estimating 6D Object Poses with Temporal Motion Reasoning for Robot Grasping in Cluttered Scenes

6D object pose estimation is an essential task in vision-based robotic grasping and manipulation. Prior works extract object's 6D pose by regressing from single RGB-D frame without considering the occluded objects in the frame, limiting their performance in human-robot collaboration scenarios with heavy occlusion. In this paper, we present an end-to-end model named \textit{TemporalFusion}, which integrates the temporal motion information from RGB-D images for 6D object pose estimation. The core of proposed model is to embed and fuse the temporal motion information from multi-frame RGB-D sequences, which could handle heavy occlusion in human-robot collaboration tasks. Furthermore, the proposed deep model can also obtain stable pose sequences, which is essential for real-time robotic grasping tasks. We evaluated the proposed method in the YCB-Video dataset, and experimental results show our model outperforms state-of-the-art approaches. Our code is available at https://github.com/mufengjun260/H-MPose.

Semantic and Correlation Disentangled Graph Convolutions for Multilabel Image Recognition.

Multilabel image recognition (MLR) aims to annotate an image with comprehensive labels and suffers from object occlusion or small object sizes within images. Although the existing works attempt to capture and exploit label correlations to tackle these issues, they predominantly rely on global statistical label correlations as prior knowledge for guiding label prediction, neglecting the unique label correlations present within each image. To overcome this limitation, we propose a semantic and correlation disentangled graph convolution (SCD-GC) method, which builds the image-specific graph and employs graph propagation to reason the labels effectively. Specifically, we introduce a semantic disentangling module to extract categorywise semantic features as graph nodes and develop a correlation disentangling module to extract image-specific label correlations as graph edges. Performing graph convolutions on this image-specific graph allows for better mining of difficult labels with weak visual representations. Visualization experiments reveal that our approach successfully disentangles the dominant label correlations existing within the input image. Through extensive experimentation, we demonstrate that our method achieves superior results on the challenging Microsoft COCO (MS-COCO), PASCAL visual object classes (PASCAL-VOC), NUS web image dataset (NUS-WIDE), and Visual Genome 500 (VG-500) datasets. Code is available at GitHub: https://github.com/caigitrepo/SCDGC.

Subjective and Objective Quality Assessment of Colonoscopy Videos.

Captured colonoscopy videos usually suffer from multiple real-world distortions, such as motion blur, low brightness, abnormal exposure, and object occlusion, which impede visual interpretation. However, existing works mainly investigate the impacts of synthesized distortions, which differ from real-world distortions greatly. This research aims to carry out an in-depth study for colonoscopy Video Quality Assessment (VQA). In this study, we advance this topic by establishing both subjective and objective solutions. Firstly, we collect 1,000 colonoscopy videos with typical visual quality degradation conditions in practice and construct a multi-attribute VQA database. The quality of each video is annotated by subjective experiments from five distortion attributes (i.e., temporal-spatial visibility, brightness, specular reflection, stability, and utility), as well as an overall perspective. Secondly, we propose a Distortion Attribute Reasoning Network (DARNet) for automatic VQA. DARNet includes two streams to extract features related to spatial and temporal distortions, respectively. It adaptively aggregates the attribute-related features through a multi-attribute association module to predict the quality score of each distortion attribute. Motivated by the observation that the rating behaviors for all attributes are different, a behavior guided reasoning module is further used to fuse the attribute-aware features, resulting in the overall quality. Experimental results on the constructed database show that our DARNet correlates well with subjective ratings and is superior nine state-of-the-art methods.

Multi-View Time-Series Hypergraph Neural Network for Action Recognition.

Recently, action recognition has attracted considerable attention in the field of computer vision. In dynamic circumstances and complicated backgrounds, there are some problems, such as object occlusion, insufficient light, and weak correlation of human body joints, resulting in skeleton-based human action recognition accuracy being very low. To address this issue, we propose a Multi-View Time-Series Hypergraph Neural Network (MV-TSHGNN) method. The framework is composed of two main parts: the construction of a multi-view time-series hypergraph structure and the learning process of multi-view time-series hypergraph convolutions. Specifically, given the multi-view video sequence frames, we first extract the joint features of actions from different views. Then, limb components and adjacent joints spatial hypergraphs based on the joints of different views at the same time are constructed respectively, temporal hypergraphs are constructed joints of the same view at continuous times, which are established high-order semantic relationships and cooperatively generate complementary action features. After that, we design a multi-view time-series hypergraph neural network to efficiently learn the features of spatial and temporal hypergraphs, and effectively improve the accuracy of skeleton-based action recognition. To evaluate the effectiveness and efficiency of MV-TSHGNN, we conduct experiments on NTU RGB+D, NTU RGB+D 120 and imitating traffic police gestures datasets. The experimental results indicate that our proposed method model achieves the new state-of-the-art performance.

ClockRay: A Wrist-Rotation Based Technique for Occluded-Target Selection in Virtual Reality.

Target selection is one of essential operation made available by interaction techniques in virtual reality (VR) environments. However, effectively positioning or selecting occluded objects is under-investigated in VR, especially in the context of high-density or a high-dimensional data visualization with VR. In this paper, we propose ClockRay, an occluded-object selection technique that can maximize the intrinsic human wrist rotation skills through the integration of emerging ray selection techniques in VR environments. We describe the design space of the ClockRay technique and then evaluate its performance in a series of user studies. Drawing on the experimental results, we discuss the benefits of ClockRay compared to two popular ray selection techniques - RayCursor and RayCasting. Our findings can inform the design of VR-based interactive visualization systems for high-density data.

Tooth Motion Monitoring in Orthodontic Treatment by Mobile Device-based Multi-view Stereo.

Nowadays, orthodontics has become an important part of modern personal life to assist one in improving mastication and raising self-esteem. However, the quality of orthodontic treatment still heavily relies on the empirical evaluation of experienced doctors, which lacks quantitative assessment and requires patients to visit clinics frequently for in-person examination. To resolve the aforementioned problem, we propose a novel and practical mobile device-based framework for precisely measuring tooth movement in treatment, so as to simplify and strengthen the traditional tooth monitoring process. To this end, we formulate the tooth movement monitoring task as a multi-view multi-object pose estimation problem via different views that capture multiple texture-less and severely occluded objects (i.e. teeth). Specifically, we exploit a pre-scanned 3D tooth model and a sparse set of multi-view tooth images as inputs for our proposed tooth monitoring framework. After extracting tooth contours and localizing the initial camera pose of each view from the initial configuration, we propose a joint pose estimation scheme to precisely estimate the 3D pose of each individual tooth, so as to infer their relative offsets during treatment. Furthermore, we introduce the metric of Relative Pose Bias to evaluate the individual tooth pose accuracy in a small scale. We demonstrate that our approach is capable of reaching high accuracy and efficiency as practical orthodontic treatment monitoring requires.

Contour refinement instance segmentation for occluded objects

Contour refinement instance segmentation for occluded objects

CMPF-UNet: a ConvNeXt multi-scale pyramid fusion U-shaped network for multi-category segmentation of remote sensing images

Most U-shaped convolutional neural network (CNN) methods have the problems of insufficient feature extraction and fail to fully utilize global/multi-scale context information, which makes it difficult to distinguish similar objects and shadow occluded objects in remote sensing images. This article proposes a ConvNeXt multi-scale pyramid fusion U-shaped network (CMPF-UNet). In this work, we first propose a novel backbone network based on ConvNeXt to enhance image feature extraction, and use ConvNeXt bottleneck blocks to reconstruct the decoder. Furthermore, a scale aware pyramid fusion (SAPF) module and Residual Atrous Spatial Pyramid Pooling (RASPP) module are proposed to dynamically fuse the rich multi-scale context information in advanced features. Finally, multiple Global Pyramid Guidance (GPG) modules are embedded in the network, aiming to provide different levels of global context information for the decoder by reconstructing skip-connections. Experiments on the Vaihingen and Potsdam datasets indicate that the proposed CMPF-UNet segmentation achieves more accurate results.

Comparison of 3D Point Cloud Completion Networks for High Altitude Lidar Scans of Buildings

High altitude lidar scans allow for rapid acquisition of big spatial data representing entire city blocks. Unfortunately, the raw point clouds acquired by this method are largely incomplete due to object occlusions and restrictions in scanning angles and sensor resolution, which can negatively affect the obtained results. In recent years, many new solutions for 3D point cloud completion have been created and tested on various objects; however, the application of these methods to high-altitude lidar point clouds of buildings has not been properly investigated yet. In the above context, this paper presents the results of applying several state-of-the-art point cloud completion networks to various building exteriors acquired by simulated airborne laser scanning. Moreover, the output point clouds generated from partial data are compared with complete ground-truth point clouds. The performed tests show that the SeedFormer network trained on the ShapeNet-55 data set provides promising shape completion results.

SDSNet: Building Extraction in High-Resolution Remote Sensing Images Using a Deep Convolutional Network with Cross-Layer Feature Information Interaction Filtering

Building extraction refers to the automatic identification and separation of buildings from the background in remote sensing images. It plays a significant role in urban planning, land management, and disaster monitoring. Deep-learning methods have shown advantages in building extraction, but they still face challenges such as variations in building types, object occlusions, and complex backgrounds. To address these issues, SDSNet, a deep convolutional network that incorporates global multi-scale feature extraction and cross-level feature fusion, is proposed. SDSNet consists of three modules: semantic information extraction (SIE), multi-level merge (MLM), and semantic information fusion (SIF). The SIE module extracts contextual information and improves recognition of multi-scale buildings. The MLM module filters irrelevant details guided by high-level semantic information, aiding in the restoration of edge details for buildings. The SIF module combines filtered detail information with extracted semantic information for refined building extraction. A series of experiments conducted on two distinct public datasets for building extraction consistently demonstrate that SDSNet outperforms the state-of-the-art deep-learning models for building extraction tasks. On the WHU building dataset, the overall accuracy (OA) and intersection over union (IoU) achieved impressive scores of 98.86% and 90.17%, respectively. Meanwhile, on the Massachusetts dataset, SDSNet achieved OA and IoU scores of 94.05% and 71.6%, respectively. SDSNet exhibits a unique advantage in recovering fine details along building edges, enabling automated and intelligent building extraction. This capability effectively supports urban planning, resource management, and disaster monitoring.

DetTrack: An Algorithm for Multiple Object Tracking by Improving Occlusion Object Detection

Multi-object tracking (MOT) is an important problem in computer vision that has a wide range of applications. Currently, object occlusion detecting is still a serious challenge in multi-object tracking tasks. In this paper, we propose a method to simultaneously improve occluded object detection and occluded object tracking, as well as propose a tracking method for when the object is completely occluded. First, motion track prediction is utilized to improve the upper limit of occluded object detection. Then, the spatio-temporal feature information between the object and the surrounding environment is used for multi-object tracking. Finally, we use the hypothesis frame to continuously track the completely occluded object. Our study shows that we achieve competitive performances compared to the current state-of-the-art methods on popular multi-object tracking benchmarks such as MOT16, MOT17, and MOT20.

ScanGuard-YOLO: Enhancing X-ray Prohibited Item Detection with Significant Performance Gains.

To address the problem of low recall rate in the detection of prohibited items in X-ray images due to the severe object occlusion and complex background, an X-ray prohibited item detection network, ScanGuard-YOLO, based on the YOLOv5 architecture, is proposed to effectively improve the model's recall rate and the comprehensive metric F1 score. Firstly, the RFB-s module was added to the end part of the backbone, and dilated convolution was used to increase the receptive field of the backbone network to better capture global features. In the neck section, the efficient RepGFPN module was employed to fuse multiscale information from the backbone output. This aimed to capture details and contextual information at various scales, thereby enhancing the model's understanding and representation capability of the object. Secondly, a novel detection head was introduced to unify scale-awareness, spatial-awareness, and task-awareness altogether, which significantly improved the representation ability of the object detection heads. Finally, the bounding box regression loss function was defined as the WIOUv3 loss, effectively balancing the contribution of low-quality and high-quality samples to the loss. ScanGuard-YOLO was tested on OPIXray and HiXray datasets, showing significant improvements compared to the baseline model. The mean average precision (mAP@0.5) increased by 2.3% and 1.6%, the recall rate improved by 4.5% and 2%, and the F1 score increased by 2.3% and 1%, respectively. The experimental results demonstrate that ScanGuard-YOLO effectively enhances the detection capability of prohibited items in complex backgrounds and exhibits broad prospects for application.