Object Instances Research Articles

Concealed Hazardous Object Instance Segmentation for Terahertz Security Inspection Images With Channel-Spatialized Transformer

Concealed Hazardous Object Instance Segmentation for Terahertz Security Inspection Images With Channel-Spatialized Transformer

Flooded area detection and mapping from Sentinel-1 imagery. Complementary approaches and comparative performance evaluation

ABSTRACT The current study assesses the performance of several machine learning (ML) and deep learning (DL) models for detecting and mapping floods using Sentinel-1 SAR imagery. Three distinct approaches were used: pixel classification, object-based image analysis and object instance segmentation. The ML models are Random Forest, and Support Vector Machine and the DL models are U-NET, DeepLabV3 and Mask RCNN. Five different case studies were selected to test the models’ scalability. These areas are in Romania (Prut River, at the border between Romania, the Republic of Moldova and Ukraine, Timiș River, and Râul Negru River), the United States of America (Missouri River) and Australia (Broughton Creek). For each flood, five Sentinel-1 images were used, four collected before the flood and one collected after the flood. The intensity images were stacked and scaled in the range of the intensity thresholds associated with water and non-water so that all the case studies have the same margins for intensity. Samples of water, vegetation, agricultural fields, and bare soil were collected only from the Prut River case study and used in the training process. Out of all models, the U-Net model returned the highest accuracy with a value for Intersect over Union of 0.763 for a tile size of 128x128 pixels.

Rethinking the Non-Maximum Suppression Step in 3D Object Detection from a Bird’s-Eye View

In camera-based bird’s-eye view (BEV) 3D object detection, non-maximum suppression (NMS) plays a crucial role. However, traditional NMS methods become ineffective in BEV scenarios where the predicted bounding boxes of small object instances often have no overlapping areas. To address this issue, this paper proposes a BEV intersection over union (IoU) computation method based on relative position and absolute spatial information, referred to as B-IoU. Additionally, a BEV circular search method, called B-Grouping, is introduced to handle prediction boxes of varying scales. Utilizing these two methods, a novel NMS strategy called BEV-NMS is developed to handle the complex prediction boxes in BEV perspectives. This BEV-NMS strategy is implemented in several existing algorithms. Based on the results from the nuScenes validation set, there was an average increase of 7.9% in mAP when compared to the strategy without NMS. The NDS also showed an average increase of 7.9% under the same comparison. Furthermore, compared to the Scale-NMS strategy, the mAP increased by an average of 3.4%, and the NDS saw an average improvement of 3.1%.

Self-supervised learning-based re-parameterization instance segmentation for hazardous object in terahertz image

Recently, terahertz imaging technology has shown widespread potential applications in the public security screening field. Previous terahertz hazardous object detection techniques primarily relied on manual recognition and image processing methods. Some solutions incorporating deep learning technologies depend on large quantities of high-quality data, making it challenging to achieve low-cost and high-performance detection. To solve the issue of dependency on large amounts of high-quality data, we propose a diversified dynamic structure You Only Look Once (DDS-YOLO) model based on masked image modeling and structural reparameterization for the instance segmentation of hazardous objects in terahertz security inspection images. To address the scarcity and annotating difficulty problems of terahertz security inspection image samples, we combine automatic data strategies with masked image modeling for self-supervised learning. We propose a multilevel feature refinement fusion mechanism to enhance the quality of learned feature representations. The backbone parameter transfer and fine-tuning training strategies are employed to achieve hazardous object instance segmentation on the terahertz dataset. To address the low detection accuracy issue caused by the poor terahertz image quality, we develop a reparameterizable hierarchical structure for the backbone, improve a multi-scale feature-integrating neck, and design a dynamically decoupled head with lower computational requirements to enhance the performance of the instance segmentation model. Experimental results demonstrate that the proposed model accurately outputs detection boxes, categories, and segmentation masks for hazardous objects with minimal training samples. The comparative experimental results indicate that the proposed model outperforms existing state-of-the-art methods in terms of detection performance. The proposed DDS-YOLO model achieves 59.3 % in mask mean Average Precision (mAP) and 61.3 % in box mAP, and the model parameters and computational requirements also meet practical application scenarios.

Managing uncertainty in product and process design for the circular factory

Abstract In the circular factory, uncertain attributes of object instances and process steps are found at diverse occasions. Even if uncertainty can also be found to some extent in linear production, the high variation of product attributes of used objects causes the process steps in the circular factory to generate a much higher variability of the properties of the objects handled in circular processes. In consequence, a methodology is needed to model, handle and manage uncertainties at all relevant situations within the circular factory. In contrast to linear production, the uncertainty of attributes cannot be extended to an object class (with the same production history), but must be assigned to each object instance (with its own history) individually. In this contribution, the basic concepts for managing uncertainty in the circular factory are introduced. As a common basis, probabilities are used to express uncertainty, thus being compatible with the traditional and proven concepts of measurement science and stochastics. To describe the individual information state of object instances, it is complemented with a joint probability distribution describing all relevant object attributes. Some examples for processes within the circular factory demonstrate how uncertainty is considered to manage the uncertainty related challenges of used objects.

Enabling the vision of a perpetual innovative product – predicting function fulfillment of new product generations in a circular factory

Abstract The need for sustainability is driving the transition from linear to circular production. Customers, driven by the desire for innovation, prefer new products with the latest cutting-edge functions and design. The problem is that used products from older product generations are hence less attractive, leading to a limited market for remanufactured products and generating less revenue than their new counterparts. Therefore, this research addresses the problem of reprocessing and technologically updating used products into the newest product generation. To address this problem of reprocessing and technological updating with a Circular Factory, the product’s function fulfillment must be predicted in different situations in a Circular Factory. Three key challenges in which the function fulfillment must be predicted are identified as: Analysis and Diagnostics of Used Products and Subsystems, Prediction for Pairing and Reprocess of the Embodiment and Design for Circular Factory. A model-based approach is presented to address these three key challenges by predicting product function fulfillment to enable cross-generational use of subsystems and components. Five different types of models are proposed: A Functional Model, a Reliability Model and a Reference Architecture Model at the high level of general product modeling, as well as a Digital Twin of an object instance and a Tolerance Scheme at the level of individual instances. A theoretical case study shows the possible application in a Circular Factory using the example of an angle grinder. However, it’s essential to acknowledge that substantial work is needed for the enhancement or development of these models before practical application. Successful application of these models allows the Circular Factory not only to maintain but also to increase the value of reprocessed products to enable offering on the primary market and contribute to the vision of realizing a perpetual innovative product.

Multi-scale deep learning and clustering-based tabletop object instance segmentation for robot manipulation

3D object instance segmentation plays a vital role in various applications such as autonomous driving, robotics and virtual reality. However, tabletop scenes exhibit diverse object complexities and size variations. The challenge is to enhance the accuracy of segmenting these scenes for multiple object instances. This limitation directly impacts robots’ capabilities to effectively grasp and manipulate objects. In this paper, we propose a multi-scale deep learning and clustering-based approach for object instance segmentation in tabletop scenes. Our approach incorporates a multi-scale neighborhood feature sampling (MNFS) module specifically designed to extract local features, and a clustering algorithm to eliminate noise and preserve instance integrity. Furthermore, we combine the strength of both methods through ScoreNet and non-maximal suppression. We conducted extensive experiments on TO-Scene, the first large-scale dataset of 3D tabletop scenes, and observed an average mIoU improvement of approximately 4.07% compared to existing methods. This highlights the superior performance of our proposed method. In addition, we tested our algorithm on a real-scene robotics platform and showed that it has good performance and generalization capabilities to support future applications such as robot grasping.

Few-Shot Object Detection: A Comprehensive Survey.

Humans are able to learn to recognize new objects even from a few examples. In contrast, training deep-learning-based object detectors requires huge amounts of annotated data. To avoid the need to acquire and annotate these huge amounts of data, few-shot object detection (FSOD) aims to learn from few object instances of new categories in the target domain. In this survey, we provide an overview of the state of the art in FSOD. We categorize approaches according to their training scheme and architectural layout. For each type of approach, we describe the general realization as well as concepts to improve the performance on novel categories. Whenever appropriate, we give short takeaways regarding these concepts in order to highlight the best ideas. Eventually, we introduce commonly used datasets and their evaluation protocols and analyze the reported benchmark results. As a result, we emphasize common challenges in evaluation and identify the most promising current trends in this emerging field of FSOD.

Experimental Comparison of Two 6D Pose Estimation Algorithms in Robotic Fruit-Picking Tasks

This paper presents an experimental comparison between two existing methods representative of two categories of 6D pose estimation algorithms nowadays commonly used in the robotics community. The first category includes purely deep learning methods, while the second one includes hybrid approaches combining learning pipelines and geometric reasoning. The hybrid method considered in this paper is a pipeline of an instance-level deep neural network based on RGB data only and a geometric pose refinement algorithm based on the availability of the depth map and the CAD model of the target object. Such a method can handle objects whose dimensions differ from those of the CAD. The pure learning method considered in this comparison is DenseFusion, a consolidated state-of-the-art pose estimation algorithm selected because it uses the same input data, namely, RGB image and depth map. The comparison is carried out by testing the success rate of fresh food pick-and-place operations. The fruit-picking scenario has been selected for the comparison because it is challenging due to the high variability of object instances in appearance and dimensions. The experiments carried out with apples and limes show that the hybrid method outperforms the pure learning one in terms of accuracy, thus allowing the pick-and-place operation of fruits with a higher success rate. An extensive discussion is also presented to help the robotics community select the category of 6D pose estimation algorithms most suitable to the specific application.

Polygonal Approximation Learning for Convex Object Segmentation in Biomedical Images With Bounding Box Supervision.

As a common and critical medical image analysis task, deep learning based biomedical image segmentation is hindered by the dependence on costly fine-grained annotations. To alleviate this data dependence, in this article, a novel approach, called Polygonal Approximation Learning (PAL), is proposed for convex object instance segmentation with only bounding-box supervision. The key idea behind PAL is that the detection model for convex objects already contains the necessary information for segmenting them since their convex hulls, which can be generated approximately by the intersection of bounding boxes, are equivalent to the masks representing the objects. To extract the essential information from the detection model, a repeated detection approach is employed on biomedical images where various rotation angles are applied and a dice loss with the projection of the rotated detection results is utilized as a supervised signal in training our segmentation model. In biomedical imaging tasks involving convex objects, such as nuclei instance segmentation, PAL outperforms the known models (e.g., BoxInst) that rely solely on box supervision. Furthermore, PAL achieves comparable performance with mask-supervised models including Mask R-CNN and Cascade Mask R-CNN. Interestingly, PAL also demonstrates remarkable performance on non-convex object instance segmentation tasks, for example, surgical instrument and organ instance segmentation.

Human object interaction detection in paintings using multi-task learning

Human Object Interaction (HOI) detection can provide valuable insights into the meaning and interpretation of a painting, as the interactions between humans and objects can reveal information about the scene, characters, and story depicted in the artwork. Automatically detecting HOI in paintings is a challenging task, as the paintings often contain complex scenes with intricate details and variations in artistic style. Additionally, unlike in real-world images, the context and physics of the painting may not follow physical rules, which can further complicate the detection process. This paper proposes a novel system for detecting HOIs in paintings using multi-task learning. The system utilizes an object detection model to detect instances of human figures and objects, and extracts from them visual and spatial features. The appearance features are then combined to produce an optimized model for detecting HOIs. In order to enhance our model's performance on HOI detection, we train it in a multi-task learning setting with four different tasks. This approach allows us to leverage shared representations across multiple tasks, leading to improved accuracy and efficiency of HOI detection in our system. To train and test our model, we introduce a new benchmark for HOI detection in paintings, by augmenting the existing SemArt dataset with instance detection annotations and interaction classes and call it SemArt-HOI. Through our experiments, we show that our model is able to outperform the state-of-the-art one-stage transformer-based HOI detection model in both single-task and multi-task settings. Furthermore, our system's superior efficiency, training four times faster than the state-of-the-art model and using fewer resources, makes it ideal for practical and large-scale HOI detection in paintings.

Semantic 3D reconstruction-oriented image dataset for building component segmentation

In image-driven 3D building reconstruction, instance segmentation is fundamental to pixel-wise building component detection, which can be fused with 3D data like point clouds and meshes via camera projection for semantic reconstruction. While deep learning-based segmentation has obtained promising results, it relies heavily on large-scale datasets for training. Unfortunately, existing large-scale image datasets often include irrelevant objects that obstruct building components, making them unsuitable for 3D building reconstruction. This paper addresses this gap by introducing a large-scale building image dataset to facilitate building component segmentation for 3D reconstruction. The dataset comprises 3378 images captured from both interiors and exteriors of 36 university buildings, annotated with 49,380 object instances across 11 classes. Rigorous quality control measures were employed during data collection and annotation. Evaluation of five typical deep learning-based instance segmentation models demonstrates the dataset's suitability for training and its value as a benchmark dataset for building component segmentation.

Zero‐Shot 3D Pose Estimation of Unseen Object by Two‐step RGB-D Fusion

3D Object pose estimation is a critical task in many real-world applications, e.g., robotic manipulation and augmented reality. Most existing methods focus on estimating the object instances or categories which have been seen in the training phase. However, it is imperative to estimate the pose of unseen objects without re-training the network in real world. Therefore, we proposed a 3D pose estimation method for unseen objects without re-training. Specifically, given the CAD model of the unseen object, a set of template RGB-D images (RGB images and depth images) is rendered at different viewpoints. Then a feature embedding network, named PoseFusion, is designed to extract the scene feature. In this network, RGB-D images are utilized to extract the texture feature and geometric feature, respectively. Afterwards, a cross-modality alignment module is proposed to eliminate the noise in single modality. The aligned texture feature and aligned geometric feature are fused through a geometry guided fusion module. Thus, by PoseFusion, the template RGB-D images generated from the CAD model are abstracted into a set of template scene features, and the query scene features are also embedded from the captured RGB-D images from the unseen object. Finally, the query scene features are matched with the template scene features by calculating the masked local similarity. Then the identity and pose of unseen object are determined by the most similar template. Experiments on LINEMOD and T-LESS datasets demonstrate that our method outperforms other methods and generalizes better to unseen objects. Extensive ablation studies are performed to verify the effectiveness of the PoseFusion.

Interpreting convolutional neural network by joint evaluation of multiple feature maps and an improved NSGA-II algorithm

The ’black box’ characteristics of Convolutional Neural Networks (CNNs) present significant risks to their application scenarios, such as reliability, security, and division of responsibilities. Addressing the interpretability of CNN emerges as an urgent and critical issue in the field of machine learning. Recent research on CNN interpretability has either yielded unstable or inconsistent interpretations, or produced coarse-scale interpretable heatmaps, limiting their applicability in various scenarios. In this work, we propose a novel method of CNNs interpretation by incorporating a joint evaluation of multiple feature maps and employing multi-objective optimization (JE&MOO-CAM). Firstly, a method of joint evaluation for all feature maps is proposed to preserve the complete object instances and improve the overall activation values. Secondly, an interpretation method of CNNs under the MOO framework is proposed to avoid the instability and inconsistency of interpretation. Finally, the operators of selection, crossover, and mutation, along with the method of population initialization in NSGA-II, are redesigned to properly express the characteristics of CNNs. The experimental results, including both qualitative and quantitative assessments along with a sanity check conducted on three classic CNN models—VGG16, AlexNet, and ResNet50—demonstrate the superior performance of the proposed JE&MOO-CAM model. This model not only accurately pinpoints the instances within the image requiring explanation but also preserves the integrity of these instances to the greatest extent possible. These capabilities signify that JE&MOO-CAM surpasses six other leading state-of-the-art methods across four established evaluation criteria.

An Optimized Mask R-CNN with Bag-of-Visual Words and Fast+Surf Algorithm in Sharp Object Instance Segmentation for X-ray Security

Automated security X-ray analysis is highly desired for efficiently inspecting sharp objects. The research formulated an optimized approach for sharp object detection using a Mask R-CNN architecture. The dataset used during the training phase consists of 238 balanced raw images extracted from GitHub named OPIXray. The researchers utilized recent advances in computer vision algorithms, including the Bag-of-Words and Fast+Surf feature extraction techniques, to improve the accuracy and reliability of object deletion. The research demonstrated that the optimized versions of the classification and object detection models have significantly improved accuracy for most categories, with a 5% improvement for the clear category and a 3% improvement for both the scissor and straight knife detection.

Open-set marine object instance segmentation with prototype learning

The ocean world is full of Unknown Marine Objects (UMOs), making it difficult to deal with unknown ocean targets using the traditional instance segmentation model. This is because the traditional instance segmentation networks are trained on a closed dataset, assuming that all detected objects are Known Marine Objects (KMOs). Consequently, traditional closed-set networks often misclassify UMOs as KMOs. To address this problem, this paper proposes a new open-set instance segmentation model for object instance segmentation in marine environments with UMOs. Specifically, we integrate two learning modules in the model, namely a prototype module and an unknown learning module. Through the learnable prototype, the prototype module improves the class’s compactness and boundary detection capabilities while also increasing the classification accuracy. Through the uncertainty of low probability samples, the unknown learning module forecasts the unknown probability. Experimental results illustrate that the proposed method has competitive known class recognition accuracy compared to existing instance segmentation models, and can accurately distinguish unknown targets.

Recognizing task-level events from user interaction data

User interaction data comprises events that capture individual actions that a user performs on their computer. Such events provide detailed records about how users carry out their tasks in a process, even when this involves different applications. Although the comprehensiveness of such data provides a promising basis for process mining, user interaction events cannot be used directly for this purpose, because they do not meet two essential requirements. In particular, they neither indicate their relation to a process-level activity nor their relation to a specific process execution. Therefore, user interaction data needs to be transformed so that it meets these requirements before process mining techniques can be applied. This transformation problem comprises identifying tasks and their types and determining the relation between tasks and process executions. While some existing approaches tackle parts of this problem, none address it comprehensively. Therefore, we propose an unsupervised approach for recognizing task-level events from user interaction data that addresses it in full. It segments user interaction data to identify tasks, categorizes these according to their type, and relates tasks to each other via object instances it extracts from the user interaction events. In this manner, our approach creates task-level events that meet the requirements of process mining settings. Our evaluation demonstrates the approach’s efficacy and shows that its combined consideration of control-flow, data, and semantic information allows it to outperform baseline approaches in both online and offline settings.

Category Level Object Pose Estimation via Global High-Order Pooling

Category level 6D object pose estimation aims to predict the rotation, translation and size of object instances in any scene. In current research methods, global average pooling (first-order) is usually used to explore geometric features, which can only capture the first-order statistical information of the features and do not fully utilize the potential of the network. In this work, we propose a new high-order pose estimation network (HoPENet), which enhances feature representation by collecting high-order statistics to model high-order geometric features at each stage of the network. HoPENet introduces a global high-order enhancement module and utilizes global high-order pooling operations to capture the correlation between features and fuse global information. In addition, this module can capture long-term statistical correlations and make full use of contextual information. The entire network finally obtains a more discriminative feature representation. Experiments on two benchmarks, the virtual dataset CAMERA25 and the real dataset REAL275, demonstrate the effectiveness of HoPENet, achieving state-of-the-art (SOTA) pose estimation performance.

Hierarchical mussel farm reconstruction from video with object tracking

ABSTRACT As the mussel farming industry expands, particularly in regions such as New Zealand, there is a growing need for advanced monitoring and management solutions to ensure sustainability and operational efficiency. The current reliance on manual infrequent observation of aquaculture structures limits farmers' ability to monitor them in real time. Addressing these challenges, large-scale 3D reconstruction provides a practical solution by facilitating the creation of replicable depictions of mussel farm scenes and floatation buoys derived from recorded video, supporting off-site evaluation and enabling precise decision making. This paper introduces a novel approach to enhance the visualisation and monitoring of mussel farm floatation through a hierarchical reconstruction process. In contrast to earlier studies, we focus on recovering not only the overall environment (background) but also the finer details of key elements such as buoys to create a comprehensive representation of mussel farm geometry and appearance. We propose to segment the scene into the background and granular object instances and reconstruct them separately in a multi-stage process. The initial 3D scene reconstruction is performed using the Structure-from-Motion (SfM) technique, leveraging video footage captured by a vessel-mounted camera. This coarse reconstruction serves as the foundation for subsequent fine-grained enhancements. To recover finer details, object tracking is applied and the trajectories obtained are then conjunct with geometry triangulation to determine the real-world positions of individual buoys. A multiple-scale denoising method, grounded in dominant direction correlation, is implemented to eliminate non-reliable tracking objects and reduce reconstruction artifacts, ensuring the accuracy of the final results. This hierarchical reconstruction approach contributes to the advancement of mussel farm management by offering a powerful technology for comprehensive visualisation, enabling farmers to make informed decisions based on a detailed understanding of the mussel farm's geometry and dynamics.

QDETRv: Query-Guided DETR for One-Shot Object Localization in Videos

In this work, we study one-shot video object localization problem that aims to localize instances of unseen objects in the target video using a single query image of the object. Toward addressing this challenging problem, we extend a popular and successful object detection method, namely DETR (Detection Transformer), and introduce a novel approach –query-guided detection transformer for videos (QDETRv). A distinctive feature of QDETRv is its capacity to exploit information from the query image and spatio-temporal context of the target video, which significantly aids in precisely pinpointing the desired object in the video. We incorporate cross-attention mechanisms that capture temporal relationships across adjacent frames to handle the dynamic context in videos effectively. Further, to ensure strong initialization for QDETRv, we also introduce a novel unsupervised pretraining technique tailored to videos. This involves training our model on synthetic object trajectories with an analogous objective as the query-guided localization task. During this pretraining phase, we incorporate recurrent object queries and loss functions that encourage accurate patch feature reconstruction. These additions enable better temporal understanding and robust representation learning. Our experiments show that the proposed model significantly outperforms the competitive baselines on two public benchmarks, VidOR and ImageNet-VidVRD, extended for one-shot open-set localization tasks.