Object Instances Research Articles

Generating Activity Snippets by Learning Human-Scene Interactions

We present an approach to generate virtual activity snippets, which comprise sequenced keyframes of multi-character, multi-object interaction scenarios in 3D environments, by learning from recordings of human-scene interactions. The generation consists of two stages. First, we use a sequential deep graph generative model with a temporal module to iteratively generate keyframe descriptions, which represent abstract interactions using graphs, while preserving spatial-temporal relations through the activities. Second, we devise an optimization framework to instantiate the activity snippets in virtual 3D environments guided by the generated keyframe descriptions. Our approach optimizes the poses of character and object instances encoded by the graph nodes to satisfy the relations and constraints encoded by the graph edges. The instantiation process includes a coarse 2D optimization followed by a fine 3D optimization to effectively explore the complex solution space for placing and posing the instances. Through experiments and a perceptual study, we applied our approach to generate plausible activity snippets under different settings.

Mobile Robot Object Detection Method Based on Deep Learning

The task of object detection is to accurately and efficiently identify and locate a large number of predefined categories of object instances from images. With the wide application of deep learning, the accuracy and efficiency of target detection have been greatly improved. However, deep learning-based target detection still faces challenges from key technologies such as improving and optimizing the performance of mainstream target detection algorithms, improving the detection accuracy of small target objects, realizing multi-class object detection and lightweight detection model. In response to the above challenges, based on extensive literature research, this paper analyses methods for lightweight detection models and improved detection accuracy from the perspective of YOLOv5s network structure. The problems to be solved in target detection and the future research direction are predicted and prospected.

외형특징 추출 기능을 개선한 딥러닝 기반의 다중객체 추적

Multi-Object Tracking (MOT) aims to track the trajectories of multiple objects simultaneously in a video over time. In MOT, object detection is used to locate instances of objects in images or videos, and re-identification is employed to match objects with the same ID. Among recent MOT methods, one-shot MOT, which relies on a single neck and head for object detection and appearance feature extraction, faces a conflict between object category classification and unique ID classification. To address this challenge, we propose a solution that integrates an FPN neck-based appearance feature extraction module into YOLOX. This method effectively mitigates the conflict between object category classification and unique ID classification in one-shot MOT. Furthermore, the proposed variable offset mechanism enables correction of the extraction position, even in cases of object occlusion. The proposed technique achieved 83.7% mIDF1 and 75% mMOTA in the experimental results for the TITAN dataset. This improved mIDF1 and mMOTA by 0.2% and 0.2% respectively over the IOU-based MOT, and by 1.2% and 0.7% respectively over the one-shot MOT.

INITIAL ASSESSMENT ON THE USE OF STATE-OF-THE-ART NERF NEURAL NETWORK 3D RECONSTRUCTION FOR HERITAGE DOCUMENTATION

Abstract. In recent decades, photogrammetry has re-emerged as a viable solution for heritage documentation. Developments in various computer vision methods have helped photogrammetry to compete against the laser scanning technology, eventually becoming complementary solutions for the purpose of heritage recording. In the last few years, artificial intelligence (AI) has progressively entered various domains including 3D reconstruction. The Neural Radiance Fields (NeRF) method renders a 3D scene from a series of overlapping images, similar to photogrammetry. However, instead of relying on geometrical relations between the image and world spaces, it uses neural networks to recreate the so-called radiance fields. The result is a significantly faster method of recreating 3D scenes. While not designed to generate 3D models, simple computer graphics methods can be used to convert these recreated radiance fields into the familiar point cloud. In this paper, we implemented the Nerfacto architecture to recreate two instances of heritage objects and then compared them to traditional photogrammetric multi-view stereo (MVS). While the initial hypothesis posits that NeRF is not yet capable to reach the level of accuracy and density achieved by MVS as can be observed in the results, NeRF nevertheless shows a great potential due to its fractionally faster processing speed.

Scene-Level Sketch-Based Image Retrieval with Minimal Pairwise Supervision

The sketch-based image retrieval (SBIR) task has long been researched at the instance level, where both query sketches and candidate images are assumed to contain only one dominant object. This strong assumption constrains its application, especially with the increasingly popular intelligent terminals and human-computer interaction technology. In this work, a more general scene-level SBIR task is explored, where sketches and images can both contain multiple object instances. The new general task is extremely challenging due to several factors: (i) scene-level SBIR inherently shares sketch-specific difficulties with instance-level SBIR (e.g., sparsity, abstractness, and diversity), (ii) the cross-modal similarity is measured between two partially aligned domains (i.e., not all objects in images are drawn in scene sketches), and (iii) besides instance-level visual similarity, a more complex multi-dimensional scene-level feature matching problem is imposed (including appearance, semantics, layout, etc.). Addressing these challenges, a novel Conditional Graph Autoencoder model is proposed to deal with scene-level sketch-images retrieval. More importantly, the model can be trained with only pairwise supervision, which distinguishes our study from others in that elaborate instance-level annotations (for example, bounding boxes) are no longer required. Extensive experiments confirm the ability of our model to robustly retrieve multiple related objects at the scene level and exhibit superior performance beyond strong competitors.

Amodal Instance Segmentation via Prior-Guided Expansion

Amodal instance segmentation aims to infer the amodal mask, including both the visible part and occluded part of each object instance. Predicting the occluded parts is challenging. Existing methods often produce incomplete amodal boxes and amodal masks, probably due to lacking visual evidences to expand the boxes and masks. To this end, we propose a prior-guided expansion framework, which builds on a two-stage segmentation model (i.e., Mask R-CNN) and performs box-level (resp., pixel-level) expansion for amodal box (resp., mask) prediction, by retrieving regression (resp., flow) transformations from a memory bank of expansion prior. We conduct extensive experiments on KINS, D2SA, and COCOA cls datasets, which show the effectiveness of our method.

Utilizing Deep Learning to Detect Objects in Real Time

Abstract: Computer vision is related to object detection. Detecting instances of objects in images and videos is made possible by object detection. It recognizes the component of Pictures rather than conventional article recognition techniques and produces an keen comprehension of pictures very much like human vision works. In this paper, We restored starts the concise presentation of profound learning and item discovery system like Convolutional Brain Network (CNN), Repetitive brain network (RNN), quicker RNN, You just look once (Consequences be damned). After that, we concentrate on the modifications to our object detection architectures that we have proposed. In images, the conventional model can identify a small object. We have a few changes to the model. The method we propose yields the correct result precisely.

Generate-Paste-Blend-Detect: Synthetic dataset for object detection in the agriculture domain

Object detection is a challenging task, hindered by the scarcity of large annotated datasets. In agriculture, the lack of annotated insect datasets often results in domain-specific models that lack generalization. Data collection and annotation can be expensive and time-consuming. This paper proposes a simple approach to generate synthetic datasets for object detection that requires only a small dataset of target objects and a larger background dataset that fits the desired environment. The approach named Generate-Paste-Blend-Detect uses Denoising Diffusion Probabilistic Models (DDPM) to artificially “generate” objects, “paste” them on a background image, “blend” them with the environment to avoid pixel artifacts which result in poor performance for trained models, and finally use an object detection model to “detect” the artificially added object instances. The proposed methodology is demonstrated in the agricultural domain to detect whiteflies achieving a mean average precision (mAP50) of 0.66 with the state-of-the-art YOLOv8 object detection model. This approach enables domain-specific detection with minimal labor and cost.

Triple critical feature capture network: A triple critical feature capture network for weakly supervised object detection

AbstractWeakly supervised object detection (WSOD) is becoming increasingly important for computer vision tasks, as it alleviates the burden of manual annotation. Most WSOD techniques rely on multiple instance learning (MIL), which tends to localise the discriminative parts of salient objects instead of the whole object. In addition, network training is often supervised using simple image‐level annotations, without including object quantities or location information. However, this can lead to ambiguous differentiation of object instances, both in terms of location and semantics. To address these issues, propose an end‐to‐end triple critical feature capture network (TCFCNet) for WSOD is proposed. Specifically, a multi‐task branch, which can perform fully supervised classification and regression task, was integrated with a PCL in an end‐to‐end network for refining object locations in an online method. A cyclic parametric dropblock module (CPDM) was then designed to help the detector focus on the contextual information by using cyclic masking techniques to maximise the removal of the discriminative components of an object instance to alleviate the part domination problem. Finally, a feature decoupling module (FDM) is proposed to further reduce the ambiguous distinction of object instances by adaptively constructing robust critical features that adapt to multi‐task branch for classification and regression tasks, which contains a feature enhancement module and task‐specific polarisation functions. Comprehensive experiments are carried out on the challenging Pascal VOC 2007 and VOC 2012 datasets. The proposed method achieves a 54.6% mAP and a 44.3% mAP on the Pascal VOC 2007 and VOC 2012 datasets respectively, showed that our method outperformed existing mainstream techniques by a considerable margin.

What’s in a name?

The development and evolution of the “Einstein–Æther Theory” (Æ-theory) shows that there is a field in cosmology where the word ether is being used again. It is unclear, however, whether this æther may be regarded in continuation with previous ethers, or it is an altogether new entity. The main goal of this paper is to understand the nature of this new ether in the context of previous instances of this scientific object. In order to do so, we shall first give a brief historical account of the distinct uses the word had assumed in the late nineteenth and early twentieth centuries, before its demise. Then, we shall describe the major attempts to revive the ether over the last century, focusing on the last endeavor: the Æ-theory. In this article, we do not intend to support or reject this new use of the word, but to stress the complexity of establishing a consistent historical narrative of some scientific objects like the ether.

HIT-UAV: A high-altitude infrared thermal dataset for Unmanned Aerial Vehicle-based object detection

We present the HIT-UAV dataset, a high-altitude infrared thermal dataset for object detection applications on Unmanned Aerial Vehicles (UAVs). The dataset comprises 2,898 infrared thermal images extracted from 43,470 frames in hundreds of videos captured by UAVs in various scenarios, such as schools, parking lots, roads, and playgrounds. Moreover, the HIT-UAV provides essential flight data for each image, including flight altitude, camera perspective, date, and daylight intensity. For each image, we have manually annotated object instances with bounding boxes of two types (oriented and standard) to tackle the challenge of significant overlap of object instances in aerial images. To the best of our knowledge, the HIT-UAV is the first publicly available high-altitude UAV-based infrared thermal dataset for detecting persons and vehicles. We have trained and evaluated well-established object detection algorithms on the HIT-UAV. Our results demonstrate that the detection algorithms perform exceptionally well on the HIT-UAV compared to visual light datasets, since infrared thermal images do not contain significant irrelevant information about objects. We believe that the HIT-UAV will contribute to various UAV-based applications and researches. The dataset is freely available at https://pegasus.ac.cn.

DiSECt: a differentiable simulator for parameter inference and control in robotic cutting

Robotic cutting of soft materials is critical for applications such as food processing, household automation, and surgical manipulation. As in other areas of robotics, simulators can facilitate controller verification, policy learning, and dataset generation. Moreover, differentiable simulators can enable gradient-based optimization, which is invaluable for calibrating simulation parameters and optimizing controllers. In this work, we present DiSECt: the first differentiable simulator for cutting soft materials. The simulator augments the finite element method with a continuous contact model based on signed distance fields, as well as a continuous damage model that inserts springs on opposite sides of the cutting plane and allows them to weaken until zero stiffness, enabling crack formation. Through various experiments, we evaluate the performance of the simulator. We first show that the simulator can be calibrated to match resultant forces and deformation fields from a state-of-the-art commercial solver and real-world cutting datasets, with generality across cutting velocities and object instances. We then show that Bayesian inference can be performed efficiently by leveraging the differentiability of the simulator, estimating posteriors over hundreds of parameters in a fraction of the time of derivative-free methods. Next, we illustrate that control parameters in the simulation can be optimized to minimize cutting forces via lateral slicing motions. Finally, we conduct experiments on a real robot arm equipped with a slicing knife to infer simulation parameters from force measurements. By optimizing the slicing motion of the knife, we show on fruit cutting scenarios that the average knife force can be reduced by more than $$40\%$$ compared to a vertical cutting motion. We publish code and additional materials on our project website at https://diff-cutting-sim.github.io .

Volume conductor: interactive visibility management for crowded volumes

We present a novel smart visibility system for visualizing crowded volumetric data containing many object instances. The presented approach allows users to form groups of objects through membership predicates and to individually control the visibility of the instances in each group. Unlike previous smart visibility approaches, our approach controls the visibility on a per-instance basis and decides which instances are displayed or hidden based on the membership predicates and the current view. Thus, cluttered and dense volumes that are notoriously difficult to explore effectively are automatically sparsified so that the essential information is extracted and presented to the user. The proposed system is generic and can be easily integrated into existing volume rendering applications and applied to many different domains. We demonstrate the use of the volume conductor for visualizing fiber-reinforced polymers and intracellular organelle structures.

Super Intendo: Semantic Robot Programming from Multiple Demonstrations for taskable robots

When an end-user instructs a taskable robot on a new task, it is important for the robot to learn the user’s intention for the task. Knowing the user’s intention, represented as desired goal conditions, allows the robot to generalize across variations of the learned task seen at execution time. However, it has proven challenging to learn goal conditions due to the large, noisy, and complex space of goal conditions expressed by human users. This paper introduces Semantic Robot Programming with Multiple Demonstrations (SRP-MD) to learn a generative model of latent end-user task goal conditions from multiple end-user demonstrations in a shared workspace. By learning a generative model of the goal conditions, SRP-MD generalizes to task instances even when the quantity of objects to be arranged is not in the training set or novel object instances are included. At test time, a new goal is pulled from the learned generative model given the objects present in the initial scene. The efficacy of SRP-MD as a step toward taskable robots is shown on a Fetch robot learning and executing bin packing tasks in a simulated environment with grocery items.

DEVELOPMENT OF ANIMAL TRACKING SYSTEM USINGCOMPUTER VISION

The behavior of animals in the wild is studied animal migration tracking in wildlife biology, conservation biology, and ecology and wildlife management. Animal banding, which involved attaching passive ID tags to animal’s legs to identify it in a sub sequent catch and release operation, was one of the earliest techniques used. A tiny radio transmitter is fastened to the animals during radio tracking, and a receiver tracks the signal.In genius temporary methods use GP tags that retain are cord of the animal’s where abuts as well as satellites to follow tagged animals. Devices tailored to the species or item being monitored by the you look only once algorithm may now be developed thanks to the development of new technology. One of the numerous goals of animal migration research has been to determine the creature’s final locations. The goal of object detection, a field of computer science related to computer vision and image processing, is to find instances of semantic objects belonging to a specific class in digital pictures and videos. Two well researched object detection fields are face and pedestrian detection. Object detection is used in several computer vision domains, including video surveillance and OPEN CV library.

Cycle-object consistency for image-to-image domain adaptation

Recent advances in generative adversarial networks (GANs) have been proven effective in performing domain adaptation for object detectors through data augmentation. While GANs are exceptionally successful, those methods that can preserve objects well in the image-to-image translation task usually require an auxiliary task, such as semantic segmentation to prevent the image content from being too distorted. However, pixel-level annotations are difficult to obtain in practice. Alternatively, instance-aware image-translation model treats object instances and background separately. Yet, it requires object detectors at test time, assuming that off-the-shelf detectors work well in both domains. In this work, we present AugGAN-Det, which introduces Cycle-object Consistency (CoCo) loss to generate instance-aware translated images across complex domains. The object detector of the target domain is directly leveraged in generator training and guides the preserved objects in the translated images to carry target-domain appearances. Compared to previous models, which e.g., require pixel-level semantic segmentation to force the latent distribution to be object-preserving, this work only needs bounding box annotations which are significantly easier to acquire. Next, as to the instance-aware GAN models, our model, AugGAN-Det, internalizes global and object style-transfer without explicitly aligning the instance features. Most importantly, a detector is not required at test time. Experimental results demonstrate that our model outperforms recent object-preserving and instance-level models and achieves state-of-the-art detection accuracy and visual perceptual quality.

An In-depth Study of Java Deserialization Remote-Code Execution Exploits and Vulnerabilities

Nowadays, an increasing number of applications use deserialization. This technique, based on rebuilding the instance of objects from serialized byte streams, can be dangerous since it can open the application to attacks such as remote code execution (RCE) if the data to deserialize is originating from an untrusted source. Deserialization vulnerabilities are so critical that they are in OWASP’s list of top 10 security risks for web applications. This is mainly caused by faults in the development process of applications and by flaws in their dependencies, i.e., flaws in the libraries used by these applications. No previous work has studied deserialization attacks in-depth: How are they performed? How are weaknesses introduced and patched? And for how long are vulnerabilities present in the codebase? To yield a deeper understanding of this important kind of vulnerability, we perform two main analyses: one on attack gadgets, i.e., exploitable pieces of code, present in Java libraries, and one on vulnerabilities present in Java applications. For the first analysis, we conduct an exploratory large-scale study by running 256515 experiments in which we vary the versions of libraries for each of the 19 publicly available exploits. Such attacks rely on a combination of gadgets present in one or multiple Java libraries. A gadget is a method which is using objects or fields that can be attacker-controlled. Our goal is to precisely identify library versions containing gadgets and to understand how gadgets have been introduced and how they have been patched. We observe that the modification of one innocent-looking detail in a class – such as making it public – can already introduce a gadget. Furthermore, we noticed that among the studied libraries, 37.5% are not patched, leaving gadgets available for future attacks. For the second analysis, we manually analyze 104 deserialization vulnerabilities CVEs to understand how vulnerabilities are introduced and patched in real-life Java applications. Results indicate that the vulnerabilities are not always completely patched or that a workaround solution is proposed. With a workaround solution, applications are still vulnerable since the code itself is unchanged.

Using Mask R-CNN for Underwater Fish Instance Segmentation as Novel Objects: A Proof of Concept

Instance Segmentation in general deals with detecting, segmenting and classifying individual instances of objects in an image. Underwater instance segmentation methods often involve aquatic animals like fish as the things to be detected. In order to train deep learning models for instance segmentation in an underwater environment, rigorous human annotation in form of instance segmentation masks with labels is usually required, since the aquatic environment poses challenges due to dynamic background patterns and optical distortions.However, annotating instance segmentation masks on images is especially time- and cost-intensive compared to classification tasks.Here we show an unsupervised instance learning and segmentation approach that introduces a novel class, e.g., ""fish"" to a pre-trained Mask R-CNN model using its own detection and segmentation capabilities in underwater images.Our results demonstrate a robust detection and segmentation of underwater fish in aquaculture without the need for human annotations.This proof of concept shows that there is room for novel objects within trained instance segmentation models in the paradigm of supervised learning.

Multiple Instance Complementary Detection and Difficulty Evaluation for Weakly Supervised Object Detection in Remote Sensing Images

Weakly supervised object detection (WSOD) in remote sensing images (RSIs) has attracted lots of attention because it solely employs image-level labels to drive the model training. Most of the WSOD methods incline to mine salient object as positive instance, and the less salient objects are considered as negative instances, which will cause the problem of missing instances. In addition, the quantity of hard and easy instances is usually imbalanced, consequently, the cumulative loss of a large amount of easy instances dominates the training loss, which limits the upper bound of WSOD performance. To handle the first problem, a complementary detection network (CDN) is proposed, which consists of a complementary multiple instance detection network (CMIDN) and a complementary feature learning (CFL) module. The CDN can capture robust complementary information from two basic multiple instance detection networks (MIDNs) and mine more object instances. To handle the second problem, an instance difficulty evaluation metric named instance difficulty score (IDS) is proposed, which is employed as the weight of each instance in the training loss. Consequently, the hard instances will be assigned larger weights according to the IDS, which can improve the upper bound of WSOD performance. The ablation experiments demonstrate that our method significantly increases the baseline method by large margins, i.e. 23.6% (10.2%) mAP and 32.4% (13.1%) CorLoc gains on the NWPU VHR-10.v2 (DIOR) dataset. Our method obtains 58.1% (26.7%) mAP and 72.4% (47.9%) CorLoc on the NWPU VHR-10.v2 (DIOR) dataset, which achieves better performance compared with seven advanced WSOD methods.

Video Instance Segmentation by Instance Flow Assembly

Instance segmentation is a challenging task aiming at classifying and segmenting all object instances of specific classes. While two-stage box-based methods achieve top performances in the image domain, they cannot easily extend their superiority into the video domain. This is because they usually deal with features or images cropped from the detected bounding boxes without alignment, failing to capture pixel-level temporal consistency. We embrace the observation that bottom-up methods dealing with box-free features could offer accurate spacial correlations across frames, which can be fully utilized for object and pixel level tracking. We first propose our bottom-up framework equipped with a temporal context fusion module to better encode inter-frame correlations. Intra-frame cues for semantic segmentation and object localization are simultaneously extracted and reconstructed by corresponding decoders after a shared backbone. For efficient and robust tracking among instances, we introduce an instance-level correspondence across adjacent frames, which is represented by a center-to-center flow, termed as instance flow, to assemble messy dense temporal correspondences. Experiments demonstrate that the proposed method outperforms the state-of-the-art online methods (taking image-level input) on the challenging Youtube-VIS dataset [46].