Object Segmentation Algorithm Research Articles

A Curated Cell Life Imaging Dataset of Immune-enriched Pancreatic Cancer Organoids with Pre-trained AI Models

Tumor organoids are three-dimensional in vitro models which can recapitulate the complex mutational landscape and tissue architecture observed in cancer patients, providing a realistic model for testing novel therapies, including immunotherapies. A significant challenge in organoid research in oncology lies in developing efficient and reliable methods for segmenting organoid images, quantifying organoid growth, regression and response to treatments, as well as predicting the behavior of organoid systems. Up to now, a curated dataset of organoids co-cultured with immune cells is not available. To address this gap, we present a new public dataset, comprising both phase-contrast images of murine and patient-derived tumor organoids of one of the deadliest cancer types, the Pancreatic Ductal Adenocarcinoma, co-cultured with immune cells, and state-of-the-art algorithms for object detection and segmentation. Our dataset, OrganoIDNetData, encompassing 180 images with 33906 organoids, can be a potential common benchmark for different organoids segmentation protocols, moving beyond the current practice of training and testing these algorithms on isolated datasets.

Video object segmentation based on dynamic perception update and feature fusion

The current popular video object segmentation algorithms based on memory network indiscriminately update the frame information to the memory pool, fails to make reasonable use of the historical frame information, causing frame information redundancy in the memory pool, resulting in the increase of the computation amount. At the same time, the mask refinement method is relatively rough, resulting in blurred edges of the generated mask. To solve these problems, This paper proposes a video object segmentation algorithm based on dynamic perception update and feature fusion. In order to reasonably utilize the historical frame information, a dynamic perception update module is proposed to selectively update the segmentation frame mask. Meanwhile, a mask refinement module is established to enhance the detail information of the shallow features of the backbone network. This module uses a double kernels fusion block to fuse the different scale information of the features, and finally uses the Laplacian operator to sharpen the edges of the mask. The experimental results show that on the public datasets DAVIS2016, DAVIS2017 and YouTube-VOS18, the comprehensive performance of the algorithm in this paper reaches 86.9%, 79.3% and 71.6%, respectively, and the segmentation speed reaches 15FPS on the DAVIS2016 dataset. Compared with many mainstream algorithms in recent years, it has obvious advantages in performance.

A video object segmentation-based fish individual recognition method for underwater complex environments

Currently, aquaculture methods tend to combine scale and intelligence, which saves manpower and improves the survival rate of seafood at the same time. High-precision and high-efficiency fish individual recognition can provide key technical support for fish disease detection, feeding habits, body condition, etc. In the realm of intelligent aquaculture, it provides robust data support for precision fish farming. However, the current research methods for individual fish recognition struggle to maintain the network model's focus on the fish body in real marine underwater complex environments (e.g., environmental background interference such as coral reefs, overlap between fish bodies, light noise, etc.), leading to unsatisfactory recognition results. To this end, this paper proposes a method for fish individual recognition in underwater complex environments based on video object segmentation, which consists of three parts, including a fish individual segmentation detection module, a fish individual recognition module, and an all-in-one visualization module. The work adopts a combination of deep learning methods and video object segmentation algorithms to solve the problem of low attention and poor detection accuracy of fish individuals in real underwater complex environments, which effectively improves the accuracy and efficiency of fish individual recognition, and analyzes and discusses the comparison of recognition effects using different weights. The results of the simulation experiments show that the key metric Rank1 value of the method achieves more than 96% accuracy on the public datasets DlouFish, WideFish, and the Fish-seg dataset produced in this paper, and improves over the state-of-the-art methods for fish individual recognition by 2.23%, 1.33%, and 1.25%, respectively.

The application of Hyperpixel segmentation algorithm in Chinese painting

Chinese painting, as an important component of traditional Chinese art, not only expresses Chinese culture and history, but also contains rich artistic connotations. Therefore, extracting and analyzing the main objects of Chinese painting is of great significance. The research introduces a fusion deep network based super pixel segmentation algorithm for image segmentation, and utilizes user marked segmentation target objects and backgrounds. The maximum similarity region merging idea is adopted to extract the main subject objects in Chinese painting, thereby achieving a deep understanding of the work. The results show that the boundary recall and segmentation accuracy of the Chinese painting subject object segmentation algorithm based on super pixels on the BuptPainting dataset are as high as 94.38% and 98.06%, respectively, with an undersegmentation error rate of only 7.69%. Meanwhile, the average accuracy value, optimal data scale value, and optimal image scale value of this method on the BuptPainting dataset are as high as 0.657, 0.541, and 0.529, respectively. The method proposed by the research institute has significant advantages in segmenting the main objects of Chinese painting, providing new ideas and methods for image analysis and processing of modern Chinese painting, and helping to further explore the artistic charm and cultural connotations of Chinese painting.

Analysis of pig posture detection in group-housed pigs using deep learning-based mask scoring instance segmentation.

Pig posture is closely linked with livestock health and welfare. There has been significant interest among researchers in using deep learning techniques for pig posture detection. However, this task is challenging due to variations in image angles and times, as well as the presence of multiple pigs in a single image. In this study, we explore an object detection and segmentation algorithm based on instance segmentation scoring to detect different pig postures (sternal lying, lateral lying, walking, and sitting) and segment pig areas in group images, thereby enabling the identification of individual pig postures within a group. The algorithm combines a residual network with 50 layers and a feature pyramid network to extract feature maps from input images. These feature maps are then used to generate regions of interest (RoI) using a region candidate network. For each RoI, the algorithm performs regression to determine the location, classification, and segmentation of each pig posture. To address challenges such as missing targets and error detections among overlapping pigs in group housing, non-maximum suppression (NMS) is used with a threshold of 0.7. Through extensive hyperparameter analysis, a learning rate of 0.01, a batch size of 512, and 4 images per batch offer superior performance, with accuracy surpassing 96%. Similarly, the mean average precision (mAP) exceeds 83% for object detection and instance segmentation under these settings. Additionally, we compare the method with the faster R-CNN object detection model. Further, execution times on different processing units considering various hyperparameters and iterations have been analyzed.

Black-box Attack against Self-supervised Video Object Segmentation Models with Contrastive Loss

Deep learning models have been proven to be susceptible to malicious adversarial attacks, which manipulate input images to deceive the model into making erroneous decisions. Consequently, the threat posed to these models serves as a poignant reminder of the necessity to focus on the model security of object segmentation algorithms based on deep learning. However, the current landscape of research on adversarial attacks primarily centers around static images, resulting in a dearth of studies on adversarial attacks targeting Video Object Segmentation (VOS) models. Given that a majority of self-supervised VOS models rely on affinity matrices to learn feature representations of video sequences and achieve robust pixel correspondence, our investigation has delved into the impact of adversarial attacks on self-supervised VOS models. In response, we propose an innovative black-box attack method incorporating contrastive loss. This method induces segmentation errors in the model through perturbations in the feature space and the application of a pixel-level loss function. Diverging from conventional gradient-based attack techniques, we adopt an iterative black-box attack strategy that incorporates contrastive loss across the current frame, any two consecutive frames, and multiple frames. Through extensive experimentation conducted on the DAVIS 2016 and DAVIS 2017 datasets using three self-supervised VOS models and one unsupervised VOS model, we unequivocally demonstrate the potent attack efficiency of the black-box approach. Remarkably, the J&F metric value experiences a significant decline of up to 50.08% post-attack.

Retracted: Design and Analysis of Interactive Multimedia Online Physical Education Platform considering Moving Object Segmentation Algorithm

Retracted: Design and Analysis of Interactive Multimedia Online Physical Education Platform considering Moving Object Segmentation Algorithm

A Multitask Benchmark Dataset for Satellite Video: Object Detection, Tracking, and Segmentation

Video satellites can continuously image large areas and provide dynamic, real-time monitoring of hotspots and objects. The intelligent processing and analysis of satellite video have become a research hotspot in the field of remote sensing. However, the lack of high-quality satellite video datasets limits the development of relevant object detection, object tracking, and object segmentation. In this paper, we build the largest scale satellite video dataset with the most task types supported and object categories, named Satellite Video Multi-Mission Benchmark (SAT-MTB). First, multi-task annotation of aircraft, ships, cars, trains, and their corresponding 14 categories of fine-grained objects in 249 satellite videos is performed based on horizontal bounding boxes (HBB), oriented bounding boxes (OBB), masks, which cover more than 50,000 frames and 1,033,511 annotated object instances. Then, we review the tasks of object detection, object tracking, and object segmentation based on satellite videos, providing a comprehensive overview of progress in related datasets and algorithm research. Finally, we establish the first public benchmark of multi-task algorithms for satellite video object detection, object tracking, and object segmentation, evaluating and analyzing the performance of a total of 47 representative algorithms under different tasks on the constructed dataset. The proposed SAT-MTB will significantly advance research in intelligent processing and analysis of satellite video and related applications.

Slender Flexible Object Segmentation Based on Object Correlation Module and Loss Function Optimization

Slender flexible objects are ubiquitous in real-world circumstances. The existing object detection and segmentation algorithms have achieved high accuracy and speed in rigid objects, but the detection effect of slender flexible objects is not ideal. Extreme aspect ratios and dynamically changeable geometric appearance characterize slender flexible objects, to the extent that it is difficult to locate them in instance segmentation. In this paper, a new instance segmentation method based on the object correlation module and loss function optimization is proposed for the detection of slender flexible objects. In order to achieve more accurate anchor box positioning, a GIoU bounding-box regression loss function is selected to overcome the problem of inconsistency between training objectives and assessment indicators. Furthermore, due to the Mask Scoring R-CNN detection network ignoring the relationships between objects, the object correlation module is proposed to achieve end-to-end learning and modeling of the correlation features between all objects in the image to improve slender flexible objects detection accuracy. The results of the experiments on the self-built flexible object dataset for the power grid operation sites demonstrate that the method presented in this research can efficiently recognize and segment slender flexible objects, with a detection accuracy of 44.8%. The ablation experiment also shows that the addition object correlation module and the revised bounding-box regression loss function are both effective and can enhance slender flexible object detection accuracy by 1.2% and 0.5%, respectively. The proposed instance segmentation method considers the correlation characteristics between objects and improves the bounding-box regression loss function to increase the segmentation accuracy of slender flexible objects.

Lightweight video object segmentation algorithm based on adaptive weight update

Lightweight video object segmentation algorithm based on adaptive weight update

Object Segmentation by Spraying Robot Based on Multi-Layer Perceptron

The vision system provides an important way for construction robots to obtain the type and spatial location information of the object. The characteristics of the construction environment, construction object, and robot structure are jointly examined in this paper to propose an approach of object segmentation by spraying the robot based on multi-layer perceptron. Firstly, the hand-eye system experimental platform is built through establishing the mathematical model of the system and calibrating the parameters of the model. Secondly, effort is made to carry out research on image preprocessing algorithms and related experiments, and compare the effects of different binocular stereo-matching algorithms in the actual engineering environment. Finally, research and an experiment are conducted to identify the applicability and effect of the depth image object segmentation algorithm based on multi-layer perceptron. The experimental results prove that the application of multi-layer perceptron to object segmentation by spraying robots can meet the requirement on solution accuracy and is suitable for the object segmentation of complex projects in real life. This approach not only overcomes the shortcomings of the existing recognition methods that are poor in accuracy and difficult to be used widely, but also provides basic data for the subsequent three-dimensional reconstruction, thus making a significant contribution to the research of image processing by spraying robots.

Autonomous detection and sorting of litter using deep learning and soft robotic grippers.

Road infrastructure is one of the most vital assets of any country. Keeping the road infrastructure clean and unpolluted is important for ensuring road safety and reducing environmental risk. However, roadside litter picking is an extremely laborious, expensive, monotonous and hazardous task. Automating the process would save taxpayers money and reduce the risk for road users and the maintenance crew. This work presents LitterBot, an autonomous robotic system capable of detecting, localizing and classifying common roadside litter. We use a learning-based object detection and segmentation algorithm trained on the TACO dataset for identifying and classifying garbage. We develop a robust modular manipulation framework by using soft robotic grippers and a real-time visual-servoing strategy. This enables the manipulator to pick up objects of variable sizes and shapes even in dynamic environments. The robot achieves greater than 80% classified picking and binning success rates for all experiments; which was validated on a wide variety of test litter objects in static single and cluttered configurations and with dynamically moving test objects. Our results showcase how a deep model trained on an online dataset can be deployed in real-world applications with high accuracy by the appropriate design of a control framework around it.

A Discriminative Single-Shot Segmentation Network for Visual Object Tracking.

Template-based discriminative trackers are currently the dominant tracking paradigm due to their robustness, but are restricted to bounding box tracking and a limited range of transformation models, which reduces their localization accuracy. We propose a discriminative single-shot segmentation tracker - D3S 2, which narrows the gap between visual object tracking and video object segmentation. A single-shot network applies two target models with complementary geometric properties, one invariant to a broad range of transformations, including non-rigid deformations, the other assuming a rigid object to simultaneously achieve robust online target segmentation. The overall tracking reliability is further increased by decoupling the object and feature scale estimation. Without per-dataset finetuning, and trained only for segmentation as the primary output, D3S 2 outperforms all published trackers on the recent short-term tracking benchmark VOT2020 and performs very close to the state-of-the-art trackers on the GOT-10k, TrackingNet, OTB100 and LaSoT. D3S 2 outperforms the leading segmentation tracker SiamMask on video object segmentation benchmarks and performs on par with top video object segmentation algorithms.

Intelligent Object Tracking with an Automatic Image Zoom Algorithm for a Camera Sensing Surveillance System.

Current surveillance systems frequently use fixed-angle cameras and record a feed from those cameras. There are several disadvantages to such systems, including a low resolution for far away objects, a limited frame range and wasted disk space. This paper presents a novel algorithm for automatically detecting, tracking and zooming in on active targets. The object tracking system is connected to a camera that has a 360° horizontal and 90° vertical movement range. The combination of tracking, movement identification and zoom means that the system is able to effectively improve the resolution of small or distant objects. The object detection system allows for the disk space to be conserved as the system ceases recording when no valid targets are detected. Using an adaptive object segmentation algorithm, it is possible to detect the shape of moving objects efficiently. When processing multiple targets, each target is assigned a color and is treated separately. The tracking algorithm is able to adapt to targets moving at different speeds and is able to control the camera according to a predictive formula to prevent the loss of image quality due to camera trail. In the test environment, the zoom can sufficiently lock onto the head of a moving human; however, simultaneous tracking and zooming occasionally results in a failure to track. If this system is deployed with a facial recognition algorithm, the recognition accuracy can be effectively improved.

Обнаружение и отслеживание объектов на основе карты внимания для БПЛА

Object detection and tracking is one of the most important areas of computer vision, as well as applications in unmanned aerial vehicles (UAVs). However, due to the ultra-maneuverability of the UAV mobile platform, the image in the air is usually blurry and has a low signal-to-noise ratio, which is associated with the working environment. To solve the problem of detecting ground objects on the UAV platform, traditional methods of pattern recognition and classification are often ineffective. The visual attention model is a kind of bionic vision model with good stability. In the article for the detection of ground objects using UAVs, a method based on the visual saliency model is proposed, applied to a complex terrain background for object detection. To quickly isolate terrestrial features in aerial photographs, the extracted gradient function is applied to detect an area of visual significance, and then the target image is extracted using a segmentation algorithm. The subject of the study is the method and algorithm for detecting ground objects on the map of the underlying surface using the visual saliency model and object segmentation. The object of the study is a set of video sequences of terrain maps with different terrain. The novelty of the work is an algorithm that allows detecting ground targets based on an attention map using an object segmentation algorithm. A new method for segmenting objects in a video sequence is proposed. Experimental studies were carried out on the basis of video sequences of the map of the underlying surface with different backgrounds and an analysis of the results obtained was carried out. The results obtained make it possible to identify objects in the region of interest. As a result of solving the formulated tasks, the following conclusions can be drawn: An algorithm for detecting ground objects based on an saliency map has been developed. An algorithm for segmentation of ground objects based on a gradient map has been developed. An analysis of the results of the study showed that the proposed algorithm makes it possible to detect and highlight ground objects on a map of terrain with different terrain.

Pavement crack measurement based on aerial 3D reconstruction and learning-based segmentation method

Rapid inspection of urban road cracks is vital to maintain traffic smoothness and ensure traffic safety. A rapid pavement crack inspection method uses low-altitude aerial images captured by an unmanned aerial system (UAS) and deep-learning aided 3D reconstruction, and a learning-based object segmentation algorithm is proposed to measure road cracks automatically. The contributions include: (a) An efficient 3D reconstruction method for low-altitude aerial images captured by a UAS is proposed, which applies an instance segmentation network to segment road targets from raw images with complex backgrounds first and then performs structure from motion to reconstruct a large-scale road orthophoto from a large number of aerial images. (b) To detect cracks from the reconstructed large-size road orthophoto, a sliding window algorithm and U-Net model optimized with a transformer structure are used to automatically identify and segment the cracks from the orthophoto at the pixel level. Then, a connected domain feature analysis method is used to measure the road crack length. The proposed method is applied to detection of road cracks in a 1.5 km2 area of a city. The results show that the proposed method can effectively and accurately detect cracks and measure the length of cracks in the 4-km-long road, which proves the practicality of the proposed method.

LiDAR Intensity Completion: Fully Exploiting the Message from LiDAR Sensors.

Light Detection and Ranging (LiDAR) systems are novel sensors that provide robust distance and reflection strength by active pulsed laser beams. They have significant advantages over visual cameras by providing active depth and intensity measurements that are robust to ambient illumination. However, the systemsstill pay limited attention to intensity measurements since the output intensity maps of LiDAR sensors are different from conventional cameras and are too sparse. In this work, we propose exploiting the information from both intensity and depth measurements simultaneously to complete the LiDAR intensity maps. With the completed intensity maps, mature computer vision techniques can work well on the LiDAR data without any specific adjustment. We propose an end-to-end convolutional neural network named LiDAR-Net to jointly complete the sparse intensity and depth measurements by exploiting their correlations. For network training, an intensity fusion method is proposed to generate the ground truth. Experiment results indicate that intensity–depth fusion can benefit the task and improve performance. We further apply an off-the-shelf object (lane) segmentation algorithm to the completed intensity maps, which delivers consistent robust to ambient illumination performance. We believe that the intensity completion method allows LiDAR sensors to cope with a broader range of practice applications.

Learning Target-Oriented Push-Grasping Synergy in Clutter With Action Space Decoupling

We explore a method for grasping novel target objects through push-grasping synergy in a cluttered environment without using object detection and segmentation algorithms. The target information is represented by a color heightmap of the target object. The agent needs to implicitly find the corresponding object in the cluttered scene according to the target information and gives action instructions. We propose an action space decoupling framework to efficiently address the problem of policy learning in large state spaces, which predicts the grasping position and the grasping angle separately. Our system needs to learn two policy networks: position net and angle net. The position net infers the appropriate grasping position and the starting position of the pushing motion, and the angle net predicts the grasping angle based on the grasping position. In addition, the angle net is also a coordinator to determine whether the current state should perform the grasping action or the pushing action. A series of experiments show that the proposed method can quickly learn complex push-grasping coordinated policies in a cluttered environment and the task success rate and the grasping efficiency are greatly improved compared to the baseline methods and it has good generalization ability for novel objects. Furthermore, our method can be transferred to the real world without fine-tuning.

Sequential Clique Optimization for Unsupervised and Weakly Supervised Video Object Segmentation

A novel video object segmentation algorithm, which segments out multiple objects in a video sequence in unsupervised or weakly supervised manners, is proposed in this work. First, we match visually important object instances to construct salient object tracks through a video sequence without any user supervision. We formulate this matching process as the problem to find maximal weight cliques in a complete k-partite graph and develop the sequential clique optimization algorithm to determine the cliques efficiently. Then, we convert the resultant salient object tracks into object segmentation results and refine them based on Markov random field optimization. Second, we adapt the sequential clique optimization algorithm to perform weakly supervised video object segmentation. To this end, we develop a sparse-to-dense network to convert the point cliques into segmentation results. The experimental results demonstrate that the proposed algorithm provides comparable or better performances than recent state-of-the-art VOS algorithms.

Exploring the Semi-Supervised Video Object Segmentation Problem from a Cyclic Perspective

Modern video object segmentation (VOS) algorithms have achieved remarkably high performance in a sequential processing order, while most of currently prevailing pipelines still show some obvious inadequacy like accumulative error, unknown robustness or lack of proper interpretation tools. In this paper, we place the semi-supervised video object segmentation problem into a cyclic workflow and find the defects above can be collectively addressed via the inherent cyclic property of semi-supervised VOS systems. Firstly, a cyclic mechanism incorporated to the standard sequential flow can produce more consistent representations for pixel-wise correspondance. Relying on the accurate reference mask in the starting frame, we show that the error propagation problem can be mitigated. Next, a simple gradient correction module, which naturally extends the offline cyclic pipeline to an online manner, can highlight the high-frequent and detailed part of results to further improve the segmentation quality while keeping feasible computation cost. Meanwhile such correction can protect the network from severe performance degration resulted from interference signals. Finally we develop cycle effective receptive field (cycle-ERF) based on gradient correction process to provide a new perspective into analyzing object-specific regions of interests. We conduct comprehensive comparison and detailed analysis on challenging benchmarks of DAVIS16, DAVIS17 and Youtube-VOS, demonstrating that the cyclic mechanism is helpful to enhance segmentation quality, improve the robustness of VOS systems, and further provide qualitative comparison and interpretation on how different VOS algorithms work. The code of this project can be found at https://github.com/lyxok1/STM-Training.