Object Segmentation Algorithm Research Articles

GCNet: A Deep Learning Framework for Enhanced Grape Cluster Segmentation and Yield Estimation Incorporating Occluded Grape Detection with a Correction Factor for Indoor Experimentation

Object segmentation algorithms have heavily relied on deep learning techniques to estimate the count of grapes which is a strong indicator for the yield success of grapes. The issue with using object segmentation algorithms for grape analytics is that they are limited to counting only the visible grapes, thus omitting hidden grapes, which affect the true estimate of grape yield. Many grapes are occluded because of either the compactness of the grape bunch cluster or due to canopy interference. This introduces the need for models to be able to estimate the unseen berries to give a more accurate estimate of the grape yield by improving grape cluster segmentation. We propose the Grape Counting Network (GCNet), a novel framework for grape cluster segmentation, integrating deep learning techniques with correction factors to address challenges in indoor yield estimation. GCNet incorporates occlusion adjustments, enhancing segmentation accuracy even under conditions of foliage and cluster compactness, and setting new standards in agricultural indoor imaging analysis. This approach improves yield estimation accuracy, achieving a R² of 0.96 and reducing mean absolute error (MAE) by 10% compared to previous methods. We also propose a new dataset called GrapeSet which contains visible imagery of grape clusters imaged indoors, along with their ground truth mask, total grape count, and weight in grams. The proposed framework aims to encourage future research in determining which features of grapes can be leveraged to estimate the correct grape yield count, equip grape harvesters with the knowledge of early yield estimation, and produce accurate results in object segmentation algorithms for grape analytics.

Deep Learning for Weed Detection and Segmentation in Agricultural Crops Using Images Captured by an Unmanned Aerial Vehicle

Artificial Intelligence (AI) has changed how processes are developed, and decisions are made in the agricultural area replacing manual and repetitive processes with automated and more efficient ones. This study presents the application of deep learning techniques to detect and segment weeds in agricultural crops by applying models with different architectures in the analysis of images captured by an Unmanned Aerial Vehicle (UAV). This study contributes to the computer vision field by comparing the performance of the You Only Look Once (YOLOv8n, YOLOv8s, YOLOv8m, and YOLOv8l), Mask R-CNN (with framework Detectron2), and U-Net models, making public the dataset with aerial images of soybeans and beans. The models were trained using a dataset consisting of 3021 images, randomly divided into test, validation, and training sets, which were annotated, resized, and increased using the Roboflow application interface. Evaluation metrics were used, which included training efficiency (mAP50 and mAP50-90), precision, accuracy, and recall in the model’s evaluation and comparison. The YOLOv8s variant achieved higher performance with an mAP50 of 97%, precision of 99.7%, and recall of 99% when compared to the other models. The data from this manuscript show that deep learning models can generate efficient results for automatic weed detection when trained with a well-labeled and large set. Furthermore, this study demonstrated the great potential of using advanced object segmentation algorithms in detecting weeds in soybean and bean crops.

Lightweight video object segmentation: Integrating online knowledge distillation for fast segmentation

The typical shortcoming of STM (Space-Time Memory Network) mode video object segmentation algorithms is their high segmentation performance coupled with slow processing speeds, which poses challenges in meeting real-world application demands. In this work, we propose using an online knowledge distillation method to develop a lightweight video segmentation algorithm based on the STM mode, achieving fast segmentation while maintaining performance. Specifically, we utilize a novel adaptive learning rate to tackle the issue of inverse learning during distillation. Subsequently, we introduce a Smooth Block mechanism to reduce the impact of structural disparities between the teacher and student models on distillation outcomes. Moreover, to reduce the fitting difficulty of the student model on single-frame features, we design the Space-Time Feature Fusion (STFF) module to provide appearance and position priors for the feature fitting process of each frame. Finally, we employ a simple Discriminator module for adversarial training with the student model, to encourage the student model to learn the feature distribution of the teacher model. Extensive experiments show that our algorithm attains performance comparable to the current state-of-the-art on both DAVIS and YouTube datasets, despite running up to ×4 faster, with ×20 fewer parameters and ×30 fewer GFLOPS.

AIDCON: An Aerial Image Dataset and Benchmark for Construction Machinery

Applying deep learning algorithms in the construction industry holds tremendous potential for enhancing site management, safety, and efficiency. The development of such algorithms necessitates a comprehensive and diverse image dataset. This study introduces the Aerial Image Dataset for Construction (AIDCON), a novel aerial image collection containing 9563 construction machines across nine categories annotated at the pixel level, carrying critical value for researchers and professionals seeking to develop and refine object detection and segmentation algorithms across various construction projects. The study highlights the benefits of utilizing UAV-captured images by evaluating the performance of five cutting-edge deep learning algorithms—Mask R-CNN, Cascade Mask R-CNN, Mask Scoring R-CNN, Hybrid Task Cascade, and Pointrend—on the AIDCON dataset. It underscores the significance of clustering strategies for generating reliable and robust outcomes. The AIDCON dataset’s unique aerial perspective aids in reducing occlusions and provides comprehensive site overviews, facilitating better object positioning and segmentation. The findings presented in this paper have far-reaching implications for the construction industry, as they enhance construction site efficiency while setting the stage for future advancements in construction site monitoring and management utilizing remote sensing technologies.

A Curated Cell Life Imaging Dataset of Immune-enriched Pancreatic Cancer Organoids with Pre-trained AI Models

Tumor organoids are three-dimensional in vitro models which can recapitulate the complex mutational landscape and tissue architecture observed in cancer patients, providing a realistic model for testing novel therapies, including immunotherapies. A significant challenge in organoid research in oncology lies in developing efficient and reliable methods for segmenting organoid images, quantifying organoid growth, regression and response to treatments, as well as predicting the behavior of organoid systems. Up to now, a curated dataset of organoids co-cultured with immune cells is not available. To address this gap, we present a new public dataset, comprising both phase-contrast images of murine and patient-derived tumor organoids of one of the deadliest cancer types, the Pancreatic Ductal Adenocarcinoma, co-cultured with immune cells, and state-of-the-art algorithms for object detection and segmentation. Our dataset, OrganoIDNetData, encompassing 180 images with 33906 organoids, can be a potential common benchmark for different organoids segmentation protocols, moving beyond the current practice of training and testing these algorithms on isolated datasets.

Video object segmentation based on dynamic perception update and feature fusion

The current popular video object segmentation algorithms based on memory network indiscriminately update the frame information to the memory pool, fails to make reasonable use of the historical frame information, causing frame information redundancy in the memory pool, resulting in the increase of the computation amount. At the same time, the mask refinement method is relatively rough, resulting in blurred edges of the generated mask. To solve these problems, This paper proposes a video object segmentation algorithm based on dynamic perception update and feature fusion. In order to reasonably utilize the historical frame information, a dynamic perception update module is proposed to selectively update the segmentation frame mask. Meanwhile, a mask refinement module is established to enhance the detail information of the shallow features of the backbone network. This module uses a double kernels fusion block to fuse the different scale information of the features, and finally uses the Laplacian operator to sharpen the edges of the mask. The experimental results show that on the public datasets DAVIS2016, DAVIS2017 and YouTube-VOS18, the comprehensive performance of the algorithm in this paper reaches 86.9%, 79.3% and 71.6%, respectively, and the segmentation speed reaches 15FPS on the DAVIS2016 dataset. Compared with many mainstream algorithms in recent years, it has obvious advantages in performance.

A video object segmentation-based fish individual recognition method for underwater complex environments

Currently, aquaculture methods tend to combine scale and intelligence, which saves manpower and improves the survival rate of seafood at the same time. High-precision and high-efficiency fish individual recognition can provide key technical support for fish disease detection, feeding habits, body condition, etc. In the realm of intelligent aquaculture, it provides robust data support for precision fish farming. However, the current research methods for individual fish recognition struggle to maintain the network model's focus on the fish body in real marine underwater complex environments (e.g., environmental background interference such as coral reefs, overlap between fish bodies, light noise, etc.), leading to unsatisfactory recognition results. To this end, this paper proposes a method for fish individual recognition in underwater complex environments based on video object segmentation, which consists of three parts, including a fish individual segmentation detection module, a fish individual recognition module, and an all-in-one visualization module. The work adopts a combination of deep learning methods and video object segmentation algorithms to solve the problem of low attention and poor detection accuracy of fish individuals in real underwater complex environments, which effectively improves the accuracy and efficiency of fish individual recognition, and analyzes and discusses the comparison of recognition effects using different weights. The results of the simulation experiments show that the key metric Rank1 value of the method achieves more than 96% accuracy on the public datasets DlouFish, WideFish, and the Fish-seg dataset produced in this paper, and improves over the state-of-the-art methods for fish individual recognition by 2.23%, 1.33%, and 1.25%, respectively.

The application of Hyperpixel segmentation algorithm in Chinese painting

Chinese painting, as an important component of traditional Chinese art, not only expresses Chinese culture and history, but also contains rich artistic connotations. Therefore, extracting and analyzing the main objects of Chinese painting is of great significance. The research introduces a fusion deep network based super pixel segmentation algorithm for image segmentation, and utilizes user marked segmentation target objects and backgrounds. The maximum similarity region merging idea is adopted to extract the main subject objects in Chinese painting, thereby achieving a deep understanding of the work. The results show that the boundary recall and segmentation accuracy of the Chinese painting subject object segmentation algorithm based on super pixels on the BuptPainting dataset are as high as 94.38% and 98.06%, respectively, with an undersegmentation error rate of only 7.69%. Meanwhile, the average accuracy value, optimal data scale value, and optimal image scale value of this method on the BuptPainting dataset are as high as 0.657, 0.541, and 0.529, respectively. The method proposed by the research institute has significant advantages in segmenting the main objects of Chinese painting, providing new ideas and methods for image analysis and processing of modern Chinese painting, and helping to further explore the artistic charm and cultural connotations of Chinese painting.

Analysis of pig posture detection in group-housed pigs using deep learning-based mask scoring instance segmentation.

Pig posture is closely linked with livestock health and welfare. There has been significant interest among researchers in using deep learning techniques for pig posture detection. However, this task is challenging due to variations in image angles and times, as well as the presence of multiple pigs in a single image. In this study, we explore an object detection and segmentation algorithm based on instance segmentation scoring to detect different pig postures (sternal lying, lateral lying, walking, and sitting) and segment pig areas in group images, thereby enabling the identification of individual pig postures within a group. The algorithm combines a residual network with 50 layers and a feature pyramid network to extract feature maps from input images. These feature maps are then used to generate regions of interest (RoI) using a region candidate network. For each RoI, the algorithm performs regression to determine the location, classification, and segmentation of each pig posture. To address challenges such as missing targets and error detections among overlapping pigs in group housing, non-maximum suppression (NMS) is used with a threshold of 0.7. Through extensive hyperparameter analysis, a learning rate of 0.01, a batch size of 512, and 4 images per batch offer superior performance, with accuracy surpassing 96%. Similarly, the mean average precision (mAP) exceeds 83% for object detection and instance segmentation under these settings. Additionally, we compare the method with the faster R-CNN object detection model. Further, execution times on different processing units considering various hyperparameters and iterations have been analyzed.

Black-box Attack against Self-supervised Video Object Segmentation Models with Contrastive Loss

Deep learning models have been proven to be susceptible to malicious adversarial attacks, which manipulate input images to deceive the model into making erroneous decisions. Consequently, the threat posed to these models serves as a poignant reminder of the necessity to focus on the model security of object segmentation algorithms based on deep learning. However, the current landscape of research on adversarial attacks primarily centers around static images, resulting in a dearth of studies on adversarial attacks targeting Video Object Segmentation (VOS) models. Given that a majority of self-supervised VOS models rely on affinity matrices to learn feature representations of video sequences and achieve robust pixel correspondence, our investigation has delved into the impact of adversarial attacks on self-supervised VOS models. In response, we propose an innovative black-box attack method incorporating contrastive loss. This method induces segmentation errors in the model through perturbations in the feature space and the application of a pixel-level loss function. Diverging from conventional gradient-based attack techniques, we adopt an iterative black-box attack strategy that incorporates contrastive loss across the current frame, any two consecutive frames, and multiple frames. Through extensive experimentation conducted on the DAVIS 2016 and DAVIS 2017 datasets using three self-supervised VOS models and one unsupervised VOS model, we unequivocally demonstrate the potent attack efficiency of the black-box approach. Remarkably, theJ&Fmetric value experiences a significant decline of up to 50.08% post-attack.

Retracted: Design and Analysis of Interactive Multimedia Online Physical Education Platform considering Moving Object Segmentation Algorithm

Retracted: Design and Analysis of Interactive Multimedia Online Physical Education Platform considering Moving Object Segmentation Algorithm

A Multitask Benchmark Dataset for Satellite Video: Object Detection, Tracking, and Segmentation

Video satellites can continuously image large areas and provide dynamic, real-time monitoring of hotspots and objects. The intelligent processing and analysis of satellite video have become a research hotspot in the field of remote sensing. However, the lack of high-quality satellite video datasets limits the development of relevant object detection, object tracking, and object segmentation. In this paper, we build the largest scale satellite video dataset with the most task types supported and object categories, named Satellite Video Multi-Mission Benchmark (SAT-MTB). First, multi-task annotation of aircraft, ships, cars, trains, and their corresponding 14 categories of fine-grained objects in 249 satellite videos is performed based on horizontal bounding boxes (HBB), oriented bounding boxes (OBB), masks, which cover more than 50,000 frames and 1,033,511 annotated object instances. Then, we review the tasks of object detection, object tracking, and object segmentation based on satellite videos, providing a comprehensive overview of progress in related datasets and algorithm research. Finally, we establish the first public benchmark of multi-task algorithms for satellite video object detection, object tracking, and object segmentation, evaluating and analyzing the performance of a total of 47 representative algorithms under different tasks on the constructed dataset. The proposed SAT-MTB will significantly advance research in intelligent processing and analysis of satellite video and related applications.

Slender Flexible Object Segmentation Based on Object Correlation Module and Loss Function Optimization

Slender flexible objects are ubiquitous in real-world circumstances. The existing object detection and segmentation algorithms have achieved high accuracy and speed in rigid objects, but the detection effect of slender flexible objects is not ideal. Extreme aspect ratios and dynamically changeable geometric appearance characterize slender flexible objects, to the extent that it is difficult to locate them in instance segmentation. In this paper, a new instance segmentation method based on the object correlation module and loss function optimization is proposed for the detection of slender flexible objects. In order to achieve more accurate anchor box positioning, a GIoU bounding-box regression loss function is selected to overcome the problem of inconsistency between training objectives and assessment indicators. Furthermore, due to the Mask Scoring R-CNN detection network ignoring the relationships between objects, the object correlation module is proposed to achieve end-to-end learning and modeling of the correlation features between all objects in the image to improve slender flexible objects detection accuracy. The results of the experiments on the self-built flexible object dataset for the power grid operation sites demonstrate that the method presented in this research can efficiently recognize and segment slender flexible objects, with a detection accuracy of 44.8%. The ablation experiment also shows that the addition object correlation module and the revised bounding-box regression loss function are both effective and can enhance slender flexible object detection accuracy by 1.2% and 0.5%, respectively. The proposed instance segmentation method considers the correlation characteristics between objects and improves the bounding-box regression loss function to increase the segmentation accuracy of slender flexible objects.

Lightweight video object segmentation algorithm based on adaptive weight update

Lightweight video object segmentation algorithm based on adaptive weight update

Object Segmentation by Spraying Robot Based on Multi-Layer Perceptron

The vision system provides an important way for construction robots to obtain the type and spatial location information of the object. The characteristics of the construction environment, construction object, and robot structure are jointly examined in this paper to propose an approach of object segmentation by spraying the robot based on multi-layer perceptron. Firstly, the hand-eye system experimental platform is built through establishing the mathematical model of the system and calibrating the parameters of the model. Secondly, effort is made to carry out research on image preprocessing algorithms and related experiments, and compare the effects of different binocular stereo-matching algorithms in the actual engineering environment. Finally, research and an experiment are conducted to identify the applicability and effect of the depth image object segmentation algorithm based on multi-layer perceptron. The experimental results prove that the application of multi-layer perceptron to object segmentation by spraying robots can meet the requirement on solution accuracy and is suitable for the object segmentation of complex projects in real life. This approach not only overcomes the shortcomings of the existing recognition methods that are poor in accuracy and difficult to be used widely, but also provides basic data for the subsequent three-dimensional reconstruction, thus making a significant contribution to the research of image processing by spraying robots.

Autonomous detection and sorting of litter using deep learning and soft robotic grippers.

Road infrastructure is one of the most vital assets of any country. Keeping the road infrastructure clean and unpolluted is important for ensuring road safety and reducing environmental risk. However, roadside litter picking is an extremely laborious, expensive, monotonous and hazardous task. Automating the process would save taxpayers money and reduce the risk for road users and the maintenance crew. This work presents LitterBot, an autonomous robotic system capable of detecting, localizing and classifying common roadside litter. We use a learning-based object detection and segmentation algorithm trained on the TACO dataset for identifying and classifying garbage. We develop a robust modular manipulation framework by using soft robotic grippers and a real-time visual-servoing strategy. This enables the manipulator to pick up objects of variable sizes and shapes even in dynamic environments. The robot achieves greater than 80% classified picking and binning success rates for all experiments; which was validated on a wide variety of test litter objects in static single and cluttered configurations and with dynamically moving test objects. Our results showcase how a deep model trained on an online dataset can be deployed in real-world applications with high accuracy by the appropriate design of a control framework around it.

A Discriminative Single-Shot Segmentation Network for Visual Object Tracking.

Template-based discriminative trackers are currently the dominant tracking paradigm due to their robustness, but are restricted to bounding box tracking and a limited range of transformation models, which reduces their localization accuracy. We propose a discriminative single-shot segmentation tracker - D3S 2, which narrows the gap between visual object tracking and video object segmentation. A single-shot network applies two target models with complementary geometric properties, one invariant to a broad range of transformations, including non-rigid deformations, the other assuming a rigid object to simultaneously achieve robust online target segmentation. The overall tracking reliability is further increased by decoupling the object and feature scale estimation. Without per-dataset finetuning, and trained only for segmentation as the primary output, D3S 2 outperforms all published trackers on the recent short-term tracking benchmark VOT2020 and performs very close to the state-of-the-art trackers on the GOT-10k, TrackingNet, OTB100 and LaSoT. D3S 2 outperforms the leading segmentation tracker SiamMask on video object segmentation benchmarks and performs on par with top video object segmentation algorithms.

Intelligent Object Tracking with an Automatic Image Zoom Algorithm for a Camera Sensing Surveillance System.

Current surveillance systems frequently use fixed-angle cameras and record a feed from those cameras. There are several disadvantages to such systems, including a low resolution for far away objects, a limited frame range and wasted disk space. This paper presents a novel algorithm for automatically detecting, tracking and zooming in on active targets. The object tracking system is connected to a camera that has a 360° horizontal and 90° vertical movement range. The combination of tracking, movement identification and zoom means that the system is able to effectively improve the resolution of small or distant objects. The object detection system allows for the disk space to be conserved as the system ceases recording when no valid targets are detected. Using an adaptive object segmentation algorithm, it is possible to detect the shape of moving objects efficiently. When processing multiple targets, each target is assigned a color and is treated separately. The tracking algorithm is able to adapt to targets moving at different speeds and is able to control the camera according to a predictive formula to prevent the loss of image quality due to camera trail. In the test environment, the zoom can sufficiently lock onto the head of a moving human; however, simultaneous tracking and zooming occasionally results in a failure to track. If this system is deployed with a facial recognition algorithm, the recognition accuracy can be effectively improved.

Обнаружение и отслеживание объектов на основе карты внимания для БПЛА

Object detection and tracking is one of the most important areas of computer vision, as well as applications in unmanned aerial vehicles (UAVs). However, due to the ultra-maneuverability of the UAV mobile platform, the image in the air is usually blurry and has a low signal-to-noise ratio, which is associated with the working environment. To solve the problem of detecting ground objects on the UAV platform, traditional methods of pattern recognition and classification are often ineffective. The visual attention model is a kind of bionic vision model with good stability. In the article for the detection of ground objects using UAVs, a method based on the visual saliency model is proposed, applied to a complex terrain background for object detection. To quickly isolate terrestrial features in aerial photographs, the extracted gradient function is applied to detect an area of visual significance, and then the target image is extracted using a segmentation algorithm. The subject of the study is the method and algorithm for detecting ground objects on the map of the underlying surface using the visual saliency model and object segmentation. The object of the study is a set of video sequences of terrain maps with different terrain. The novelty of the work is an algorithm that allows detecting ground targets based on an attention map using an object segmentation algorithm. A new method for segmenting objects in a video sequence is proposed. Experimental studies were carried out on the basis of video sequences of the map of the underlying surface with different backgrounds and an analysis of the results obtained was carried out. The results obtained make it possible to identify objects in the region of interest. As a result of solving the formulated tasks, the following conclusions can be drawn: An algorithm for detecting ground objects based on an saliency map has been developed. An algorithm for segmentation of ground objects based on a gradient map has been developed. An analysis of the results of the study showed that the proposed algorithm makes it possible to detect and highlight ground objects on a map of terrain with different terrain.

Pavement crack measurement based on aerial 3D reconstruction and learning-based segmentation method

Rapid inspection of urban road cracks is vital to maintain traffic smoothness and ensure traffic safety. A rapid pavement crack inspection method uses low-altitude aerial images captured by an unmanned aerial system (UAS) and deep-learning aided 3D reconstruction, and a learning-based object segmentation algorithm is proposed to measure road cracks automatically. The contributions include: (a) An efficient 3D reconstruction method for low-altitude aerial images captured by a UAS is proposed, which applies an instance segmentation network to segment road targets from raw images with complex backgrounds first and then performs structure from motion to reconstruct a large-scale road orthophoto from a large number of aerial images. (b) To detect cracks from the reconstructed large-size road orthophoto, a sliding window algorithm and U-Net model optimized with a transformer structure are used to automatically identify and segment the cracks from the orthophoto at the pixel level. Then, a connected domain feature analysis method is used to measure the road crack length. The proposed method is applied to detection of road cracks in a 1.5 km2 area of a city. The results show that the proposed method can effectively and accurately detect cracks and measure the length of cracks in the 4-km-long road, which proves the practicality of the proposed method.