Object Segmentation Research Articles

Using Hough transform to define track boundaries as part of machine vision by train traffic-tracking hardware and software systems

Aim. In the context of machine vision, the problem of detected object boundary definition is usually solved using semantic segmentation that requires a high computational resource. Its application increases the complexity and the cost of the implemented hardware and software systems. This paper proposes an alternative method for defining the boundary of a segmented object, i.e., a railway track, for a train traffic tracking system. Methods. Since a railway track in an image can be represented by an n-th order polynomial, it is suggested to solve the problem of track boundary detection by using approximations in the form of straight lines. It is suggested using the Hough transform to detect straight lines. The former’s parametric space will be arranged in accordance with the problem to be solved. Conclusions. The proposed approximation will allow abandoning semantic segmentation and reduce the computational complexity and load.

A novel multiple moving objects recognition and segmentation based on dense optical flow and K-means clustering

A novel multiple moving objects recognition and segmentation based on dense optical flow and K-means clustering

AnnotateXR: An Extended Reality Workflow for Automating Data Annotation to Support Computer Vision Applications

Abstract Computer vision (CV) algorithms require large annotated datasets that are often labor-intensive and expensive to create. We propose AnnotateXR, an extended reality (XR) workflow to collect various high-fidelity data and auto-annotate it in a single demonstration. AnnotateXR allows users to align virtual models over physical objects, tracked with six degrees-of-freedom (6DOF) sensors. AnnotateXR utilizes a hand tracking capable XR head-mounted display coupled with 6DOF information and collision detection to enable algorithmic segmentation of different actions in videos through its digital twin. The virtual–physical mapping provides a tight bounding volume to generate semantic segmentation masks for the captured image data. Alongside supporting object and action segmentation, we also support other dimensions of annotation required by modern CV, such as human–object, object–object, and rich 3D recordings, all with a single demonstration. Our user study shows AnnotateXR produced over 112,000 annotated data points in 67 min.

Generation of Rule-Based Explanations of CNN Classifiers Using Regional Features

Although Deep Learning networks generally outperform traditional machine learning approaches based on tailored features, they often lack explainability. To address this issue, numerous methods have been proposed, particularly for image-related tasks such as image classification or object segmentation. These methods generate a heatmap that visually explains the classification problem by identifying the most important regions for the classifier. However, these explanations remain purely visual. To overcome this limitation, we introduce a novel CNN explainability method that identifies the most relevant regions in an image and generates a decision tree based on meaningful regional features, providing a rule-based explanation of the classification model. We evaluated the proposed method on a synthetic blob’s dataset and subsequently applied it to two cell image classification datasets with healthy and pathological patterns.

Semantic redundancy-aware implicit neural compression for multidimensional biomedical image data

The surge in advanced imaging techniques has generated vast biomedical image data with diverse dimensions in space, time and spectrum, posing big challenges to conventional compression techniques in image storage, transmission, and sharing. Here, we propose an intelligent image compression approach with the first-proved semantic redundancy of biomedical data in the implicit neural function domain. This Semantic redundancy based Implicit Neural Compression guided with Saliency map (SINCS) can notably improve the compression efficiency for arbitrary-dimensional image data in terms of compression ratio and fidelity. Moreover, with weight transfer and residual entropy coding strategies, it shows improved compression speed while maintaining high quality. SINCS yields high quality compression with over 2000-fold compression ratio on 2D, 2D-T, 3D, 4D biomedical images of diverse targets ranging from single virus to entire human organs, and ensures reliable downstream tasks, such as object segmentation and quantitative analyses, to be conducted at high efficiency.

Scalable Video Object Segmentation With Identification Mechanism.

This paper delves into the challenges of achieving scalable and effective multi-object modeling for semi-supervised Video Object Segmentation (VOS). Previous VOS methods decode features with a single positive object, limiting the learning of multi-object representation as they must match and segment each target separately under multi-object scenarios. Additionally, earlier techniques catered to specific application objectives and lacked the flexibility to fulfill different speed-accuracy requirements. To address these problems, we present two innovative approaches, Associating Objects with Transformers (AOT) and Associating Objects with Scalable Transformers (AOST). In pursuing effective multi-object modeling, AOT introduces the IDentification (ID) mechanism to allocate each object a unique identity. This approach enables the network to model the associations among all objects simultaneously, thus facilitating the tracking and segmentation of objects in a single network pass. To address the challenge of inflexible deployment, AOST further integrates scalable long short-term transformers that incorporate scalable supervision and layer-wise ID-based attention. This enables online architecture scalability in VOS for the first time and overcomes ID embeddings' representation limitations. Given the absence of a benchmark for VOS involving densely multi-object annotations, we propose a challenging Video Object Segmentation in the Wild (VOSW) benchmark to validate our approaches. We evaluated various AOT and AOST variants using extensive experiments across VOSW and five commonly used VOS benchmarks, including YouTube-VOS 2018 & 2019 Val, DAVIS-2017 Val & Test, and DAVIS-2016. Our approaches surpass the state-of-the-art competitors and display exceptional efficiency and scalability consistently across all six benchmarks. Moreover, we notably achieved the 1st position in the 3 rd Large-scale Video Object Segmentation Challenge.

Transformer fusion for indoor RGB-D semantic segmentation

Fusing geometric cues with visual appearance is an imperative theme for RGB-D indoor semantic segmentation. Existing methods commonly adopt convolutional modules to aggregate multi-modal features, paying little attention to explicitly leveraging the long-range dependencies in feature fusion. Therefore, it is challenging for existing methods to accurately segment objects with large-scale variations. In this paper, we propose a novel transformer-based fusion scheme, named TransD-Fusion, to better model contextualized awareness. Specifically, TransD-Fusion consists of a self-refinement module, a calibration scheme with cross-interaction, and a depth-guided fusion. The objective is to first improve modality-specific features with self- and cross-attention, and then explore the geometric cues to better segment objects sharing a similar visual appearance. Additionally, our transformer fusion benefits from a semantic-aware position encoding which spatially constrains the attention to neighboring pixels. Extensive experiments on RGB-D benchmarks demonstrate that the proposed method performs well over the state-of-the-art methods by large margins.

HybridCSF model for magnetic resonance image based brain tumor segmentation

<p>The human brain comprises a complex interconnection of nerve cells and vital organs, which regulates crucial bodily processes. Although neurons commonly undergo developmental stages, they may occasionally experience abnormalities, leading to abnormal growths known as brain tumors. The objective of brain tumor segmentation is to produce precise boundaries of brain tumor regions. This study extensively analyzes deep learning methods for brain tumor detection, evaluating their effectiveness across diverse datasets. It introduces a hybrid model, which is proposed by the name HybriCSF: hybrid convolutional-SVM-fuzzy C-means model combining convolutional neural network (CNN) with the classifier support vector machine (SVM) and clustering technique fuzzy C-means (FCM). The proposed model was implemented on Br35H, BraTs 2020 and BraTs2021 datasets. The suggested model outperformed the existing methods by achieving 98.6% of accuracy on Br35H dataset and dice score of 0.63, 0.87, 0.81 on BraTs 2020 dataset for enhancing tumor (ET), whole tumor (WT), and tumor core (TC), respectively. The achieved dice scores on the BraTs 2021datasets are 0.89, 0.95, and 0.89 for ET, WT, and TC, respectively. The results show that the suggested model HybriCSF outperforms the other CNN-based models in terms of accuracy.</p>

Implementation of YOLOv8-seg on store products to speed up the scanning process at point of sales

<p>You only look once v8 (YOLOv8)-seg and its variants are implemented to accelerate the collection of goods for a store for selling activity in Indonesia. The method used here is object detection and segmentation of these objects, a combination of detection and segmentation called instance segmentation. The novelty lies in the customization and optimization of YOLOv8-seg for detecting and segmenting 30 specific Indonesian products. The use of augmented data (125 images augmented into 1,250 images) enhances the model's ability to generalize and perform well in various scenarios. The small number of data points and the small number of epochs have proven reliable algorithms to implement on store products instead of using QR codes in a digital manner. Five models are examined, i.e., YOLOv8-seg, YOLOv8s-seg, YOLOv8m-seg, YOLOv8l-seg, and YOLOv8x-seg, with a data distribution of 64% for the training dataset, 16% for the validation dataset, and 20% for the testing dataset. The best model, YOLOv8l-seg, was obtained with the highest mean average precision (mAP) box value of 99.372% and a mAPmask value of 99.372% from testing the testing dataset. However, the YOLOv8mseg model can be the best alternative model with a mAPbox value of 99.330% since the number of parameters and the computational speed are the best compared to other models.</p>

An Efficient Robotic Pushing and Grasping Method in Cluttered Scene.

Pushing and grasping (PG) are crucial skills for intelligent robots. These skills enable robots to perform complex grasping tasks in various scenarios. These PG methods can be categorized into single-stage and multistage approaches. Single-stage methods are faster but less accurate, while multistage methods offer high accuracy at the expense of time efficiency. To address this issue, a novel end-to-end PG method called efficient PG network (EPGNet) is proposed in this article. EPGNet achieves both high accuracy and efficiency simultaneously. To optimize performance with fewer parameters, EfficientNet-B0 is used as the backbone of EPGNet. Additionally, a novel cross-fusion module is introduced to enhance network performance in robotic PG tasks. This module fuses and utilizes local and global features, aiding the network in handling objects of varying sizes in different scenes. EPGNet consists of two branches dedicated to predicting PG actions, respectively. Both branches are trained simultaneously within a Q-learning framework. Training data is collected through trial and error, involving the robot performing PG actions. To bridge the gap between simulation and reality, a unique PG dataset is proposed. Additionally, a YOLACT network is trained on the PG dataset to facilitate object detection and segmentation. A comprehensive set of experiments is conducted in simulated environments and real-world scenarios. The results demonstrate that EPGNet outperforms single-stage methods and offers competitive performance compared to multistage methods, all while utilizing fewer parameters. A video is available at https://youtu.be/HNKJjQH0MPc.

MetaSeg: Content-Aware Meta-Net for Omni-Supervised Semantic Segmentation.

Noisy labels, inevitably existing in pseudo-segmentation labels generated from weak object-level annotations, severely hamper model optimization for semantic segmentation. Previous works often rely on massive handcrafted losses and carefully tuned hyperparameters to resist noise, suffering poor generalization capability and high model complexity. Inspired by recent advances in meta-learning, we argue that rather than struggling to tolerate noise hidden behind clean labels passively, a more feasible solution would be to find out the noisy regions actively, so as to simply ignore them during model optimization. With this in mind, this work presents a novel meta-learning-based semantic segmentation method, MetaSeg, that comprises a primary content-aware meta-net (CAM-Net) to serve as a noise indicator for an arbitrary segmentation model counterpart. Specifically, CAM-Net learns to generate pixel-wise weights to suppress noisy regions with incorrect pseudo-labels while highlighting clean ones by exploiting hybrid strengthened features from image content, providing straightforward and reliable guidance for optimizing the segmentation model. Moreover, to break the barrier of time-consuming training when applying meta-learning to common large segmentation models, we further present a new decoupled training strategy that optimizes different model layers in a divide-and-conquer manner. Extensive experiments on object, medical, remote sensing, and human segmentation show that our method achieves superior performance, approaching that of fully supervised settings, which paves a new promising way for omni-supervised semantic segmentation.

HyperSOR: Context-Aware Graph Hypernetwork for Salient Object Ranking.

Salient object ranking (SOR) aims to segment salient objects in an image and simultaneously predict their saliency rankings, according to the shifted human attention over different objects. The existing SOR approaches mainly focus on object-based attention, e.g., the semantic and appearance of object. However, we find that the scene context plays a vital role in SOR, in which the saliency ranking of the same object varies a lot at different scenes. In this paper, we thus make the first attempt towards explicitly learning scene context for SOR. Specifically, we establish a large-scale SOR dataset of 24,373 images with rich context annotations, i.e., scene graphs, segmentation, and saliency rankings. Inspired by the data analysis on our dataset, we propose a novel graph hypernetwork, named HyperSOR, for context-aware SOR. In HyperSOR, an initial graph module is developed to segment objects and construct an initial graph by considering both geometry and semantic information. Then, a scene graph generation module with multi-path graph attention mechanism is designed to learn semantic relationships among objects based on the initial graph. Finally, a saliency ranking prediction module dynamically adopts the learned scene context through a novel graph hypernetwork, for inferring the saliency rankings. Experimental results show that our HyperSOR can significantly improve the performance of SOR.

Image segmentation of Komering script using bounding box

The development of deep learning technology is widely used for various purposes, including recognizing characters in a document. One of the scripts that can benefit from this deep learning technology is the Komering script, which is a local script in the South Sumatra region. However, there are challenges in reading documents written in this script, requiring a method to separate each character in a document. Therefore, there is a need for a technology that can automatically segment images of documents written in the Komering script. This research introduces an innovative technique for segmenting images of characters in documents that contain Komering script characters. The segmentation technique employs bounding box technology to separate each Komering script character, subsequently recognized by a pre-trained deep learning model. The bounding box approach imposes restrictions on the segmented object area. To recognize Komering characters, a deep learning model with a convolutional neural network (CNN) algorithm is employed.

High-Precision Mango Orchard Mapping Using a Deep Learning Pipeline Leveraging Object Detection and Segmentation

High-Precision Mango Orchard Mapping Using a Deep Learning Pipeline Leveraging Object Detection and Segmentation

The prediction of segmental and tonal information in Mandarin Chinese: an eye-tracking investigation

ABSTRACT There is controversy about the extent to which people predict phonology during comprehension. In three visual-world experiments, we ask whether it occurs in Mandarin, a tonal language. Participants heard sentences containing a target word that was highly predictable (Cloze 80.2%, Experiment 1) or very highly predictable (Cloze 93.9%, Experiments 2–3) and saw an array of objects containing one whose name matched the target word (Experiments 1–2), was unrelated to the target word (Experiments 1–3), or matched the target word in segment and tone (Experiments 1–3), in segment only (Experiments 1–3), or tone only (Experiment 3). In comparison to the unrelated object, participants looked more at the segment + tone object (Experiments 1–3), and sometimes at the segment object (Experiments 1 and 3), but not at the tone object. We conclude that participants predict segmental information independently of tone.

Developing Machine Vision in Tree-Fruit Applications-Fruit Count, Fruit Size and Branch Avoidance in Automated Harvesting.

Recent developments in affordable depth imaging hardware and the use of 2D Convolutional Neural Networks (CNN) in object detection and segmentation have accelerated the adoption of machine vision in a range of applications, with mainstream models often out-performing previous application-specific architectures. The need for the release of training and test datasets with any work reporting model development is emphasized to enable the re-evaluation of published work. An additional reporting need is the documentation of the performance of the re-training of a given model, quantifying the impact of stochastic processes in training. Three mango orchard applications were considered: the (i) fruit count, (ii) fruit size and (iii) branch avoidance in automated harvesting. All training and test datasets used in this work are available publicly. The mAP 'coefficient of variation' (Standard Deviation, SD, divided by mean of predictions using models of repeated trainings × 100) was approximately 0.2% for the fruit detection model and 1 and 2% for the fruit and branch segmentation models, respectively. A YOLOv8m model achieved a mAP50 of 99.3%, outperforming the previous benchmark, the purpose-designed 'MangoYOLO', for the application of the real-time detection of mango fruit on images of tree canopies using an edge computing device as a viable use case. YOLOv8 and v9 models outperformed the benchmark MaskR-CNN model in terms of their accuracy and inference time, achieving up to a 98.8% mAP50 on fruit predictions and 66.2% on branches in a leafy canopy. For fruit sizing, the accuracy of YOLOv8m-seg was like that achieved using Mask R-CNN, but the inference time was much shorter, again an enabler for the field adoption of this technology. A branch avoidance algorithm was proposed, where the implementation of this algorithm in real-time on an edge computing device was enabled by the short inference time of a YOLOv8-seg model for branches and fruit. This capability contributes to the development of automated fruit harvesting.

Few-shot remote sensing image segmentation based on label propagation and open-set domain adaptation

ABSTRACT Few-shot segmentation aims to segment objects in a task dataset with only a few labelled target examples, aided by a lot of labelled auxiliary datasets. Existing approaches typically assume that the task dataset and auxiliary dataset originate from the same source but have disjoint category sets. However, it is common in remote sensing image segmentation tasks that the datasets are from different sources and the category sets are often partially overlapped, where models trained on the auxiliary dataset degrade in performance on the task dataset. To address these issues, we present a few-shot segmentation method based on graph network and open-set domain adaptation. The proposed method utilizes a graph model to capture the relationships between super pixels and to predict labels for the unlabelled samples by label propagation, meanwhile, leverages open-set domain adaptation to reduce inter-domain discrepancies and enhances model transferability. The proposed method is evaluated on two publicly available datasets, and the effectiveness of the proposed approach is demonstrated by the significant improvement in accuracy compared to existing methods.

RetinaViT: Efficient Visual Backbone for Online Video Streams.

In online video understanding, which has a wide range of real-world applications, inference speed is crucial. Many approaches involve frame-level visual feature extraction, which often represents the biggest bottleneck. We propose RetinaViT, an efficient method for extracting frame-level visual features in an online video stream, aiming to fundamentally enhance the efficiency of online video understanding tasks. RetinaViT is composed of efficiently approximated Transformer blocks that only take changed tokens (event tokens) as queries and reuse the already processed tokens from the previous timestep for the others. Furthermore, we restrict keys and values to the spatial neighborhoods of event tokens to further improve efficiency. RetinaViT involves tuning multiple parameters, which we determine through a multi-step process. During model training, we randomly vary these parameters and then perform black-box optimization to maximize accuracy and efficiency on the pre-trained model. We conducted extensive experiments on various online video recognition tasks, including action recognition, pose estimation, and object segmentation, validating the effectiveness of each component in RetinaViT and demonstrating improvements in the speed/accuracy trade-off compared to baselines. In particular, for action recognition, RetinaViT built on ViT-B16 reduces inference time by approximately 61.9% on the CPU and 50.8% on the GPU, while achieving slight accuracy improvements rather than degradation.

Masked autoencoder: influence of self-supervised pretraining on object segmentation in industrial images

The amount of labelled data in industrial use cases is limited because the annotation process is time-consuming and costly. As in research, self-supervised pretraining such as MAE resulted in training segmentation models with fewer labels, this is also an interesting direction for industry. The reduction of required labels is achieved with large amounts of unlabelled images for the pretraining that aims to learn image features. This paper analyses the influence of MAE pretraining on the efficiency of label usage for semantic segmentation with UNETR. This is investigated for the use case of log-yard cranes. Additionally, two transfer learning cases with respect to crane type and perspective are considered in the context of label-efficiency. The results show that MAE is successfully applicable to the use case. With respect to the segmentation, an IoU improvement of 3.26% is reached while using 2000 labels. The strongest positive influence is found for all experiments in the lower label amounts. The highest effect is achieved with transfer learning regarding cranes, where IoU and Recall increase about 4.31% and 8.58%, respectively. Further analyses show that improvements result from a better distinction between the background and the segmented crane objects.

DiagSWin: A multi-scale vision transformer with diagonal-shaped windows for object detection and segmentation

Recently, Vision Transformer and its variants have demonstrated remarkable performance on various computer vision tasks, thanks to its competence in capturing global visual dependencies through self-attention. However, global self-attention suffers from high computational cost due to quadratic computational overhead, especially for the high-resolution vision tasks (e.g., object detection and semantic segmentation). Many recent works have attempted to reduce the cost by applying fine-grained local attention, but these approaches cripple the long-range modeling power of the original self-attention mechanism. Furthermore, these approaches usually have similar receptive fields within each layer, thus limiting the ability of each self-attention layer to capture multi-scale features, resulting in performance degradation when handling images with objects of different scales. To address these issues, we develop the Diagonal-shaped Window (DiagSWin) attention mechanism for modeling attentions in diagonal regions at hybrid scales per attention layer. The key idea of DiagSWin attention is to inject multi-scale receptive field sizes into tokens: before computing the self-attention matrix, each token attends its closest surrounding tokens at fine granularity and the tokens far away at coarse granularity. This mechanism is able to effectively capture multi-scale context information while reducing computational complexity. With DiagSwin attention, we present a new variant of Vision Transformer models, called DiagSWin Transformers, and demonstrate their superiority in extensive experiments across various tasks. Specifically, the DiagSwin Transformer with a large size achieves 84.4% Top-1 accuracy and outperforms the SOTA CSWin Transformer on ImageNet with 40% fewer model size and computation cost. When employed as backbones, DiagSWin Transformers achieve significant improvements over the current SOTA modules. In addition, our DiagSWin-Base model yields 51.1 box mAP and 45.8 mask mAP on COCO for object detection and segmentation, and 52.3 mIoU on the ADE20K for semantic segmentation.