Real-time Instance Research Articles

Real-Time Instance Segmentation Models for Identification of Vehicle Parts

Automated assessment of car damage is a major challenge in the auto repair and damage assessment industries. The domain has several application areas, ranging from car assessment companies, such as car rentals and body shops, to accidental damage assessment for car insurance companies. In vehicle assessment, the damage can take many forms, from scratches, minor dents, and major dents to missing parts. Often, the assessment area has a significant level of noise, such as dirt, grease, oil, or rush, which makes accurate identification challenging. Moreover, in the repair industry, identifying a particular part is the first step in obtaining an accurate labor and part assessment, where the presence of different car models, shapes, and sizes makes the task even more challenging for a machine-learning model to perform well. To address these challenges, this study explores and applies various instance segmentation methodologies to determine the best-performing models. This study focuses on two genres of real-time instance segmentation models, namely, SipMask and YOLACT, owing to their industrial significance. These methodologies were evaluated against a previously reported car parts dataset (DSMLR) as well as an internally curated dataset extracted from local car repair workshops. The YOLACT-based part localization and segmentation method outperformed other real-time instance mechanisms with an mAP of 66.5. For the workshop repair dataset, SipMask++ reported better accuracy for object detection with a mAP of 57.0, with outcomes for A P I o U = . 50 and A P I o U = . 75 reporting 72.0 and 67.0, respectively, whereas YOLACT was observed to be a better performer for A P s with 44.0 and 2.6 for object detection and segmentation categories, respectively.

A depth information aided real-time instance segmentation method for space task scenarios under CPU platform

Visual instance segmentation ability is one of the effective means to promote the autonomy and intelligence of space agents. However, due to the limited airborne computing capacity, current methods are difficult to deploy on space agents, because these methods are developed based on GPU. To this end, this paper proposes a novel framework of instance segmentation by introducing depth information and combining traditional computer vision techniques with an object detection method. This framework provides a new idea for the implementation of instance segmentation. The experiment results show that the proposed method achieves a real-time performance under a common laptop CPU platform. In addition, thanks to the introduction of depth information, the proposed method can obtain better segmentation results compared to Mask R–CNN and SOLOv2 in complex scenes (poor illumination and occlusion). Finally, because the semantic information is obtained by the object detection method in this paper, the model training adopts a weakly supervised manner from bounding-box annotations, which can reduce various costs of data labeling to a certain extent.

UAV imagery based potential safety hazard evaluation for high-speed railroad using Real-time instance segmentation

Potential safety hazards (PSHs) along the track needs to be inspected and evaluated regularly to ensure a safe environment for high-speed railroad operations. Other than track inspection, evaluating potential safety hazards in the nearby areas often requires inspectors to patrol along the track and visually identify potential threads to the train operation. The current visual inspection approach is very time-consuming and may raise safety concerns for the inspectors, especially in remote areas. Using the unmanned aerial vehicle (UAV) has great potential to complement the visual inspection by providing a better view from the top and ease the safety concerns in many cases. This study develops an automatic PSH detection framework named YOLARC (You Only Look at Railroad Coefficients) using UAV imagery for high-speed railroad monitoring. First, YOLARC is equipped with a new backbone having multiple available receptive fields to strengthen the multi-scale representation capability at a granular level and enrich the semantic information in the feature space. Then, the system integrates the abundant semantic features at different high-level layers by a light weighted feature pyramid network (FPN) with multi-scale pyramidal architecture and a Protonet with residual structure to precisely predict the track areas and PSHs. A hazard level evaluation (HLE) method, which calculates the distance between identified PSH and the track, is also developed and integrated for quantifying the hazard level. Experiments conducted on the UAV imagery of high-speed railroad dataset show the proposed system can quickly and effectively turn UAV images into useful information with a high detection rate and processing speed.

Fusion of colour contrasted images for early detection of oesophageal squamous cell dysplasia from endoscopic videos in real time

Standard white light (WL) endoscopy often misses precancerous oesophageal changes due to their only subtle differences to the surrounding normal mucosa. While deep learning (DL) based decision support systems benefit to a large extent, they face two challenges, which are limited annotated data sets and insufficient generalisation. This paper aims to fuse a DL system with human perception by exploiting computational enhancement of colour contrast. Instead of employing conventional data augmentation techniques by alternating RGB values of an image, this study employs a human colour appearance model, CIECAM, to enhance the colours of an image. When testing on a frame of endoscopic videos, the developed system firstly generates its contrast-enhanced image, then processes both original and enhanced images one after another to create initial segmentation masks. Finally, fusion takes place on the assembled list of masks obtained from both images to determine the finishing bounding boxes, segments and class labels that are rendered on the original video frame, through the application of non-maxima suppression technique (NMS). This deep learning system is built upon real-time instance segmentation network Yolact. In comparison with the same system without fusion, the sensitivity and specificity for detecting early stage of oesophagus cancer, i.e. low-grade dysplasia (LGD) increased from 75% and 88% to 83% and 97%, respectively. The video processing/play back speed is 33.46 frames per second. The main contribution includes alleviation of data source dependency of existing deep learning systems and the fusion of human perception for data augmentation.

Machine Learning Framework for Real-Time Assessment of Traffic Safety Utilizing Connected Vehicle Data

Assessment of roadway safety in real-time is a necessary component for providing proactive safety countermeasures to ensure the continued safety and efficiency of roadways. A framework for utilizing data from connected vehicles and other probe sources is proposed in this study. Connected vehicles present an opportunity to provide live fingerprinting and activity monitoring on roadways. Taking advantage of high-resolution trajectory data streaming directly from connected vehicles, variables are extracted and the relationship with crashes are explored utilizing statistical and machine learning models. Hard acceleration events, in conjunction with segment miles are shown to have strong positive correlations with historical crash outcomes as proven by OLS, Poisson and Gradient Booster regression models. An XGBoost classification model is then trained to predict the real-time instances of crash outcomes at 5 min temporal bins with high levels of accuracy when trained with data including the real-time segment speed, reference speed, segment miles, a segment crash risk factor and other variables related to the difference in speeds between consecutive segments as well as the hour of the day. A weighted ensemble model achieved the best performance with an accuracy of 0.95. The results present evidence that the framework can capitalize on the richness of data available via connected vehicles and is implementable as a component in Advanced Traffic Management Systems for the analysis of safety critical situations in real-time.

A Deep-DrpXML and IAG-GWO based CHST fostered blockchain technology for secured dynamic optimal routing for wireless sensor networks

In wireless sensor network (WSN), routing is one of the substantial maneuvers for distributing data packets to the base station. But malevolent node outbreaks will happen during routing process, which exaggerate the wireless sensor network operations. Therefore, a secure routing protocol is required, which safeguards the routing fortification and the wireless sensor network effectiveness. The existing routing protocol is dynamically volatile during real time instances, and it is very hard to recognize the unsecured routing node performances. In this manuscript, a Deep Dropout extreme Machine learning optimized Improved Alpha-Guided Grey Wolf based Crypto Hash Signature Token fostered Blockchain Technology is proposed for secure dynamic optimal routing in Wireless Sensor Networks (SDOR-DEML-IAgGWO-CHS-BWSN). In this, Crypto Hash signature (CHS) token are generated for flow accesses with a secret key owned by each routing sensor node and it also offers an optimal path for data transmission. Then the secured dynamic optimal routing information is delivered through the proposed Blockchain based wireless sensor network platform with the help of Deep Dropout Extreme Machine learning optimized Improved Alpha-Guided Grey Wolf routing algorithm. Then the proposed method is simulated using the NS-2 (Network Simulator) tool. The simulation performance of the proposed SDOR-DEML-IAgGWO-CHS-BWSN method provide 76.26%, 65.57%, 60.85%, 48.99% and 42.9% lower delay during 30% malicious routing environment, 73.06%, 63.82%, 59.25%, 44.79% and 38.84% lower delay during 60% malicious routing environment is compared with the existing methods.

Real-time instance segmentation of surgical instruments using attention and multi-scale feature fusion.

Precise instrument segmentation aids surgeons to navigate the body more easily and increases patient safety. While accurate tracking of surgical instruments in real-time plays a crucial role in minimally invasive computer-assisted surgeries, it is a challenging task to achieve, mainly due to: (1) a complex surgical environment, and (2) model design trade-off in terms of both optimal accuracy and speed. Deep learning gives us the opportunity to learn complex environment from large surgery scene environments and placements of these instruments in real world scenarios. The Robust Medical Instrument Segmentation 2019 challenge (ROBUST-MIS) provides more than 10,000 frames with surgical tools in different clinical settings. In this paper, we propose a light-weight single stage instance segmentation model complemented with a convolutional block attention module for achieving both faster and accurate inference. We further improve accuracy through data augmentation and optimal anchor localization strategies. To our knowledge, this is the first work that explicitly focuses on both real-time performance and improved accuracy. Our approach out-performed top team performances in the most recent edition of ROBUST-MIS challenge with over 44% improvement on area-based multi-instance dice metric MI_DSC and 39% on distance-based multi-instance normalized surface dice MI_NSD. We also demonstrate real-time performance (>60 frames-per-second) with different but competitive variants of our final approach.

Contour-Based Wild Animal Instance Segmentation Using a Few-Shot Detector.

Simple SummaryBiodiversity monitoring is one of the primary means of ecological research. With the development of convolutional neural networks (CNNs) in the field of instance segmentation, CNNs are also used for species recognition. Almost all species recognition models apply pixel-based instance segmentation to recognize animal species. However, pixel-based instance segmentation models require a large number of annotations and labels, which makes them time-consuming and unsuitable for small datasets. Therefore, in this paper, we propose a contour-based wild animal instance segmentation model that can reach a balance between accuracy and real-time performance.Camera traps are widely used in wildlife research, conservation, and management, and abundant images are acquired every day. Efficient real-time instance segmentation networks can help ecologists label and study wild animals. However, existing deep convolutional neural networks require a large number of annotations and labels, which makes them unsuitable for small datasets. In this paper, we propose a two-stage method for the instance segmentation of wildlife, including object detection and contour approximation. In the object detection stage, we use FSOD (few-shot object detection) to recognize animal species and detect the initial bounding boxes of animals. In the case of a small wildlife dataset, this method may improve the generalization ability of the wild animal species recognition and even identify new species that only have a small number of training samples. In the second stage, deep snake is used as the contour approximation model for the instance segmentation of wild mammals. The initial bounding boxes generated in the first stage are input to deep snake to approximate the contours of the animal bodies. The model fuses the advantages of detecting new species and real-time instance segmentation. The experimental results show that the proposed method is more suitable for wild animal instance segmentation, in comparison with pixel-wise segmentation methods. In particular, the proposed method shows a better performance when facing challenging images.

RISAT: real-time instance segmentation with adversarial training

RISAT: real-time instance segmentation with adversarial training

Real-time instance segmentation with assembly parallel task

Real-time instance segmentation with assembly parallel task

The extraction of pixel-wise visual multi-cues for AHP-based privacy measurement

Visual privacy measurement is a computable evaluation method to quantitatively describe privacy-sensitive content in images or videos. The visual cues and their quantitative evaluation model are two key points to obtain satisfactory results for the visual privacy measurement. To this end, this paper proposes a novel visual privacy measurement framework based on analytic hierarchy process to achieve instance-level visual privacy measurement by using pixel-wise multi-cues, including: firstly, a boundary constraint-based multi-objects saliency detection module with gestalt-laws prior is proposed to generate fine salient maps in complex scenes. And a real-time instance segmentation module is built to generate pixel-wise masks by training new privacy-related annotations. The two module are combined to generate multiple visual cues, such as saliency feature, category feature and area feature. Then, a measurement model based on analytic hierarchy process is established to determine the correlated weights of multi-cues to the privacy levels, which depends on the pixel-wise visual cues including saliency feature, the category feature and the area feature. Finally, the privacy levels are calculated according to the feature values and their weights. Experimental results show that the proposed methods provide more reliable visual cues of multi-objects with fine structure and high efficiency, and the measurement results are highly consistent with the human evaluation results.

Assessing YOLACT++ for real time and robust instance segmentation of medical instruments in endoscopic procedures.

Image-based tracking of laparoscopic instruments plays a fundamental role in computer and robotic-assisted surgeries by aiding surgeons and increasing patient safety. Computer vision contests, such as the Robust Medical Instrument Segmentation (ROBUST-MIS) Challenge, seek to encourage the development of robust models for such purposes, providing large, diverse, and high-quality datasets. To date, most of the existing models for instance segmentation of medical instruments were based on two-stage detectors, which provide robust results but are nowhere near to the real-time, running at 5 frames-per-second (fps) at most. However, for the method to be clinically applicable, a real-time capability is utmost required along with high accuracy. In this paper, we propose the addition of attention mechanisms to the YOLACT architecture to allow real-time instance segmentation of instruments with improved accuracy on the ROBUST-MIS dataset. Our proposed approach achieves competitive performance compared to the winner of the 2019 ROBUST-MIS challenge in terms of robustness scores, obtaining 0.313 ML_DSC and 0.338 MLNSD while reaching real-time performance at >45 fps.

Development and analysis of priority decision rules using MCDM approach for a flexible job shop scheduling: A simulation study

Flexible job shop scheduling problem (FJSSP) is a critical operational planning step in manufacturing that ensures on-time delivery of goods leading to customer satisfaction. The process of assigning jobs to machines has a larger solution space and consumes a huge amount of time and effort. This research focuses on solving FJSSP using the discrete event simulation (DES) model and integrates it with Composite Dispatching Rules (CDRs) and multi-criteria decision making (MCDMs) based priority rules. Performance evaluation of the flexible job shop is carried out to minimize the Makespan, mean Flow-Time, mean Tardiness, and maximum Tardiness. Best performing CDRs and hybrid MCDM techniques are employed for minimizing the objectives considered in this study. The proposed solution framework integrates Fuzzy Analytic Hierarchy Process (FAHP) for assigning weights to the criteria and MCDM approaches namely technique for order of preference by similarity to ideal solution (TOPSIS), The Evaluation based on Distance from Average Solution (EDAS), Compromise Programming (CP), and Weighted Average Method (WAM) separately to prioritize the jobs. Benchmark problems of different sizes are tested using the best performing CDRs available in the literature and proposed MCDM approaches. The proposed approach is implemented in a real-world large-scale flexible job shop (FJS) that produces auto components handling 114 job variants with 245 operations using 28 machines. The performance analysis indicates that no single rule outperforms all the objective measures considered in this study. The MCDMs based rules perform better compared to CDRs for large-scale problems by considering realistic criteria such as demand, due date, setup time, process time, customer priority, and number of operations. The methodology proposed in this study can be tailored in terms of the criteria used for ranking and production parameters to be executable for any real-time instance.

A Semantic SLAM System for Catadioptric Panoramic Cameras in Dynamic Environments.

When a traditional visual SLAM system works in a dynamic environment, it will be disturbed by dynamic objects and perform poorly. In order to overcome the interference of dynamic objects, we propose a semantic SLAM system for catadioptric panoramic cameras in dynamic environments. A real-time instance segmentation network is used to detect potential moving targets in the panoramic image. In order to find the real dynamic targets, potential moving targets are verified according to the sphere’s epipolar constraints. Then, when extracting feature points, the dynamic objects in the panoramic image are masked. Only static feature points are used to estimate the pose of the panoramic camera, so as to improve the accuracy of pose estimation. In order to verify the performance of our system, experiments were conducted on public data sets. The experiments showed that in a highly dynamic environment, the accuracy of our system is significantly better than traditional algorithms. By calculating the RMSE of the absolute trajectory error, we found that our system performed up to 96.3% better than traditional SLAM. Our catadioptric panoramic camera semantic SLAM system has higher accuracy and robustness in complex dynamic environments.

Hierarchical deep reinforcement learning to drag heavy objects by adult-sized humanoid robot

Most research on robot manipulation focuses on objects that are light enough for the robot to pick them up. However, in our daily life, some objects are too big or too heavy to be picked up or carried, so that dragging them is necessary. Although bipedal humanoid robots have nowadays good mobility on level ground, dragging unfamiliar objects including large and heavy objects on various surfaces is an interesting research area with many applications, which will provide insights into human manipulation and will encourage the development of novel algorithms for robot motion planning and control. This is a challenging problem, not only because of the unknown and potentially variable friction of the foot, but also because the feet of the robot may slip during unbalanced poses. In this paper, we propose a novel hierarchical deep learning algorithm that learns how to drag heavy objects with an adult-sized humanoid robot for the first time. First, we present a Three-layered Convolution Volumetric Network (TCVN) for 3D object classification with point clouds volumetric occupancy grid integration. Second, we propose a lightweight real-time instance segmentation method named Tiny-YOLACT for the detection and classification of the floor surface. Third, we propose a deep Q-learning algorithm to learn the policy control of the Center of Mass of the robot (DQL-COM). The DQL-COM algorithm learning is bootstrapped using the ROS Gazebo simulator. After initial training, we complete training on the THORMANG-Wolf, a 1.4 m tall adult-sized humanoid robot with 27 degrees of freedom and weighing 48 kg, on three distinct types of surfaces. We evaluate the performance of our approach by dragging eight different types of objects (e.g., a small suitcase, a large suitcase, a chair). The extensive experiments (480 times on the real robot) included dragging a heavy object with a mass of 84.6 kg (two times greater than the robot’s weight) and showed remarkable success rates of 92.92% when combined with the force–torque sensors, and 83.75% without force–torque sensors.

A fast instance segmentation with one-stage multi-task deep neural network for autonomous driving

An accurate real-time instance segmentation, which can perform both object detection and semantic segmentation at the same time with a multi-task neural network, is important for autonomous driving. This paper proposes a fast one-stage multi-task neural network for instance segmentation, which can meet the requirements of real-time processing with sufficient accuracy and that is more desirable for self-driving applications. With a one-stage strategy, it can perform object detection and segmentation concurrently. This paper conducts the related experiments with two public datasets. The cross-validation was carried out with set of variables to determine the optimal combination of each model and compared with the mainstream instance segmentation algorithms. According to our experiment, the proposed algorithm has five times the performance compared to the previous algorithms, which can meet the real-time requirement for autonomous driving applications.

Automatic segmentation, inpainting, and classification of defective patterns on ancient architecture using multiple deep learning algorithms

Ancient architectures have a lot of distinctive patterns on their component surfaces. But due to the wind and the Sun, the patterns will present extensive deficiencies, which are not conducive to the routine inspections and the subsequent repair work. To overcome these limits, an automatic deep learning-based method of segmentation, inpainting, and classification for defective patterns on ancient architecture is proposed. First, You Only Look At CoefficienTs, which is a real-time instance segmentation network, is applied to obtain the mask of the defective parts. Then Generative Image Inpainting with Contextual Attention, which is an image inpainting algorithm, is used to reconstruct the defective parts. Finally, Residual Neural Network-50 is used to classify the reconstructed images. In this paper, three types of dragon motifs in the Forbidden City were studied. The results show that the classification accuracy of the reconstructed images is increased by an average of 13.1%, with the maximum increasing by 23.5%. The proposed methods can prepare for future routine inspections in advance.

Real-Time Instance Segmentation of Traffic Videos for Embedded Devices.

The paper proposes a novel instance segmentation method for traffic videos devised for deployment on real-time embedded devices. A novel neural network architecture is proposed using a multi-resolution feature extraction backbone and improved network designs for the object detection and instance segmentation branches. A novel post-processing method is introduced to ensure a reduced rate of false detection by evaluating the quality of the output masks. An improved network training procedure is proposed based on a novel label assignment algorithm. An ablation study on speed-vs.-performance trade-off further modifies the two branches and replaces the conventional ResNet-based performance-oriented backbone with a lightweight speed-oriented design. The proposed architectural variations achieve real-time performance when deployed on embedded devices. The experimental results demonstrate that the proposed instance segmentation method for traffic videos outperforms the you only look at coefficients algorithm, the state-of-the-art real-time instance segmentation method. The proposed architecture achieves qualitative results with average precision on the COCO dataset, while its speed-oriented variations achieve speeds of up to frames per second on the Jetson AGX Xavier module.

A Real-Time Dynamic Object Segmentation Framework for SLAM System in Dynamic Scenes

To accurately detect dynamic objects in dynamic scenes (DSs), a detection framework equipped with visual-based measurement methods has been proposed in this article. First, to segment dynamic objects in real time, the real-time instance segmentation network, You Only Look At CoefficienTs (YOLACT), has been introduced. Second, the geometric constraints have been utilized to further filter the missing dynamic feature points outside the segmentation mask. The dense optical flow method with adaptive threshold has been introduced to detect the missing dynamic objects driven by humans. Third, a background inpainting strategy has been proposed to restore the features occluded by dynamic objects. In order to verify the effectiveness of the dynamic object detection, the proposed method has been embedded in the visual simultaneous localization and mapping (SLAM) system to improve its performance in dynamic environments. Experiments performed on the Technische Universitat Munchen (TUM) and KITTI datasets have proved that the proposed detection method has an excellent performance in DSs, which is of great significance to improve the robustness of the SLAM system.

An Efficient Multi-Core Resource Allocation using the Multi-Level Objective Functions in Cloud Environment

Background: In recent years, the computational memory and energy conservation have become a major problem in cloud computing environment due to the increase in data size and computing resources. Since, most of the different cloud providers offer different cloud services and resources use limited number of user’s applications. Objective: The main objective of this work is to design and implement a cloud resource allocation and resources scheduling model in the cloud environment. Methods: In the proposed model, a novel cloud server to resource management technique is proposed on real-time cloud environment to minimize the cost and time. In this model different types of cloud resources and its services are scheduled using multi-level objective constraint programming. Proposed cloud server-based resource allocation model is based on optimization functions to minimize the resource allocation time and cost. Results: Experimental results proved that the proposed model has high computational resource allocation time and cost compared to the existing resource allocation models. Conclusion: This cloud service and resource optimization model is efficiently implemented and tested in real-time cloud instances with different types of services and resource sets.