Vision-based Deep Learning Research Articles

Deep learning prediction of error and skill in robotic prostatectomy suturing.

Manual objective assessment of skill and errors in minimally invasive surgery have been validated with correlation to surgical expertise and patient outcomes. However, assessment and error annotation can be subjective and are time-consuming processes, often precluding their use. Recent years have seen the development of artificial intelligence models to work towards automating the process to allow reduction of errors and truly objective assessment. This study aimed to validate surgical skill rating and error annotations in suturing gestures to inform the development and evaluation of AI models. SAR-RARP50 open data set was blindly, independently annotated at the gesture level in Robotic-Assisted Radical Prostatectomy (RARP) suturing. Manual objective assessment tools and error annotation methodology, Objective Clinical Human Reliability Analysis (OCHRA), were used as ground truth to train and test vision-based deep learning methods to estimate skill and errors. Analysis included descriptive statistics plus tool validity and reliability. Fifty-four RARP videos (266min) were analysed. Strong/excellent inter-rater reliability (range r = 0.70-0.89, p < 0.001) and very strong correlation (r = 0.92, p < 0.001) between objective assessment tools was demonstrated. Skill estimation of OSATS and M-GEARS had a Spearman's Correlation Coefficient 0.37 and 0.36, respectively, with normalised mean absolute error representing a prediction error of 17.92% (inverted "accuracy" 82.08%) and 20.6% (inverted "accuracy" 79.4%) respectively. The best performing models in error prediction achieved mean absolute precision of 37.14%, area under the curve 65.10% and Macro-F1 58.97%. This is the first study to employ detailed error detection methodology and deep learning models within real robotic surgical video. This benchmark evaluation of AI models sets a foundation and promising approach for future advancements in automated technical skill assessment.

Advances in vision-based deep learning methods for interacting hands reconstruction: A survey

Vision-based hand reconstructions have become noteworthy tools in enhancing interactive experiences in various applications such as virtual reality, augmented reality, and autonomous driving, which enable sophisticated interactions by reconstructing complex motions of human hands. Despite significant progress driven by deep-learning methodologies, the quest for high-fidelity interacting hands reconstruction faces challenges such as limited dataset diversity, lack of detailed hand representation, occlusions, and differentiation between similar hand structures. This survey thoroughly reviews deep learning-based methods, diverse datasets, loss functions, and evaluation metrics addressing the complexities of interacting hands reconstruction. Mainstream algorithms of the past five years are systematically classified into two main categories: algorithms that employ explicit representations, such as parametric meshes and 3D Gaussian splatting, and those that utilize implicit representations, including signed distance fields and neural radiance fields. Novel deep-learning models like graph convolutional networks and transformers are applied to solve the aforementioned challenges in hand reconstruction effectively. Beyond summarizing these interaction-aware algorithms, this survey also briefly discusses hand tracking in virtual reality and augmented reality. To the best of our knowledge, this is the first survey specifically focusing on the reconstruction of both hands and their interactions with objects. The survey contains the various facets of hand modeling, deep learning approaches, and datasets, broadening the horizon of hand reconstruction research and future innovation in natural user interactions.

When segment anything model meets inventorying of roadway assets

Automating the identification, localization, and monitoring of roadway assets distributed widely in the roadway network is critical for the traffic management system. It can efficiently provide up-to-date information in supporting transportation asset management. Collecting videos with vehicle-mounted cameras and processing the data with computer vision-based deep learning methods is garnering increased attention from transportation agencies. While promising, challenges arise due to the lack of high-quality annotations for roadway assets in images, difficulties in identifying these assets, and limited solutions. The Segment Anything Model (SAM), a visual foundation model, demonstrates robust zero-shot capability for general image segmentation under various prompts. This study evaluates SAM’s applicability and efficiency in extracting roadway assets from images. Specifically, it examines the impacts of model size and prompt quality on SAM’s performance in segmenting roadway assets. Five state-of-the-art semantic segmentation models are trained and compared with SAM. Results show that a lightweight SAM with human-rendered prompts outperforms the five semantic segmentation models. Based on the evaluation results, future work will explore incorporating SAM into transportation asset management applications, promoting collaboration between human experts and artificial intelligence.

Spatio-temporal deep learning framework for pedestrian intention prediction in urban traffic scenes

Pedestrian intent prediction is an essential task for ensuring the safety of pedestrians and vehicles on the road. This task involves predicting whether a pedestrian intends to cross a road or not based on their behavior and surrounding environment. Previous studies have explored feature-based machine learning and vision-based deep learning models for this task but these methods have limitations in capturing the global spatio-temporal context and fusing different features of data effectively. To address these issues, we propose a novel hybrid framework HSTGCN for pedestrian intent prediction that combines spatio-temporal graph convolutional neural networks (STGCN) and long short-term memory (LSTM) networks. The proposed framework utilizes the strengths of both models by fusing multiple features, including skeleton pose, trajectory, height, orientation, and ego-vehicle speed, to predict their intentions accurately. The framework’s performance have been evaluated on the JAAD benchmark dataset and the results show that it outperforms the state-of-the-art methods. The proposed framework has potential applications in developing intelligent transportation systems, autonomous vehicles, and pedestrian safety technologies. The utilization of multiple features can significantly improve the performance of the pedestrian intent prediction task.

MoistNet: Machine vision-based deep learning models for wood chip moisture content measurement

Quick and reliable measurement of wood chip moisture content is an everlasting problem for numerous forest-reliant industries such as biofuel, pulp and paper, and bio-refineries. Moisture content is a critical attribute of wood chips due to its direct relationship with the final product quality. Conventional techniques for determining moisture content, such as oven-drying, possess some drawbacks in terms of their time-consuming nature, potential sample damage, and lack of real-time feasibility. Furthermore, alternative techniques, including NIR spectroscopy, electrical capacitance, X-rays, and microwaves, have demonstrated potential; nevertheless, they are still constrained by issues related to portability, precision, and the expense of the required equipment. Hence, there is a need for a moisture content determination method that is instant, portable, non-destructive, inexpensive, and precise. This study explores the use of deep learning and machine vision to predict moisture content classes from RGB images of wood chips. A large-scale image dataset comprising 1,600 RGB images of wood chips has been collected and annotated with ground truth labels, utilizing the results of the oven-drying technique. Two high-performing neural networks, MoistNetLite and MoistNetMax, have been developed leveraging Neural Architecture Search (NAS) and hyperparameter optimization. The developed models are evaluated and compared with state-of-the-art deep learning models. Results demonstrate that MoistNetLite achieves 87% accuracy with minimal computational overhead, while MoistNetMax exhibits exceptional precision with a 91% accuracy in wood chip moisture content class prediction. With improved accuracy (9.6% improvement in accuracy by MoistNetMax compared to the best baseline model ResNet152V2) and faster prediction speed (MoistNetLite being twice as fast as MobileNet), our proposed MoistNet models hold great promise for the wood chip processing industry to be efficiently deployed on portable devices, such as smartphones.

A review on vision-based deep learning techniques for damage detection in bolted joints

A review on vision-based deep learning techniques for damage detection in bolted joints

Deep learning model for detection of hotspots using infrared thermographic images of electrical installations

Hotspots in electrical power equipment or installations are a major issue whenever it occurs within the power system. Factors responsible for this phenomenon are many, sometimes inter-related and other times they are isolated. Electrical hotspots caused by poor connections are common. Deep learning models have become popular for diagnosing anomalies in physical and biological systems, by the instrumentality of feature extraction of images in convolutional neural networks. In this work, a VGG-16 deep neural network model is applied for identifying electrical hotspots by means of transfer learning. This model was achieved by first augmenting the acquired infrared thermographic images, using the pre-trained ImageNet weights of the VGG-16 algorithm with additional global average pooling in place of conventional fully connected layers and a softmax layer at the output. With the categorical cross-entropy loss function, the model was implemented using the Adam optimizer at learning rate of 0.0001 as well as some variants of the Adam optimization algorithm. On evaluation, with a test IRT image dataset, and a comparison with similar works, the research showed that a better accuracy of 99.98% in identification of electrical hotspots was achieved. The model shows good score in performance metrics like accuracy, precision, recall, and F1-score. The obtained results proved the potential of deep learning using computer vision parameters for infrared thermographic identification of electrical hotspots in power system installations. Also, there is need for careful selection of the IR sensor’s thermal range during image acquisition, and suitable choice of color palette would make for easy hotspot isolation, reduce the pixel to pixel temperature differential across any of the images, and easily highlight the critical region of interest with high pixel values. However, it makes edge detection difficult for human visual perception which computer vision-based deep learning model could overcome.

An ontology-based framework for worker’s health reasoning enabled by machine learning

Reports of fatal and nonfatal workplace injuries depict severe circumstances involving health and safety in industries. This leads to workers staying away from work in U.S. for an average of 12 days in 2020, implicating in managerial, financial, and organizational losses. In this context, Vision-Based Deep Learning (VBDL) and Knowledge Representation and Reasoning (KRR) allow real-time data retrieval of situations along with the semantic modeling and expressivity of the real world to mitigate injuries. This article presents a framework that interoperates vision-based deep learning and ontology reasoning to identify adverse working situations, introducing a novel ontology composed of a holistic perspective of workers’ health and safety. Moreover, the article provides multi-agent framework modeling to orchestrate the components’ interoperability, describing the framework’s architecture and deployment in workrooms. As a result, a practical evaluation over five days produced 8395 axioms constituted by 1210 individuals in the ontology, which allowed a temporal analysis of harmful conditions and their multiple overlapping using SPARQL and reasoning rules, particularly relevant to understanding explanations of overexertion, physical and environmental injuries. Therefore, the proposed ontology-based framework corroborates the long-term support in identifying, assessing, and controlling risks in the industries due to a well-defined knowledge model.

Autonomous driving in traffic with end-to-end vision-based deep learning

This paper presents a shallow end-to-end vision-based deep learning approach for autonomous vehicle driving in traffic scenarios. The primary objectives include lane keeping and maintaining a safe distance from preceding vehicles. This study leverages an imitation learning approach, creating a supervised dataset for robot control from expert agent demonstrations using the state-of-the-art Carla simulator in different traffic conditions. This dataset encompasses three different versions complementary to each other and we have made it publicly available along with the rest of the materials. The PilotNet neural model is utilized in two variants: the first one with complementary outputs for brake and throttle control commands along with dropout; the second one incorporates these improvements and adds the vehicle speed. Both models have been trained with the aforementioned dataset. The experimental results demonstrate that the models, despite their simplicity and shallow architecture, including only small-scale changes, successfully drive in traffic conditions without sacrificing performance in free-road environments, broadening their area of application widely. Additionally, the second model adeptly maintains a safe distance from leading cars and exhibits satisfactory generalization capabilities to diverse vehicle types. A new evaluation metric to measure the distance to the front vehicle has been created and added to Behavior Metrics; an open-source autonomous driving assessment tool built on CARLA that performs experimental validations of autonomous driving solutions.

An ergonomic evaluation using a deep learning approach for assessing postural risks in a virtual reality-based smart manufacturing context

This study proposes an integrated ergonomic evaluation designed to identify unsafe postures, whereby postural risks during industrial work are assessed in the context of virtual reality-based smart manufacturing. Unsafe postures were recognised by identifying the displacements of the centre of mass (COM) of body keypoints using a computer vision-based deep learning (DL) convolutional neural network approach. The risk levels for the identified unsafe postures were calculated using ergonomic risk assessment tools rapid upper limb assessment and rapid whole-body assessment. An analysis of variance was conducted to determine significant differences between the vertical and horizontal directions of postural movements associated with the most unsafe postures. The findings assess the ergonomic risk levels and identify the most unsafe postures during industrial work in smart manufacturing using DL method. The identified postural risks can help industry managers and researchers acquire a better understanding of unsafe postures.

Computer vision analysis of mother-infant interaction identified efficient pup retrieval in V1b receptor knockout mice

Close contact between lactating rodent mothers and their infants is essential for effective nursing. Whether the mother’s effort to retrieve the infants to their nest requires the vasopressin-signaling via V1b receptor has not been fully defined. To address this question, V1b receptor knockout (V1bKO) and control mice were analyzed in pup retrieval test. Because an exploring mother in a new test cage randomly accessed to multiple infants in changing backgrounds over time, a computer vision-based deep learning analysis was applied to continuously calculate the distances between the mother and the infants as a parameter of their relationship. In an open-field, a virgin female V1bKO mice entered fewer times into the center area and moved shorter distances than wild-type (WT). While this behavioral pattern persisted in V1bKO mother, the pup retrieval test demonstrated that total distances between a V1bKO mother and infants came closer in a shorter time than with a WT mother. Moreover, in the medial preoptic area, parts of the V1b receptor transcripts were detected in galanin- and c-fos-positive neurons following maternal stimulation by infants. This research highlights the effectiveness of deep learning analysis in evaluating the mother-infant relationship and the critical role of V1b receptor in pup retrieval during the early lactation phase.

Vision-state Fusion: Improving Deep Neural Networks for Autonomous Robotics

Vision-based deep learning perception fulfills a paramount role in robotics, facilitating solutions to many challenging scenarios, such as acrobatic maneuvers of autonomous unmanned aerial vehicles (UAVs) and robot-assisted high-precision surgery. Control-oriented end-to-end perception approaches, which directly output control variables for the robot, commonly take advantage of the robot’s state estimation as an auxiliary input. When intermediate outputs are estimated and fed to a lower-level controller, i.e., mediated approaches, the robot’s state is commonly used as an input only for egocentric tasks, which estimate physical properties of the robot itself. In this work, we propose to apply a similar approach for the first time – to the best of our knowledge – to non-egocentric mediated tasks, where the estimated outputs refer to an external subject. We prove how our general methodology improves the regression performance of deep convolutional neural networks (CNNs) on a broad class of non-egocentric 3D pose estimation problems, with minimal computational cost. By analyzing three highly-different use cases, spanning from grasping with a robotic arm to following a human subject with a pocket-sized UAV, our results consistently improve the R2\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$^{2}$$\\end{document} regression metric, up to +0.51, compared to their stateless baselines. Finally, we validate the in-field performance of a closed-loop autonomous cm-scale UAV on the human pose estimation task. Our results show a significant reduction, i.e., 24% on average, on the mean absolute error of our stateful CNN, compared to a State-of-the-Art stateless counterpart.

Vision-based seam tracking for GMAW fillet welding based on keypoint detection deep learning model

Pre-programmed welding robots significantly improved the efficiency and quality of the welds in large batch production. In small and medium batch production, the robots need appropriate sensors to perform well and adapt to the changes and uncertainties in a noisy welding environment. Vision-based sensors enabled by machine learning are making it possible to sense in process previously not measurable. One challenge is developing artificial intelligent models capable of real-time seam tracking, particularly in fillet joints where visual analysis is hindered by non-perpendicular camera angles and arc reflections. In this paper, we propose a vision system that enables automated seam tracking with a collaborative robot. The vision-based deep learning classification model detects the tacks, where the seam is not visible. It is based on a keypoint detection deep learning model that addresses the challenges in distorted and noisy images of fillet joints between the pipes and flanges during the real-time Gas Metal Arc Welding to track the location of the seam in non tack images. The system is optimized for real time seam tracking by proposing the appropriate input image size. Multiple images and multiple points are also considered to provide a controllable signal of the location of the seam with less errors and outliers. Our proposed model can track the seam with more than 80 percent accuracy for errors less than 0.3 mm in fillet joints. The high accuracy of the proposed method would result in fewer flaws and defects and reduced rework, resulting in significant cost saving in manufacturing. The real-time monitoring also enables the adaptability to slight variations in gap.

Communication-Aware Motion Planning of AUV in Obstacle-Dense Environment: A Binocular Vision-Based Deep Learning Method

Communication-Aware Motion Planning of AUV in Obstacle-Dense Environment: A Binocular Vision-Based Deep Learning Method

Intelligent identification of fragmented non-magnetic materials for end-of-life refrigerator recycling

ABSTRACT E-waste is a valuable secondary resource containing numerous toxic substances and high-value components. If improperly handled, it will cause severe environmental pollution. Therefore, efficient recycling of this material can reduce environmental pollution. However, after crushing, fine crushing, and magnetic separation, a substantial quantity of fragmented non-magnetic materials with high value, such as copper andg aluminum, remain. Refrigerators, as typical e-waste, have a similar composition to fragmented non-magnetic materials. Consequently, this paper focuses on the issues of low efficiency, environmental pollution, and resource waste in sorting fragmented non-magnetic materials from waste refrigerators. This paper constructs a data set of fragmented non-magnetic materials of refrigerators, augments the data set, and identifies fragmented non-magnetic materials of refrigerators using a computer vision-based deep learning method. In this study, YOLOv5s is used as the benchmark model. The CBAM module is added to the backbone to enable intelligent identification and sorting of fragmented non-magnetic materials in refrigerators. The final identification efficiency of waste refrigerators meets the requirements of industrial applications, with an accuracy rate of 98.3%, a recall rate of 96.8%, and an average accuracy of 98%. Based on the similarity of the composition of e-waste fragmented materials, this model sorting method can be applied to sorting additional e-waste fragmented materials. Furthermore, it provides the theoretical foundation for promoting e-waste resourcefulness. Implications: This paper proposes a recognition model based on YOLOv5s to solve the problems of low sorting efficiency, environmental pollution, harm to health, and resource waste of non-magnetic crushed material from refrigerators. The recognition model principally addresses the following issues: a deep learning model is developed for recognition and sorting to improve e-waste recognition and sorting efficiency. Concerning the issue of environmental benefits in an ecological environment, a vision-based automatic identification method is proposed to sort harmful waste, such as foam, to preserve the ecological environment. In response to the problem of resource waste, this project improves the purity of precious metals, resulting in a recovery rate of 99.1% for copper and 96.44% for aluminum. In other words, the cost of recovering metals has increased. The identification model of non-magnetic crushed material in refrigerators satisfies production identification and sorting requirements. In addition, the method has application and promotion value, sorting a theoretical foundation and method for identifying and classifying e-waste.

Automatic spacing inspection of rebar spacers on reinforcement skeletons using vision-based deep learning and computational geometry

Improper spacing of rebar spacers (RSs) in reinforced concrete structures can lead to corrosion and damage of rebars, ultimately decreasing the strength and durability of the structure. The current method for RSs spacing inspection entails a manual inspection that utilizes a measuring tape, which is both labor-intensive and time-consuming. This paper presents a novel method for automatically inspecting RSs spacing using a vision-based deep learning and computational geometry. The proposed method consists of three modules including the RSs location and classification module, the keypoints detection module and RSs spacing inspection module. The RSs location and classification module employ object detection of faster-region convolutional neural network (Faster-RCNN)-based deep learning to accurately locate and classify RSs. The keypoints detection module uses human pose estimation of cascaded pyramid network (CPN)-based deep learning to detect the three keypoints necessary for accurate RSs spacing calculations. In RSs spacing inspection module, the computational geometry is applied to derive an accurate expression for calculating RSs spacing in the transverse and longitudinal directions, based on the RSs keypoints coordinates and camera parameters. The RSs spacing is subsequently evaluated to check its compliance. In the experiment, the algorithms of the three modules were integrated into a robot that can automatically move and take pictures to accomplish the task of automatically inspecting RSs spacing in a large reinforced concrete component plant. The experimental results indicate that this method yields an average RSs spacing measurement error of 1.35 cm. Finally, the robustness of the proposed method was validated by changing the conditions of image acquisition processes.

SViT: A Spectral Vision Transformer for the Detection of REM Sleep Behavior Disorder.

REM sleep behavior disorder (RBD) is a parasomnia with dream enactment and presence of REM sleep without atonia (RSWA). RBD diagnosed manually via polysomnography (PSG) scoring, which is time intensive. Isolated RBD (iRBD) is also associated with a high probability of conversion to Parkinson's disease. Diagnosis of iRBD is largely based on clinical evaluation and subjective PSG ratings of REM sleep without atonia. Here we show the first application of a novel spectral vision transformer (SViT) to PSG signals for detection of RBD and compare the results to the more conventional convolutional neural network architecture. The vision-based deep learning models were applied to scalograms (30 or 300 s windows) of the PSG data (EEG, EMG and EOG) and the predictions interpreted. A total of 153 RBD (96 iRBD and 57 RBD with PD) and 190 controls were included in the study and 5-fold bagged ensemble was used. Model outputs were analyzed per-patient (averaged), with regards to sleep stage, and the SViT was interpreted using integrated gradients. Models had a similar per-epoch test F1 score. However, the vision transformer had the best per-patient performance, with an F1 score 0.87. Training the SViT on channel subsets, it achieved an F1 score of 0.93 on a combination of EEG and EOG. EMG is thought to have the highest diagnostic yield, but interpretation of our model showed that high relevance was placed on EEG and EOG, indicating these channels could be included for diagnosing RBD.

딥러닝을 이용한 의류 이미지의 텍스타일 소재 분류

As online transactions increase, the image of clothing has a great influence on consumer purchasing decisions. The importance of image information for clothing materials has been emphasized, and it is important for the fashion industry to analyze clothing images and grasp the materials used. Textile materials used for clothing are difficult to identify with the naked eye, and much time and cost are consumed in sorting. This study aims to classify the materials of textiles from clothing images based on deep learning algorithms. Classifying materials can help reduce clothing production costs, increase the efficiency of the manufacturing process, and contribute to the service of recommending products of specific materials to consumers. We used machine vision-based deep learning algorithms ResNet and Vision Transformer to classify clothing images. A total of 760,949 images were collected and preprocessed to detect abnormal images. Finally, a total of 167,299 clothing images, 19 textile labels and 20 fabric labels were used. We used ResNet and Vision Transformer to classify clothing materials and compared the performance of the algorithms with the Top-k Accuracy Score metric. As a result of comparing the performance, the Vision Transformer algorithm outperforms ResNet.

Review of Vision-Based Deep Learning Parking Slot Detection on Surround View Images.

Autonomous vehicles are gaining popularity, and the development of automatic parking systems is a fundamental requirement. Detecting the parking slots accurately is the first step towards achieving an automatic parking system. However, modern parking slots present various challenges for detection task due to their different shapes, colors, functionalities, and the influence of factors like lighting and obstacles. In this comprehensive review paper, we explore the realm of vision-based deep learning methods for parking slot detection. We categorize these methods into four main categories: object detection, image segmentation, regression, and graph neural network, and provide detailed explanations and insights into the unique features and strengths of each category. Additionally, we analyze the performance of these methods using three widely used datasets: the Tongji Parking-slot Dataset 2.0 (ps 2.0), Sejong National University (SNU) dataset, and panoramic surround view (PSV) dataset, which have played a crucial role in assessing advancements in parking slot detection. Finally, we summarize the findings of each method and outline future research directions in this field.

Human Activity Recognition Using Deep Learning : A Survey

With the use of deep learning algorithms from artificial intelligence (AI), several types of research have been conducted on video data. Object localization, behaviour analysis, scene understanding, scene labelling, human activity recognition (HAR), and event recognition make up the majority of them. Among all of them, HAR is one of the most difficult jobs and key areas of research in video data processing. HAR can be used in a variety of fields, including robotics, human-computer interaction, video surveillance, and human behaviour categorization. This research seeks to compare deep learning approaches on several benchmark video datasets for vision-based human activity detection. We suggest a brand-new taxonomy for dividing up the literature into CNN- and RNN-based methods. We further categorise these approaches into four subgroups and show several methodologies, their effectiveness, and experimental datasets. To illustrate the development of HAR techniques, a brief comparison is also provided with the handcrafted feature-based approach and its merger with deep learning. Finally, we go over potential future research areas and some unresolved issues with recognising human activities. This survey's goal is to present the most recent developments in HAR techniques for vision-based deep learning using the most recent literature analysis.