Deep Learning-based Computer Vision Research Articles

A deep learning-based worker assistance system for error prevention: Case study in a real-world manual assembly

Modern assembly systems adapt to the requirements of customised and short-lived products. As assembly tasks become increasingly complex and change rapidly, the cognitive load on employees increases. This leads to the use of assistance systems for manual assembly to detect and avoid human errors and thus ensure consistent product quality. Most of these systems promise to improve the production environment but have hardly been studied quantitatively so far. Recent advances in deep learning-based computer vision have also not yet been fully exploited. This study aims to provide architectural, and implementational details of a state-of-the-art assembly assistance system based on an object detection model. The proposed architecture is intended to be representative of modern assistance systems. The error prevention potential is determined in a case study in which test subjects manually assemble a complex explosion-proof tubular lamp. The results show 51 % fewer assembly errors compared to a control group without assistance. Three of the four considered types of error classes have been reduced by at least 42 %. In particular, errors by omission are most likely to be prevented by the system. The reduction in the error rate is observed over the entire period of 30 consecutive product assemblies, comparing assisted and unassisted assembly. Furthermore, the recorded assembly data are found to be valuable regarding traceability and production improvement processes.

Foids

We present a bio-inspired fish simulation platform, which we call "Foids", to generate realistic synthetic datasets for an use in computer vision algorithm training. This is a first-of-its-kind synthetic dataset platform for fish, which generates all the 3D scenes just with a simulation. One of the major challenges in deep learning based computer vision is the preparation of the annotated dataset. It is already hard to collect a good quality video dataset with enough variations; moreover, it is a painful process to annotate a sufficiently large video dataset frame by frame. This is especially true when it comes to a fish dataset because it is difficult to set up a camera underwater and the number of fish (target objects) in the scene can range up to 30,000 in a fish cage on a fish farm. All of these fish need to be annotated with labels such as a bounding box or silhouette, which can take hours to complete manually, even for only a few minutes of video. We solve this challenge by introducing a realistic synthetic dataset generation platform that incorporates details of biology and ecology studied in the aquaculture field. Because it is a simulated scene, it is easy to generate the scene data with annotation labels from the 3D mesh geometry data and transformation matrix. To this end, we develop an automated fish counting system utilizing the part of synthetic dataset that shows comparable counting accuracy to human eyes, which reduces the time compared to the manual process, and reduces physical injuries sustained by the fish.

Computer Vision Technology Based on Sensor Data and Hybrid Deep Learning for Security Detection of Blast Furnace Bearing

It is a big challenge to realize accurate security detection of blast furnace bearing at the same time so as to guarantee the security of equipment. To end this problem, this paper proposed a computer vision technology based on sensor data and hybrid deep learning method for the solution. We use Variational Mode Decomposition (VMD) algorithm which is a new time-frequency analysis method, which can decompose multi-component signals into multiple single-component amplitude-modulated signals at one time to decompose and deal with the sensor data of bearing fault, so as to realize the effective stripping of fault components and original components from sensor data. Using the artificial intelligence mentioned above, the features can be quickly and accurately extracted. By combining the advantages of deep learning, we improve the coupling mechanism and implement a hybrid deep learning-based computer vision method which greatly improves the calculation speed and accuracy of bearing fault diagnosis. It can be fully connected with the feature extraction algorithm VMD, which overcomes the problem that the bearing feature component is easy to be submerged and difficult to extract under the condition of high temperature and strong noise. The results show that the optimal selection of parameters of computer vision technology based on sensor data and hybrid deep learning can be realized through training the sensor data obtained from the experiment. The optimized hybrid deep learning-based computer vision algorithm can achieve 97.4% bearing fault diagnosis hit rate, which is an advanced application of deep learning algorithm in the engineering field.

Automatic Asbestos Control Using Deep Learning Based Computer Vision System

The paper discusses the results of the research and development of an innovative deep learning-based computer vision system for the fully automatic asbestos content (productivity) estimation in rock chunk (stone) veins in an open pit and within the time comparable with the work of specialists (about 10 min per one open pit processing place). The discussed system is based on the applying of instance and semantic segmentation of artificial neural networks. The Mask R-CNN-based network architecture is applied to the asbestos-containing rock chunks searching images of an open pit. The U-Net-based network architecture is applied to the segmentation of asbestos veins in the images of selected rock chunks. The designed system allows an automatic search and takes images of the asbestos rocks in an open pit in the near-infrared range (NIR) and processes the obtained images. The result of the system work is the average asbestos content (productivity) estimation for each controlled open pit. It is validated to estimate asbestos content as the graduated average ratio of the vein area value to the selected rock chunk area value, both determined by the trained neural network. For both neural network training tasks the training, validation, and test datasets are collected. The designed system demonstrates an error of about 0.4% under different weather conditions in an open pit when the asbestos content is about 1.5–4%. The obtained accuracy is sufficient to use the system as a geological service tool instead of currently applied visual-based estimations.

Empowering Knowledge Distillation via Open Set Recognition for Robust 3D Point Cloud Classification

• We propose a joint training methodology for knowledge distillation and open set recognition to improve model robustness. • We demonstrate how knowledge distillation (KD) and open set recognition (OSR) can be performed on 3D point cloud data • The proposed method provides a much smaller model which additionally has enhanced OSR capabilities than the original model. • We find that KD by itself can transfer OSR ability to a student model apart from previously known dark knowledge transfer. • We show how there is a trade-off that comes into play for the student model’s open-set and closed-set performance. Real-world scenarios pose several challenges to deep learning based computer vision techniques despite their tremendous success in research. Deeper models provide better performance, but are challenging to deploy and knowledge distillation allows us to train smaller models with minimal loss in performance. A model also has to deal with open set samples from classes outside the ones it was trained on and should be able to identify them as unknown samples while classifying the known ones correctly. Finally, most existing image recognition research focuses only on using two-dimensional snapshots of the three-dimensional real world objects. In this work, we attempt to bridge these three research fields, which have been developed independently until now, despite being deeply interrelated in practice. We propose a joint knowledge distillation and open set recognition training methodology for three-dimensional object recognition. We demonstrate the effectiveness of the proposed method via various experiments on how it allows us to obtain a much smaller model, which takes a minimal hit in performance while being capable of open set recognition for 3D point cloud data.

A Video Analytic In-Class Student Concentration Monitoring System

Automatic learning feedback monitoring and analysis are becoming essential in modern education. We present a video analytic system capable of monitoring in-class student’s learning behaviors and providing feedback to the instructor. It is a common practice nowadays for students to take electronic notes or browse online using laptops and cellphones in class. However the use of technology can also impact student concentration and affect learning behaviors, which can seriously hinder their learning progress if not controlled properly. In this pioneering study, we propose a non-intrusive deep-learning based computer vision system to monitor student concentration by extracting and inferring high-level visual behavior cues, including their facial expressions, gestures and activities. Our system can automatically assist instructors with situational awareness in real time. We assume only RGB color images as input and runable system on edge devices for easy deployment. We propose two video analytic components for student behavior analysis: (1) The facial analysis component operates based on Dlib face detection and facial landmark tracking to localize each student and analyze their face orientations, eye blinking, gazes, and facial expressions. (2) The activity detection and recognition component operates based on OpenPose and COCO object detection can identify eight types of in-class gestures and behaviors including raising-hand, typing, phone-answering, crooked-head, desk napping, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">etc.</i> Experiments are performed on a newly collected real-world In-Class Student Activity Dataset (ICSAD), where we achieved nearly 80% activity detection rate. Our system is view-independent in handling facial and pose orientations with average angular error < 10°. The source code of this work is at: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/YiZengHsieh/ICSAD</uri> .

Shared Realities: Avatar Identification and Privacy Concerns in Reconstructed Experiences

Recent advances in 3D reconstruction technology allow people to capture and share their experiences in 3D. However, little is known about people's sharing preferences and privacy concerns for these reconstructed experiences. To fill this gap, we first present ReliveReality, an experience-sharing method utilizing deep learning-based computer vision techniques to reconstruct clothed humans and 3D environments and estimate 3D pose with only a RGB camera. ReliveReality can be integrated into social virtual environments, allowing others to socially relive a shared experience by moving around the experience from different perspectives, on desktop or in VR. We conducted a 44-participant within-subject study to compare ReliveReality to viewing recorded videos, and to a ReliveReality version with blurring obfuscation. Our results shed light on how people identify with reconstructed avatars, how obfuscation affects reliving experiences, and sharing preferences and privacy concerns for reconstructed experiences. We propose design implications for addressing these issues.

Machine vision based flood monitoring system using deep learning techniques and fuzzy logic on crowdsourced image data

In recent times, frequent occurrences of natural disasters have been the cause of widespread disruptions to life and property. Albeit attempts to prevent such disasters may be a lost cause, emerging technologies can be resorted to, for minimization of their impact. This study proposes a deep learning-based computer vision and crowdsourcing methodology for the detection and estimation of flood depths, one of the most intense disruptive disasters. State-of-the-art flood detection systems work off of satellite or radar images. This research deals with processing images, captured at random, from flood ravaged zones, by smartphones or digital cameras. The crowdsourced image collection of the flood scenes afford better coverage and diverse perspectives, for assessments of the flood devastation. This paper proffers a fuzzy logic-based algorithm, and image segmentation based on color, to estimate the extent of flooding by analysis of crowdsourced images. Deployment of these methods helps in classification of the flooded areas into high, medium, or low level of flooding, to facilitate cost-effective, time-critical rescue operations. This algorithm yielded an accuracy of 83.1% on our dataset.

Deep Learning-based Computer Vision for Radiation Defect Analysis: from Static Defect Segmentation to Dynamic Defect Tracking

Deep Learning-based Computer Vision for Radiation Defect Analysis: from Static Defect Segmentation to Dynamic Defect Tracking

Automatic identification and quantification of dense microcracks in high-performance fiber-reinforced cementitious composites through deep learning-based computer vision

High-performance fiber-reinforced cementitious composites (HPFRCCs) feature high mechanical strengths, crack resistance, and durability. Under excessive loading, HPFRCCs demonstrate dense microcracks that are difficult to identify using existing methods. This study presents a computer vision method for identification, quantification, and visualization of microcracks in HPFRCCs based on deep learning. The presented method integrates multiple deep learning models and computer vision techniques in a hierarchical architecture. The crack pattern (e.g., number, width, and spacing of cracks) are automatically determined from pictures without human intervention. This study shows that the presented method achieves an accuracy of 0.992 for crack detection and an accuracy finer than 50 μm (R2 > 0.984) for quantification of crack width when deep learning models are trained using only 200 pictures of HPFRCCs and 200 pictures of conventional concrete with incorporation of data augmentation. The presented method is expected to be also applicable to other materials featuring complex cracks.

Real-Time Surveillance System for Analyzing Abnormal Behavior of Pedestrians

In spite of excellent performance of deep learning-based computer vision algorithms, they are not suitable for real-time surveillance to detect abnormal behavior because of very high computational complexity. In this paper, we propose a real-time surveillance system for abnormal behavior analysis in a closed-circuit television (CCTV) environment by constructing an algorithm and system optimized for a CCTV environment. The proposed method combines pedestrian detection and tracking to extract pedestrian information in real-time, and detects abnormal behaviors such as intrusion, loitering, fall-down, and violence. To analyze an abnormal behavior, it first determines intrusion/loitering through the coordinates of an object and then determines fall-down/violence based on the behavior pattern of the object. The performance of the proposed method is evaluated using an intelligent CCTV data set distributed by Korea Internet and Security Agency (KISA).

To what extent can a computer vision system identify pathology with limited visual fields?

Deep learning based computer vision systems are used in many fields for a variety of classification and identification tasks. They are composed of a network that learns features, rather than traditional hand-curated rules. The diagnosis of human pathology is primarily based on the visual features in histology slides. These slides are prepared by sampling a small amount of the original tissue and interpretation is sometimes limited by paucity of visual material. To what extent can computer systems, which create their own visual feature map, be used to make diagnoses where visual information is limited? We present our findings in the context of renal biopsies.

How the use of the artificial intelligence could improve surgical skills in urology: state of the art and future perspectives.

As technology advances, surgical training has evolved in parallel over the previous decade. Training is commonly seen as a way to prepare surgeons for their day-to-day work; however, more importantly, it allows for certification of skills to ensure maximum patient safety. This article reviews advances in the use of machine learning and artificial intelligence for improvements of surgical skills in urology. Six studies have been published, which met the inclusion criteria. All articles assessed the application of artificial intelligence in improving surgical training. Different approaches were taken, such as using machine learning to identify and classify suturing gestures, creating automated objective evaluation reports, and determining surgical technical skill levels to predict clinical outcomes. The articles illustrated the continuously growing role of artificial intelligence to address the difficulties currently present in evaluating urological surgical skills. Artificial intelligence allows us to efficiently analyze the surmounting data related to surgical training and use it to come to conclusions that normally would require human intelligence. Although these metrics have been shown to predict surgeon expertise and surgical outcomes, evidence is still scarce regarding their ability to directly improve patient outcomes. Considering this, current active research is growing on the topic of deep learning-based computer vision to provide automated metrics needed for real-time surgeon feedback.

Virtual Anatomist: A Deep Learning‐based Smartphone Application to Identify Complex Anatomical Features in Augmented Reality

This study pertains to the development of a smartphone mixed-reality (MR) educational app intended to improve the experience of using physical 3D models in classrooms by identifying and labeling various anatomical features on models, built on a deep-learning based computer vision framework. Research at the intersection of MR applications and anatomy education has routinely demonstrated a role for new MR-based modalities in improving anatomy education, but most MR apps rely on custom illustrated projections of 3D-models into user and screen space. These virtual assets are subject to device-intrinsic or developer-based differences in display fidelity and specimen art quality. An intrinsic barrier is present in the development of digital 3D models, which are not trivial to create. Existing evidence also suggests that virtual models may produce inferior results for learning in some use-cases compared to existing physical models. Such evidence forms a case to instead place emphasis on improving the experience of using existing physical models. The described application is hence intended to improve the experience of using real models by labeling anatomical features of interest for the user. The current implementation of the application is trained solely on skull-base anatomy (with class labels including selected bones of the calvarium, paranasal sinuses, and skull processes), but may be extended to other anatomical areas of interest. This labeling is made possible by a hybrid depth-estimation and semantic segmentation-focused machine learning (ML) architecture, which is deployed on consumer-grade smartphones to promote student uptake. When creating ML-based tools, the primary barrier is often the generation of quality ground truth data. Image collections of anatomical specimens must ideally be taken under different conditions, with different augmentations applied to images to improve the ability of the application to robustly recognize and label different parts of a specimen. Manual collection and annotation of the hundreds or thousands of such images required for training is infeasible. However, using procedurally generated images from a 3D-modelled skull (here, in the open-source computer graphics software Blender), developers can produce arbitrarily large, photorealistic ground-truth training datasets with pixel-perfect semantic segmentation of anatomical features. The generalizability of the ML classifier to different models was improved through augmentation of individual renders in Blender by randomizing the model textures; lighting; background environments; skull topology via displacement mapping; the use of “distractor” objects in renders; and camera angles. This work demonstrates the feasibility of developing an anatomical landmark classifier from RGB-image data, trained on fully synthetic data. Future steps include optimization of data augmentations to emphasize shape recognition over texture recognition, formal characterization of segmentation accuracy on cadaveric specimens, and to train alternative models to incorporate depth data, to leverage depth-sensing capabilities that are available on select higher-end mobile devices.

Review of computer vision application in in vitro fertilization: the application of deep learning-based computer vision technology in the world of IVF.

In vitro fertilization has been regarded as a forefront solution in treating infertility for over four decades, yet its effectiveness has remained relatively low. This could be attributed to the lack of advancements for the method of observing and selecting the most viable embryos for implantation. The conventional morphological assessment of embryos exhibits inevitable drawbacks which include time- and effort-consuming, and imminent risks of bias associated with subjective assessments performed by individual embryologists. A combination of these disadvantages, undeterred by the introduction of the time-lapse incubator technology, has been considered as a prominent contributor to the less preferable success rate of IVF cycles. Nonetheless, a recent surge of AI-based solutions for tasks automation in IVF has been observed. An AI-powered assistant could improve the efficiency of performing certain tasks in addition to offering accurate algorithms that behave as baselines to minimize the subjectivity of the decision-making process. Through a comprehensive review, we have discovered multiple approaches of implementing deep learning technology, each with varying degrees of success, for constructing the automated systems in IVF which could evaluate and even annotate the developmental stages of an embryo.

Stabilized adaptive sampling control for reliable real-time learning-based surveillance systems

In modern security systems such as CCTV-based surveillance applications, real-time deep-learning based computer vision algorithms are actively utilized for always-on automated execution. The real-time computer vision system for surveillance applications is highly computation-intensive and exhausts computation resources when it performed on the device with a limited amount of resources. Based on the nature of Internet-of-Things networks, the device is connected to main computing platforms with offloading techniques. In addition, the real-time computer vision system such as the CCTV system with image recognition functionality performs better when arrival images are sampled at a higher rate because it minimizes missing video frame feeds. However, performing it at overwhelmingly high rates exposes the system to the risk of a queue overflow that hampers the reliability of the system. In order to deal with this issue, this paper proposes a novel queue-aware dynamic sampling rate adaptation algorithm that optimizes the sampling rates to maximize the computer vision performance(i.e., recognition ratio) while avoiding queue overflow under the concept of Lyapunov optimization framework. Through extensive system simulations, the proposed approaches are shown to provide remarkable gains.

Deep Learning-Based Computer Vision for Surveillance in ITS: Evaluation of State-of-the-Art Methods

Intelligent transportation system (ITS) collects numerous data for analysis of the transportation system. The data can be used for providing services for travellers and traffic controllers in the ITS and optimizing it, for the purpose of making the transportation more efficient and safer. Due to the wide and flexible employment of video cameras in visual surveillance system (VSS), mature edge-cloud resource scheduling for data transmission and analysis, and the fast development of deep learning, computer vision (CV) methods have been employed in the visual-based ITS services successfully. In this paper, we discuss the edge-cloud surveillance resource scheduling for the CV methods and review the deep learning-based CV methods in the VSS, including detection, classification, and tracking methods, for better understanding of the relationship between the CV-based ITS services and these methods. We experimentally compare several state-of-the-art deep learning-based methods, which have been successfully applied in the CV fields under the ITS scenario, on their performance, inference speed, computational quantity, and model size. According to the comparisons, we propose four main challenges of the deep learning-based CV methods applied in the services, as a discussion of the future research directions. Code are available at https://github.com/PRIS-CV/DL-CV-ITS .

A Training Method for Low Rank Convolutional Neural Networks Based on Alternating Tensor Compose-Decompose Method

Over the past decade, deep learning-based computer vision methods have been shown to surpass previous state-of-the-art computer vision techniques in various fields, and have made great progress in various computer vision problems, including object detection, object segmentation, face recognition, etc. Nowadays, major IT companies are adding new deep-learning-based computer technologies to edge devices such as smartphones. However, since the computational cost of deep learning-based models is still high for edge devices, research is being actively carried out to compress deep learning-based models while not sacrificing high performance. Recently, many lightweight architectures have been proposed for deep learning-based models which are based on low-rank approximation. In this paper, we propose an alternating tensor compose-decompose (ATCD) method for the training of low-rank convolutional neural networks. The proposed training method can better train a compressed low-rank deep learning model than the conventional fixed-structure based training method, so that a compressed deep learning model with higher performance can be obtained in the end of the training. As a representative and exemplary model to which the proposed training method can be applied, we propose a rank-1 convolutional neural network (CNN) which has a structure alternatively containing 3-D rank-1 filters and 1-D filters in the training stage and a 1-D structure in the testing stage. After being trained, the 3-D rank-1 filters can be permanently decomposed into 1-D filters to achieve a fast inference in the test time. The reason that the 1-D filters are not being trained directly in 1-D form in the training stage is that the training of the 3-D rank-1 filters is easier due to the better gradient flow, which makes the training possible even in the case when the fixed structured network with fixed consecutive 1-D filters cannot be trained at all. We also show that the same training method can be applied to the well-known MobileNet architecture so that better parameters can be obtained than with the conventional fixed-structure training method. Furthermore, we show that the 1-D filters in a ResNet like structure can also be trained with the proposed method, which shows the fact that the proposed method can be applied to various structures of networks.

A Shape Transformation-based Dataset Augmentation Framework for Pedestrian Detection

Deep learning-based computer vision is usually data-hungry. Many researchers attempt to augment datasets with synthesized data to improve model robustness. However, the augmentation of popular pedestrian datasets, such as Caltech and Citypersons, can be extremely challenging because real pedestrians are commonly in low quality. Due to the factors like occlusions, blurs, and low-resolution, it is significantly difficult for existing augmentation approaches, which generally synthesize data using 3D engines or generative adversarial networks (GANs), to generate realistic-looking pedestrians. Alternatively, to access much more natural-looking pedestrians, we propose to augment pedestrian detection datasets by transforming real pedestrians from the same dataset into different shapes. Accordingly, we propose the Shape Transformation-based Dataset Augmentation (STDA) framework. The proposed framework is composed of two subsequent modules, i.e. the shape-guided deformation and the environment adaptation. In the first module, we introduce a shape-guided warping field to help deform the shape of a real pedestrian into a different shape. Then, in the second stage, we propose an environment-aware blending map to better adapt the deformed pedestrians into surrounding environments, obtaining more realistic-looking pedestrians and more beneficial augmentation results for pedestrian detection. Extensive empirical studies on different pedestrian detection benchmarks show that the proposed STDA framework consistently produces much better augmentation results than other pedestrian synthesis approaches using low-quality pedestrians. By augmenting the original datasets, our proposed framework also improves the baseline pedestrian detector by up to 38% on the evaluated benchmarks, achieving state-of-the-art performance.

Robust pixel-wise concrete crack segmentation and properties retrieval using image patches

Crack identification is an essential task in periodic inspection and maintenance of buildings. The application of deep learning based computer vision techniques is increasingly popular, but suffer from challenges of insufficient performance on highly diverse field inspection scenarios as well as a requirement for large amounts of labeled training data. To address these limitations, this paper proposes a robust crack segmentation approach using image patches to detect and support further accurate retrieval of crack properties for integrity assessment. In the proposed approach, a local region-based active contour model is integrated with a convolution neural network and several post-processing morphological operations to derive a segmented crack map. Experimental validation shows significant improvement in terms of accuracy and robustness over previous work. Data labeling requirement is also comparatively lower. This paper enhances the current concrete inspection process, and lays the foundation for more data efficient methods of crack segmentation to be explored.