Vision-based Approach Research Articles

Toward a sustainable transition in automated production: enabling the vision-based approach and synchronous control for textile surface inspection systems

The textile and apparel industry is one of the largest industries in the world, with significant environmental and social impacts, such as manual labor in poor working conditions and resource depletion and environmental degradation due to ineffective production. As a result, there is an increasing need for sustainable practices and a transition toward a more circular and responsible industry. In this paper, we introduce the proposed approach in the context of a sustainable transition for the industrial textiles. Instead of human labor, the automated solution to detect defects is proposed for the 4.0 revolution in general and the textile industry in particular. Specifically, the mechanical design of a three-dimensional model for small- and medium-sized fabric rolls is portrayed. Some computational mechanics are created to ensure a stable mechanism and proper actuators. For acquiring accurate data and measuring the machine state, several installations, that is, a digital camera, load cell, and positioning sensor, and the layout of the light source are improved to intensify the image quality. The control theme, such as the cross-coupling controller, offers superior performance in synchronization and vibration-less motion. Lastly, the excellent detection method is provided by the convolutional neural network scheme for offline training to distinguish defects on products. Instead of a manual check or low rate of defect detection on products, the contributions of this research are as follows: (a) analyzing the mechanical design of the textile machining system in terms of dynamic characteristics; (b) implementing a smooth scheme based on motion control synchronization; and (c) integrating various strategies, including vision-based computation and control topology, to demonstrate superior performance. These results highlight that our approach can be widely adapted to address sustainability challenges not only in the textile field but also in global manufacturing practices.

Deep LiDAR-Radar-Visual Fusion for Object Detection in Urban Environments

Robust environmental sensing and accurate object detection are crucial in enabling autonomous driving in urban environments. To achieve this goal, autonomous mobile systems commonly integrate multiple sensor modalities onboard, aiming to enhance accuracy and robustness. In this article, we focus on achieving accurate 2D object detection in urban autonomous driving scenarios. Considering the occlusion issues of using a single sensor from a single viewpoint, as well as the limitations of current vision-based approaches in bad weather conditions, we propose a novel multi-modal sensor fusion network called LRVFNet. This network effectively combines data from LiDAR, mmWave radar, and visual sensors through a deep multi-scale attention-based architecture. LRVFNet comprises three modules: a backbone responsible for generating distinct features from various sensor modalities, a feature fusion module utilizing the attention mechanism to fuse multi-modal features, and a pyramid module for object reasoning at different scales. By effectively fusing complementary information from multi-modal sensory data, LRVFNet enhances accuracy and robustness in 2D object detection. Extensive evaluations have been conducted on the public VOD dataset and the Flow dataset. The experimental results demonstrate the superior performance of our proposed LRVFNet compared to state-of-the-art baseline methods.

Domain Knowledge Distillation and Supervised Contrastive Learning for Industrial Process Monitoring

To ensure the reliability and safety of modern industrial process monitoring, computer vision-based soft measurement has received considerable attention due to its nonintrusive property. State-of-the-art computer vision-based approaches mostly rely on feature embedding from deep neural networks. However, this kind of feature extraction suffers from noise effects and limitation of labeled training instances, leading to unsatisfactory performance in real industrial process monitoring. In this paper, we develop a novel hybrid learning framework for feature representation based on knowledge distillation and supervised contrastive learning. Firstly, we attempt to transfer the abundant semantic information in handcrafted features to deep learning feature-based network by knowledge distillation. Then, to enhance the feature discrimination, supervised contrastive learning is proposed to contrast many positive pairs against many negative pairs per anchor. Meanwhile, two important mechanisms, memory queue-based negative sample augmentation and hard negative sampling, are added into the supervised contrastive learning model to assist the proper selection of negative samples. Finally, a flotation process monitoring problem is considered to illustrate and demonstrate the effectiveness of the proposed method.

Systematic Configuration for Hyperparameters Optimization in Transferring of CNN Model to Disaster Events Classification from UAV Images

Deep learning and computer vision-based approaches incorporated with the evolution of the relevant technologies of Unmanned Aerial Vehicles (UAVs) and drones have significantly motivated the advancements of disaster management applications. This research studied a classification method for disaster event identification from UAV images that is suitable for disaster monitoring. A Convolution Neural Network (CNN) of GoogleNet models that were pretrained from ImageNet and Place365 datasets was explored to find the appropriate one for fine-tuning to classify the disaster events. In order to get the optimal performance, a systematic configuration for searching the hyperparameters in fine-tuning the CNN model was proposed. The top three hyperparameters that affect the performance, which are the initial learning rate, the number of epochs, and the minibatch size, were systematically set and tuned for each configuration. The proposed approach consists of five stages, during which three types of trials were used to monitor different sets of the hyperparameters. The experimental result revealed that by applying the proposed approach the model performance can increase up to 5%. The optimal performance achieved was 98.77 percent accuracy. For UAV/drone applications, where a small onboard model is preferred, GoogleNet that is quite small in model size and has a good structure for further fine tuning is suitable to deploy.

Quantifying Soil Particle Settlement Characteristics through Machine Vision Analysis Utilizing an RGB Camera

Soil particle size distribution is a crucial factor in determining soil properties and classifying soil types. Traditional methods, such as hydrometer tests, have limitations in terms of time required, labor, and operator dependency. In this paper, we propose a novel approach to quantify soil particle size analysis using machine vision analysis with an RGB camera. The method aims to overcome the limitations of traditional techniques by providing an efficient and automated analysis of fine-grained soils. It utilizes a digital camera to capture the settling properties of soil particles, eliminating the need for a hydrometer. Experimental results demonstrate the effectiveness of the machine vision-based approach in accurately determining soil particle size distribution. The comparison between the proposed method and traditional hydrometer tests reveals strong agreement, with an average deviation of only 2.3% in particle size measurements. This validates the reliability and accuracy of the machine vision-based approach. The proposed machine vision-based analysis offers a promising alternative to traditional techniques for assessing soil particle size distribution. The experimental results highlight its potential to revolutionize soil particle size analysis, providing precise, efficient, and cost-effective analysis for fine-grained soils.

Spatial displacement tracking of vibrating structure using multiple feature points assisted binocular visual reconstruction

Current vision-based displacement measurement methods have limitations of requiring manual target and parameter adjustment, along with significant user involvement to achieve the desired results. This paper proposes an advanced and comprehensive vision system based on ZED cameras for full-field three-dimensional (3D) vibration displacement measurement of civil structures. Compared to existing binocular systems consisting of multiple monoculars, ZED binoculars offer greater flexibility on instrument setup and data acquisition. A novel keypoint detection and matching algorithm based on deep learning is used to achieve target-free measurement, which is capable of capturing feature points with high precision. The performance of the proposed vision-based approach is evaluated through experimental studies on a six-story scaled frame structure. Statistical analysis of the comparative results in sinusoidal cases demonstrates that the displacement responses obtained from the proposed vision-based approach show consistent results in compliance with displacement sensor measurements. System identification of the target structure is achieved through a seismic case and the error analysis is also concluded in detail.

Precision landing comparison between smartphone video guidance sensor and IRLock by hardware-in-the-loop emulation

The Smartphone Video Guidance Sensor (SVGS) is an emerging technology developed by NASA Marshall Space Flight Center that uses a vision-based approach to accurately estimate the six-state position and orientation vectors of an illuminated target of known dimensions with respect to a coordinate frame fixed to the camera. SVGS is a software-based sensor that can be deployed using a host platform’s resources (CPU and camera) for proximity operations and formation flight of drones or spacecraft. The SVGS output is calculated based on photogrammetric analysis of the light blobs in each image; its accuracy in linear and angular motion at different velocities has previously been successfully demonstrated [9]. SVGS has several potential applications in guidance, navigation, motion control, and proximity operations as a reduced-cost, compact, reliable option to competing technologies such as LiDAR or infrared sensing. One of the applications envisioned by NASA for SVGS is planetary/lunar autonomous landing. This paper aims to compare the SVGS performance in autonomous landing with existing technologies: a combination of infrared beacon technology (IRLock) and LiDAR. The comparison is based on a hardware-in-the-loop emulation of a precision landing experiment using a computer-controlled linear motion to emulate the approach motion and the ROS/Gazebo environment to emulate the response of the flight controller to the environment during landing. Results suggest that SVGS performs better than the existing IRLock with LiDAR sensor combination.

Computer Vision-Based Approach for Breast Cancer Rehabilitation Evaluation: A Survey

The number of breast cancer survivors living several decades after their diagnosis is increasing, which means there is a greater need for effective rehabilitation programs. While solid evidence suggests that safe exercise can improve quality of life and reduce the side effects of cancer treatments, recent research has revealed that patients may struggle to perform suggested physical exercises, particularly those in home-based rehabilitation programs. Most patients do not meet recommended levels of activity due to a lack of confidence in safety and a lack of supervision and evaluation. As a result, computer vision-based approaches have increased use for monitoring and evaluating exercise performance. Reliable motion capture sensors with advanced machine learning capabilities provide opportunities for systematic and standardized evaluation systems. This survey explores the literature on computer vision-based approaches to rehabilitation evaluation systems, including data collection with motion capture sensors and public datasets, feature extraction and representation, and feature comparison for evaluation. The study also reviews existing rehabilitation systems by comparing their data collection methods and findings. Additionally, the paper discusses challenges and recommendations related to this topic for further research.

Image-based Gait Spatiotemporal Parameters Estimation using a Single Camera and CNN-Transformer Hybrid Network.

Vision-based gait analysis can play an important role in the remote and continuous monitoring of the elderly's health conditions. However, most vision-based approaches compute gait spatiotemporal parameters using human pose information and provide average parameters. This study aimed to propose a straightforward method for stride-by-stride gait spatiotemporal parameters estimation. A total of 160 elderly individuals participated in this study. Data were gathered with a GAITRite system and a mobile camera simultaneously. Three deep learning networks were trained with a few RGB frames as input and a continuous 1D signal containing both spatial and temporal gait parameters as output. The trained networks estimated the stride lengths with correlations of 0.938 and more and detected gait events with F1-scores of 0.914 and more.Clinical relevance- The proposed method showed excellent agreements with the GAITRite system in analyzing spatiotemporal gait parameters. Our approach can be applied to monitor the elderly's health conditions based on their gait parameters for early diagnosis of diseases, proper treatment, and timely intervention.

Automated Microsurgical Tool Categorization using a Surface-Based EMG System.

The selection of micro-surgical tools and their usage form a critical component in today's surgical outcome. The success or failure of the surgery depends on how effectively the surgeon has applied the tools on the operating patient. Popular surgical tools, including forceps, clamp applicators, micro-scissors, and needle holders, are typically used in operating procedures, and characterising the same set of tools in conjunction with the surgeon's implementation remains a topic of continued interest. There have been several computer vision-based approaches to segment and detect the tools used in the surgery, where the microscopic recordings are evaluated, however surgeon's haptic feedback and the subtle variation drawn is not evidently reported. The video-based detection system also suffers from the problem of masking or shadowing critical information while in use. Hence, efforts to extract localised information during the surgery are very valuable in addition to the video-acquired outcome. The surface-based electromyography (sEMG) system offers electro-muscular signals at the skin that characterise the actions performed. The work attempts to evaluate the adoption of sEMG signals for the detection of micro-surgical tools. In this work, a pilot study of handling micro-surgical tools and classifying the same using two-channel sEMG signals from the machine learning (ML) model is performed. A two-channel sEMG acquisition system connected to the electrodes was designed and used to build a staged and supervised data set consisting of operating surgical tools. Five hand-crafted features, each from an individual channel, were employed and utilised to design an accurate model. An accuracy of 97.43% was achieved for running the ANN model on sEMG signals to classify five surgical tools based on their press-and-release action. The use of sEMG signals for tool detection is a step towards the development of tool characterisation and the assessment of surgeons' skills.Clinical relevance- A two-channel sEMG signal acquisition system to detect micro-surgical tool usage was developed. Mapping the surgery outcome with the tool handling characteristics will be a valuable addition to the existing video-based assessment of the surgery. The subtle tool handling characteristics of the surgeon are effective in improving the overall outcome of the surgery, and it is possible to extract this information only from the proposed approach.

A Vision-Based Human Digital Twin Modeling Approach for Adaptive Human–Robot Collaboration

Abstract Human–robot collaboration (HRC) has been identified as a highly promising paradigm for human-centric smart manufacturing in the context of Industry 5.0. In order to enhance both human well-being and robotic flexibility within HRC, numerous research efforts have been dedicated to the exploration of human body perception, but many of these studies have focused only on specific facets of human recognition, lacking a holistic perspective of the human operator. A novel approach to addressing this challenge is the construction of a human digital twin (HDT), which serves as a centralized digital representation of various human data for seamless integration into the cyber-physical production system. By leveraging HDT, performance and efficiency optimization can be further achieved in an HRC system. However, the implementation of visual perception-based HDT remains underreported, particularly within the HRC realm. To this end, this study proposes an exemplary vision-based HDT model for highly dynamic HRC applications. The model mainly consists of a convolutional neural network that can simultaneously model the hierarchical human status including 3D human posture, action intention, and ergonomic risk. Then, on the basis of the constructed HDT, a robotic motion planning strategy is further introduced with the aim of adaptively optimizing the robotic motion trajectory. Further experiments and case studies are conducted in an HRC scenario to demonstrate the effectiveness of our approach.

Enhancing CSI-Based Human Activity Recognition by Edge Detection Techniques

Human Activity Recognition (HAR) has been a popular area of research in the Internet of Things (IoT) and Human–Computer Interaction (HCI) over the past decade. The objective of this field is to detect human activities through numeric or visual representations, and its applications include smart homes and buildings, action prediction, crowd counting, patient rehabilitation, and elderly monitoring. Traditionally, HAR has been performed through vision-based, sensor-based, or radar-based approaches. However, vision-based and sensor-based methods can be intrusive and raise privacy concerns, while radar-based methods require special hardware, making them more expensive. WiFi-based HAR is a cost-effective alternative, where WiFi access points serve as transmitters and users’ smartphones serve as receivers. The HAR in this method is mainly performed using two wireless-channel metrics: Received Signal Strength Indicator (RSSI) and Channel State Information (CSI). CSI provides more stable and comprehensive information about the channel compared to RSSI. In this research, we used a convolutional neural network (CNN) as a classifier and applied edge-detection techniques as a preprocessing phase to improve the quality of activity detection. We used CSI data converted into RGB images and tested our methodology on three available CSI datasets. The results showed that the proposed method achieved better accuracy and faster training times than the simple RGB-represented data. In order to justify the effectiveness of our approach, we repeated the experiment by applying raw CSI data to long short-term memory (LSTM) and Bidirectional LSTM classifiers.

Simulation-based decision support system for earthmoving operations using computer vision

A reliable Decision Support System (DSS), particularly in the construction domain, can be driven by quality input information. Although vision-based methods have been widely utilized to retrieve contextual information, their potential is not fully leveraged in construction simulation yet. This study introduces an automated framework that utilizes multi-view video footage for vision-based input modeling within simulation domains. The proposed framework addresses project uncertainties (e.g., equipment performance, operators’ skills, road network, and weather status) using a proactive approach where project task durations are modeled as probabilistic distributions. The modeled distributions are continuously calibrated using the Markov Chain Monte Carlo Bayesian Inference (MCMCBI) approach. A simulation-based Simulated Annealing (SA) optimization is also employed to provide an efficient resource assignment. The extracted vision-based data is validated statistically against actual and spatiotemporal data. The results demonstrate that the suggested vision-based approach can provide qualified DSS input. Statistical analysis also confirms that vision-based data is more consistent with actual data than spatiotemporal data. The presented approach is successfully applied to an actual case study of a large-scale earthmoving project.

Fiducial Markers Aided Position Estimation for Vertical Landing

Autonomous landing is a critical step in unmanned aerial vehicles (UAVs) and requires accurate position information. In cases where GPS signals are unavailable or obstructed, vision-based approaches can provide support for landing capabilities. In this study, a vision-based position estimation algorithm is being developed in conjunction with markers used in vertical take-off and landing (VTOL) systems of UAVs. The developed framework is designed to be compatible with various types of visual markers. The Kalman Filter is used to the calculated position to correct measurement errors and reduce the uncertainty of the position estimation. The developed algorithm is extensively tested in a simulation environment. The positions of a quadrotor aircraft are compared with real measurements to analyze the performance of the proposed vision-based position estimation algorithm. The results demonstrate an acceptable level of accuracy for the algorithm. This study discusses the potential of using visual markers and integrating Kalman filtering to improve the accuracy of positioning in vertical takeoff and landing systems of the UAVs. The development of a vision-based position estimation algorithm can enhance the reliability and precision of autonomous landing capabilities and enable successful landings in situations where GPS signals are limited or unavailable.

Identification and Tracking of Vehicles between Multiple Cameras on Bridges Using a YOLOv4 and OSNet-Based Method

The estimation of vehicle loads is a rising research hotspot in bridge structure health monitoring (SHM). Traditional methods, such as the bridge weight-in-motion system (BWIM), are widely used but they fail to record the locations of vehicles on the bridges. Computer vision-based approaches are promising ways for vehicle tracking on bridges. Nevertheless, keeping track of vehicles from the video frames of multiple cameras without an overlapped visual field poses a challenge for the tracking of vehicles across the whole bridge. In this study, a method that was You Only Look Once v4 (YOLOv4)- and Omni-Scale Net (OSNet)-based was proposed to realize vehicle detecting and tracking across multiple cameras. A modified IoU-based tracking method was proposed to track a vehicle in adjacent video frames from the same camera, which takes both the appearance of vehicles and overlapping rates between the vehicle bounding boxes into consideration. The Hungary algorithm was adopted to match vehicle photos in various videos. Moreover, a dataset with 25,080 images of 1727 vehicles for vehicle identification was established to train and evaluate four models. Field validation experiments based on videos from three surveillance cameras were conducted to validate the proposed method. Experimental results show that the proposed method has an accuracy of 97.7% in terms of vehicle tracking in the visual field of a single camera and over 92.5% in tracking across multiple cameras, which can contribute to the acquisition of the temporal–spatial distribution of vehicle loads on the whole bridge.

A Robust and Automated Vision-Based Human Fall Detection System Using 3D Multi-Stream CNNs with an Image Fusion Technique

Unintentional human falls, particularly in older adults, can result in severe injuries and death, and negatively impact quality of life. The World Health Organization (WHO) states that falls are a significant public health issue and the primary cause of injury-related fatalities worldwide. Injuries resulting from falls, such as broken bones, trauma, and internal injuries, can have severe consequences and can lead to a loss of mobility and independence. To address this problem, there have been suggestions to develop strategies to reduce the frequency of falls, in order to decrease healthcare costs and productivity loss. Vision-based fall detection approaches have proven their effectiveness in addressing falls on time, which can help to reduce fall injuries. This paper introduces an automated vision-based system for detecting falls and issuing instant alerts upon detection. The proposed system processes live footage from a monitoring surveillance camera by utilizing a fine-tuned human segmentation model and image fusion technique as pre-processing and classifying a set of live footage with a 3D multi-stream CNN model (4S-3DCNN). The system alerts when the sequence of the Falling of the monitored human, followed by having Fallen, takes place. The effectiveness of the system was assessed using the publicly available Le2i dataset. System validation revealed an impressive result, achieving an accuracy of 99.44%, sensitivity of 99.12%, specificity of 99.12%, and precision of 99.59%. Based on the reported results, the presented system can be a valuable tool for detecting human falls, preventing fall injury complications, and reducing healthcare and productivity loss costs.

WiASL: American Sign Language writing recognition system using commercial WiFi devices

Contactless human–computer interaction has been widely used in a variety of scenarios. WiFi-based approaches can effectively protect user privacy compared to vision-based and wearable sensor-based approaches. However, existing WiFi-based input systems require long movement trajectories, making input inefficient and fatiguing users quickly. This paper presents WiASL, a commercial WiFi device-based micro-motion input system. WiASL uses American Sign Language (ASL) to represent letters, which requires only finger movements for most letters. In WiASL, we merge the amplitudes and phases to detect the signal segments containing micro-motions, extract features using the Attention-based Weighted Linear Discriminant Analysis (AWLDA) algorithm and a spatiotemporal deep neural network, and utilize a fully connected neural network for classification. The experimental results demonstrate that WiASL achieves a 3.3% false rejection rate for detection and 95.10% accuracy for recognition. WiASL can significantly improve input efficiency and maintain a high recognition accuracy compared with existing systems.

Target-driven multi-input mapless robot navigation with deep reinforcement learning

Collision-free navigation without pre-built maps is a promising and challenging problem. To make full use of the information in the environment, we propose a multi-input mapless navigation framework that combines both vision-based and ranging-based approaches. It takes the position and orientation of the robot relative to the target as the reference, incorporating RGB-D images and distance measurements as state inputs to the deep reinforcement learning model. In the framework, Dual-VAE is designed to encode RGB images and depth images simultaneously, in order to effectively mine the intrinsic information of RGB-D images. Moreover, we engineer a special reward function that enables the robot to navigate with a smooth trajectory and leave redundant space with obstacles for other tasks. Target-driven multi-input mapless navigation framework has been experimentally proven to achieve satisfactory performance in simulation environments. Our source code and simulation environments are released at https://github.com/YosephYu/MultiInput-Mapless-Navigation.

Laser-Based Door Localization for Autonomous Mobile Service Robots.

For autonomous mobile service robots, closed doors that are in their way are restricting obstacles. In order to open doors with on-board manipulation skills, a robot needs to be able to localize the door's key features, such as the hinge and handle, as well as the current opening angle. While there are vision-based approaches for detecting doors and handles in images, we concentrate on analyzing 2D laser range scans. This requires less computational effort, and laser-scan sensors are available on most mobile robot platforms. Therefore, we developed three different machine learning approaches and a heuristic method based on line fitting able to extract the required position data. The algorithms are compared with respect to localization accuracy with help of a dataset containing laser range scans of doors. Our LaserDoors dataset is publicly available for academic use. Pros and cons of the individual methods are discussed; basically, the machine learning methods could outperform the heuristic method, but require special training data when applied in a real application.

Vision-Based Work Zone Safety Alert System in a Connected Vehicle Environment

Work zones, being a critical component of roadway transportation systems, can benefit greatly from computer vision-enabled roadway infrastructures, specifically in connected vehicle (CV) environments. Connected infrastructures, such as roadside units (RSU) and on-board units (OBU), can greatly improve the environmental awareness and safety of CVs driving through a work zone. The contribution of this paper lies in developing a vision-based approach to generate work zone safety messages in real time, utilizing video streams from roadside monocular traffic cameras that can be used by CV work zone safety apps on mobile devices to reliably navigate through a work zone. A monocular traffic camera calibration method is proposed to establish the accurate mapping between the image plane and Global Position System (GPS) space. Real test scenarios show that our algorithm can precisely and effectively locate work zone boundaries from a monocular traffic camera in real time. We demonstrate the capabilities and features of our system through real-world experiments where the driver cannot see the work zone. End-to-end latency analysis reveals that the vision-based work zone safety warning system satisfies the active safety latency requirements. This vision-based work zone safety alert system ensures the safety of both the worker and the driver in a CV environment.