Vision-based Sensor Research Articles

Improved Blob-Based Feature Detection and Refined Matching Algorithms for Seismic Structural Health Monitoring of Bridges Using a Vision-Based Sensor System

Improved Blob-Based Feature Detection and Refined Matching Algorithms for Seismic Structural Health Monitoring of Bridges Using a Vision-Based Sensor System

Autonomous control of an ultrasound probe for intra-operative ultrasonography using vision-based shape sensing of pneumatically attachable flexible rails

PurposeIn robotic-assisted minimally invasive surgery, surgeons often use intra-operative ultrasound to visualise endophytic structures and localise resection margins. This must be performed by a highly skilled surgeon. Automating this subtask may reduce the cognitive load for the surgeon and improve patient outcomes.MethodsWe demonstrate vision-based shape sensing of the pneumatically attachable flexible (PAF) rail by using colour-dependent image segmentation. The shape-sensing framework is evaluated on known curves ranging from r=30\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$r = 30$$\\end{document} to r=110\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$r = 110$$\\end{document} mm, replicating curvatures in a human kidney. The shape sensing is then used to inform path planning of a collaborative robot arm paired with an intra-operative ultrasound probe. We execute 15 autonomous ultrasound scans of a tumour-embedded kidney phantom and retrieve viable ultrasound images, as well as seven freehand ultrasound scans for comparison.ResultsThe vision-based sensor is shown to have comparable sensing accuracy with FBGS-based systems. We find the RMSE of the vision-based shape sensing of the PAF rail compared with ground truth to be 0.4975±0.4169\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$0.4975 \\pm 0.4169$$\\end{document} mm. The ultrasound images acquired by the robot and by the human were evaluated by two independent clinicians. The median score across all criteria for both readers was ‘3—good’ for human and ‘4—very good’ for robot.ConclusionWe have proposed a framework for autonomous intra-operative US scanning using vision-based shape sensing to inform path planning. Ultrasound images were evaluated by clinicians for sharpness of image, clarity of structures visible, and contrast of solid and fluid areas. Clinicians evaluated that robot-acquired images were superior to human-acquired images in all metrics. Future work will translate the framework to a da Vinci surgical robot.

Vision-based seam tracking for GMAW fillet welding based on keypoint detection deep learning model

Pre-programmed welding robots significantly improved the efficiency and quality of the welds in large batch production. In small and medium batch production, the robots need appropriate sensors to perform well and adapt to the changes and uncertainties in a noisy welding environment. Vision-based sensors enabled by machine learning are making it possible to sense in process previously not measurable. One challenge is developing artificial intelligent models capable of real-time seam tracking, particularly in fillet joints where visual analysis is hindered by non-perpendicular camera angles and arc reflections. In this paper, we propose a vision system that enables automated seam tracking with a collaborative robot. The vision-based deep learning classification model detects the tacks, where the seam is not visible. It is based on a keypoint detection deep learning model that addresses the challenges in distorted and noisy images of fillet joints between the pipes and flanges during the real-time Gas Metal Arc Welding to track the location of the seam in non tack images. The system is optimized for real time seam tracking by proposing the appropriate input image size. Multiple images and multiple points are also considered to provide a controllable signal of the location of the seam with less errors and outliers. Our proposed model can track the seam with more than 80 percent accuracy for errors less than 0.3 mm in fillet joints. The high accuracy of the proposed method would result in fewer flaws and defects and reduced rework, resulting in significant cost saving in manufacturing. The real-time monitoring also enables the adaptability to slight variations in gap.

Advancements in Learning-Based Navigation Systems for Robotic Applications in MRO Hangar: Review.

The field of learning-based navigation for mobile robots is experiencing a surge of interest from research and industry sectors. The application of this technology for visual aircraft inspection tasks within a maintenance, repair, and overhaul (MRO) hangar necessitates efficient perception and obstacle avoidance capabilities to ensure a reliable navigation experience. The present reliance on manual labour, static processes, and outdated technologies limits operation efficiency in the inherently dynamic and increasingly complex nature of the real-world hangar environment. The challenging environment limits the practical application of conventional methods and real-time adaptability to changes. In response to these challenges, recent years research efforts have witnessed advancement with machine learning integration aimed at enhancing navigational capability in both static and dynamic scenarios. However, most of these studies have not been specific to the MRO hangar environment, but related challenges have been addressed, and applicable solutions have been developed. This paper provides a comprehensive review of learning-based strategies with an emphasis on advancements in deep learning, object detection, and the integration of multiple approaches to create hybrid systems. The review delineates the application of learning-based methodologies to real-time navigational tasks, encompassing environment perception, obstacle detection, avoidance, and path planning through the use of vision-based sensors. The concluding section addresses the prevailing challenges and prospective development directions in this domain.

Louvain-Based Traffic Object Detection for Roadside 4D Millimeter-Wave Radar

Object detection is the fundamental task of vision-based sensors in environmental perception and sensing. To leverage the full potential of roadside 4D MMW radars, an innovative traffic detection method is proposed based on their distinctive data characteristics. First, velocity-based filtering and region of interest (ROI) extraction were employed to filter and associate point data by merging the point cloud frames to enhance the point relationship. Then, the Louvain algorithm was used to divide the graph into modularity by converting the point cloud data into graph structure and amplifying the differences with the Gaussian kernel function. Finally, a detection augmentation method is introduced to address the problems of over-clustering and under-clustering based on the object ID characteristics of 4D MMW radar data. The experimental results showed that the proposed method obtained the highest average precision and F1 score: 98.15% and 98.58%, respectively. In addition, the proposed method showcased the lowest over-clustering and under-clustering errors in various traffic scenarios compared with the other detection methods.

The first use of a photogrammetry drone to estimate population abundance and predict age structure of threatened Sumatran elephants

Wildlife monitoring in tropical rainforests poses additional challenges due to species often being elusive, cryptic, faintly colored, and preferring concealable, or difficult to access habitats. Unmanned aerial vehicles (UAVs) prove promising for wildlife surveys in different ecosystems in tropical forests and can be crucial in conserving inaccessible biodiverse areas and their associated species. Traditional surveys that involve infiltrating animal habitats could adversely affect the habits and behavior of elusive and cryptic species in response to human presence. Moreover, collecting data through traditional surveys to simultaneously estimate the abundance and demographic rates of communities of species is often prohibitively time-intensive and expensive. This study assesses the scope of drones to non-invasively access the Bukit Tigapuluh Landscape (BTL) in Riau-Jambi, Indonesia, and detect individual elephants of interest. A rotary-wing quadcopter with a vision-based sensor was tested to estimate the elephant population size and age structure. We developed hierarchical modeling and deep learning CNN to estimate elephant abundance and age structure. Drones successfully observed 96 distinct individuals at 8 locations out of 11 sampling areas. We obtained an estimate of the elephant population of 151 individuals (95% CI [124, 179]) within the study area and predicted more adult animals than subadults and juvenile individuals in the population. Our calculations may serve as a vital spark for innovation for future UAV survey designs in large areas with complex topographies while reducing operational effort.

A Feasibility Study on Extension of Measurement Distance in Vision Sensor Using Super-Resolution for Dynamic Response Measurement.

The current civil infrastructure conditions can be assessed through the measurement of displacement using conventional contact-type sensors. To address the disadvantages of traditional sensors, vision-based sensor measurement systems have been derived in numerous studies and proven as an alternative to traditional sensors. Despite the benefits of the vision sensor, it is well known that the accuracy of the vision-based displacement measurement is largely dependent on the camera extrinsic or intrinsic parameters. In this study, the feasibility study of a deep learning-based single image super-resolution (SISR) technique in a vision-based sensor system is conducted to alleviate the low spatial resolution of image frames at long measurement distance ranges. Additionally, its robustness is evaluated using shaking table tests. As a result, it is confirmed that the SISR can reconstruct definite images of natural targets resulting in an extension of the measurement distance range. Additionally, it is determined that the SISR mitigates displacement measurement error in the vision sensor-based measurement system. Based on this fundamental study of SISR in the feature point-based measurement system, further analysis such as modal analysis, damage detection, and so forth should be continued in order to explore the functionality of SR images by applying low-resolution displacement measurement footage.

A Robust Incipient Slip Detection Method With Vision-Based Tactile Sensor Based on Local Deformation Degree

Incipient slip detection is important for manipulators to achieve stable grasping. In order to make manipulators adapted to unstructured environments better and realize a wider range of applications, it is necessary to achieve incipient slip detection with robustness to object surface properties. In this paper, we propose a novel incipient slip detection method with vision-based tactile sensor. The method only uses sensor deformation data and does not need any information about the grasped object. In addition, it has the advantages of intuitive principles and a wide range of applicable working conditions. In this paper, the local deformation degree of the sensor is proved to be related to the stick/slip state at the corresponding location by kinematic aspects. Based on this, a method is proposed to select reference points in the stick region with local deformation degree. These reference points are used to fit the rigid motion of the contacted object and the slip field. Furthermore, an iterative algorithm is proposed to delineate the stick/slip region based on the characteristic of the slip field. Afterward, complex contact conditions are constructed through finite element simulation. Curved surfaces, discontinuous surfaces, and surfaces with multiple friction coefficients distribution are used to verify the correctness and superiority of the proposed method compared to other methods. Finally, the slip detection experimental bench is built, and our self-developed vision-based sensor is used to contact common objects with complex surface properties. Through the real slip detection experiments, the adaptability and good application prospects of this method are demonstrated.

Development of Obstacle Detection System Based On the Integration of Different Based Sensor for Small-Sized UAV

Due to the physical size and weight limits of small unmanned aerial vehicles (UAVs), developing a reliable obstacle detection a system that can provide an effective and safe avoidance path is extremely difficult. Prior work has tended to use a vision-based sensor as the primary detecting sensor however, this has resulted in a high reliance on texture appearance and a lack of distance sensing capabilities. Furthermore, due to the inability to detect the free region, vision-based sensor detection systems have difficulty developing a trusted safe avoidance path. However, most wide spectrum range sensors are bulky and expensive, making them unsuitable for small UAVs. This projectaims to construct an obstacles detection system with the integration of various based sensors for a small UAV. The potential obstacles are identified by categorizing feature points identified in image frame. The suggested approach was tested in a real-world setting for both of the observed scenarios, which included various obstacles configurations. Two types of scenarios are experimented in this project consists of single frontal obstacles and presence of side obstacles alongside the frontal obstacles. On top of that, the position of the side obstacle is aligned to the frontal obstacle and then will be positioned in the increment of 20cm further from the frontal obstacle in order to analyze the outcome of the proposed algorithm. The proposed detection system had a possibility to be a trustworthy system even after utilizing the depth perception technique, however this does not imply that the proposed system is faultless. The results show that the suggested algorithm system detects and distinguishes between the potential obstacles and free region for a single frontal obstacle perfectly. However, there were improvements that should be implemented with the proposed system's ability of detection for multiple obstacles.

A Multimodal IoT-Based Locomotion Classification System Using Features Engineering and Recursive Neural Network

Locomotion prediction for human welfare has gained tremendous interest in the past few years. Multimodal locomotion prediction is composed of small activities of daily living and an efficient approach to providing support for healthcare, but the complexities of motion signals along with video processing make it challenging for researchers in terms of achieving a good accuracy rate. The multimodal internet of things (IoT)-based locomotion classification has helped in solving these challenges. In this paper, we proposed a novel multimodal IoT-based locomotion classification technique using three benchmarked datasets. These datasets contain at least three types of data, such as data from physical motion, ambient, and vision-based sensors. The raw data has been filtered through different techniques for each sensor type. Then, the ambient and physical motion-based sensor data have been windowed, and a skeleton model has been retrieved from the vision-based data. Further, the features have been extracted and optimized using state-of-the-art methodologies. Lastly, experiments performed verified that the proposed locomotion classification system is superior when compared to other conventional approaches, particularly when considering multimodal data. The novel multimodal IoT-based locomotion classification system has achieved an accuracy rate of 87.67% and 86.71% over the HWU-USP and Opportunity++ datasets, respectively. The mean accuracy rate of 87.0% is higher than the traditional methods proposed in the literature.

A review of vision-based indoor HAR: state-of-the-art, challenges, and future prospects.

With the advent of technology, we are getting more comfortable with the use of gadgets, cameras, etc., and find Artificial Intelligence as an integral part of most of the tasks we perform throughout the day. In such a scenario, the use of cameras and vision-based sensors comes as an escape from many real-time problems and challenges. One major application of these vision-based systems is Indoor Human Activity Recognition (HAR) which serves in a variety of scenarios ranging from smart homes, elderly care, assisted living, and human behavior pattern analysis for identifying any abnormal behavior to abnormal activity recognition like falling, slipping, domestic violence, etc. The effect of HAR in real time has made the area of indoor activity recognition a more explored zone by the industrial segment to attract users with their products in multiple domains. Hence, considering these aspects of HAR, this work proposes a detailed survey on indoor HAR. Through this work, we have highlighted the recent methodologies and their performance in the field of indoor activity recognition. We have also discussed- the challenges, detailed study of approaches with real-world applications of indoor-HAR, datasets available for indoor activity, and their technical details in this work. We have proposed a taxonomy for indoor HAR and highlighted the state-of-the-art and future prospects by mentioning the research gaps and the shortcomings of recent surveys with respect to our work.

Sparse accelerometer-aided computer vision technology for the accurate full-field displacement estimation of beam-type bridge structures

Computer vision-based sensors are considered effective means to monitor full-field displacements; nevertheless, several limitations prevent their applications in full-scale structures. First, the accuracy of computer vision-based sensors is affected by measured distances. Second, due to the limited frame rate of cameras, computer vision-based sensors cannot record high-frequency responses. To effectively address these issues, a novel full-field displacement estimation approach based on the data fusion of camera-based measurements and limited accelerations was proposed. Full-field pseudo-static displacements were extracted from camera-based measurements, and full-field dynamic displacements were expanded by accelerations at sparse locations based on the modal superposition method. Subsequently, high-fidelity full-field displacements were obtained by data fusion. The proposed approach was numerically and experimentally verified. Further, the full-field displacements of an experimental-scale beam model and a full-scale bridge were estimated by a consumer-grade camera and several accelerometers.

Observability Analysis and Keyframe-Based Filtering for Visual Inertial Odometry With Full Self-Calibration

Camera–inertial measurement unit (IMU) sensor fusion has been extensively studied in recent decades. Numerous observability analysis and fusion schemes for motion estimation with self-calibration have been presented. However, it has been uncertain whether the intrinsic parameters of both the camera and the IMU are observable under general motion. To answer this question, by using the Lie derivatives, we first prove that for a rolling shutter (RS) camera–IMU system, all the intrinsic and extrinsic parameters, camera time offset, and readout time of the RS camera are observable with an unknown landmark. To our knowledge, we are the first to present such a proof. Next, to validate this analysis and to solve the drift issue of a structureless filter during standstills, we develop a keyframe-based sliding window filter (KSWF) for odometry and self-calibration, which works with a monocular RS camera or stereo RS cameras. Though the keyframe concept is widely used in vision-based sensor fusion, to our knowledge, the KSWF is the first of its kind to support self-calibration. Our simulation and real data tests have validated that it is possible to fully calibrate the camera–IMU system using the observations of opportunistic landmarks under diverse motion. Real data tests confirmed previous allusions that keeping landmarks in the state vector can remedy the drift in standstill and showed that the keyframe-based scheme is an alternative solution.

Employing vision-based sensing for long-term monitoring

Despite the crucial role of structural health monitoring (SHM), traditional contact-based sensors are costly and lacking automation. Vision-based sensors have recently emerged as a potential substitute, due to their non-contact nature and low cost. However, investigations lack validating their long-term performance. This study assesses vision-based sensing for robust and efficient long-term displacement monitoring. Additionally, the suitability of various machine learning (ML) algorithms in automatically extracting information from the generated data are examined. A first experiment examines the capability to monitor ambiently excited structures over long periods. The parameters assessed are: a) noise and drift (e.g., related to environmental changes such as temperature, humidity, and lighting); and b) obtainable displacement resolution related to ambient excitation. Then, a second experiment examines the capability to monitor dynamically excited structures over long periods (via the employment of an artificial excitation, e.g., an eccentrically loaded mass motor). Here, the following parameters assessed are the: a) frequency bandwidth range; b) frequency resolution; c) displacement resolution related to dynamic excitation; and d) maximum number monitored points for given computational resources. Finally, for processing the generated data, the following ML algorithms are examined: a) Artificial Neural Networks (ANNs); b) support vector machines (SVMs); c) decision trees; and d) Convolutional Neural Networks (CNNs). Concerning long-term monitoring, the paper found that non-contact sensing had comparable accuracy with traditional contact-based sensors. About the ML models, the CNNs were found to be advantageous. Whilst this study was carried out on a small-scale structure, the potential of vision-based sensing for long-term monitoring of full-scale structures was highlighted.

OPERAnet, a multimodal activity recognition dataset acquired from radio frequency and vision-based sensors

This paper presents a comprehensive dataset intended to evaluate passive Human Activity Recognition (HAR) and localization techniques with measurements obtained from synchronized Radio-Frequency (RF) devices and vision-based sensors. The dataset consists of RF data including Channel State Information (CSI) extracted from a WiFi Network Interface Card (NIC), Passive WiFi Radar (PWR) built upon a Software Defined Radio (SDR) platform, and Ultra-Wideband (UWB) signals acquired via commercial off-the-shelf hardware. It also consists of vision/Infra-red based data acquired from Kinect sensors. Approximately 8 hours of annotated measurements are provided, which are collected across two rooms from 6 participants performing 6 daily activities. This dataset can be exploited to advance WiFi and vision-based HAR, for example, using pattern recognition, skeletal representation, deep learning algorithms or other novel approaches to accurately recognize human activities. Furthermore, it can potentially be used to passively track a human in an indoor environment. Such datasets are key tools required for the development of new algorithms and methods in the context of smart homes, elderly care, and surveillance applications.

Development of Vision Based Smart Gripper for Material Handling Using Internet of Things

Robotic grippers have becoming an emerging trend due to their boundless applications in industrial automation. Nowadays, the deployment of vision based smart gripper for material handling in industrial applications remains challenging and ongoing research. As the Internet of Things (IoT) becomes more commercialized, the various concept of IoT have been integrated with the gripper due to efficient usage. Therefore, this project proposes the development of vision-based sensor of smart gripper for material handling in industrial applications that integrates with the IoT. The rationale of integrating IoT to vision based smart gripper is that it allows authenticated users to log in from any device, anywhere, and view video or images from vision based smart gripper in real-time for critical material handling. This system incorporates a vision sensor camera that acts as an “eye” to automatically detect and recognize the object with different weights and shapes and send the information to the robot for the next task. This smart gripper adopts a force sensor mounted into the fingertip to control the force applied when working with a wide range of objects with different weights. As for the electronic system, power module, communication and control module, sensor and actuator as well as user interface module have been adopted and integrated into the system. In the software development system, user interface configuration was developed through mobile application in which it communicates with Raspberry Pi B+ camera to serve as IoT platform. A series of experiments shows that the vision based gripper using IoT able to detect and recognize the objects and then send the information/command directly to the robot to execute grasping and lifting phase of the object to the desired location that has been assigned.

Human Movement Quality Assessment Using Sensor Technologies in Recreational and Professional Sports: A Scoping Review.

The use of sensor technology in sports facilitates the data-driven evaluation of human movement not only in terms of quantity but also in terms of quality. This scoping review presents an overview of sensor technologies and human movement quality assessments in ecologically-similar environments. We searched four online databases to identify 16 eligible articles with either recreational and/or professional athletes. A total of 50% of the studies used inertial sensor technology, 31% vision-based sensor technology. Most of the studies (69%) assessed human movement quality using either the comparison to an expert’s performance, to an exercise definition or to the athletes’ individual baseline performance. A total of 31% of the studies used expert-based labeling of the movements to label data. None of the included studies used a control group-based study design to investigate impact on training progress, injury prevention or behavior change. Although studies have used sensor technology for movement quality assessment, the transfer from the lab to the field in recreational and professional sports is still emerging. Hence, research would benefit from impact studies of technology-assisted training interventions including control groups as well as investigating features of human movement quality in addition to kinematic parameters.

Graph-Based Thermal–Inertial SLAM With Probabilistic Neural Networks

Simultaneous localization and mapping (SLAM) system typically employs vision-based sensors to observe the surrounding environment. However, the performance of such systems highly depends on the ambient illumination conditions. In scenarios with adverse visibility or in the presence of airborne particulates (e.g., smoke, dust, etc.), alternative modalities such as those based on thermal imaging and inertial sensors are more promising. In this article, we propose the first complete thermal–inertial SLAM system that combines neural abstraction in the SLAM front end with robust pose-graph optimization in the SLAM back end. We model the sensor abstraction in the front end by employing probabilistic deep learning parameterized by mixture density networks (MDNs). Our key strategies to successfully model this encoding from thermal imagery are the usage of normalized 14-b radiometric data, the incorporation of hallucinated visual (RGB) features, and the inclusion of feature selection to estimate the MDN parameters. To enable a full SLAM system, we also design an efficient global image descriptor that is able to detect loop closures from thermal embedding vectors. We performed extensive experiments and analysis using three datasets, namely self-collected ground robot and hand-held data taken in indoor environment, and one public dataset (SubT-tunnel) collected in underground tunnel. Finally, we demonstrate that an accurate thermal–inertial SLAM system can be realized in conditions of both benign and adverse visibility.

Accurate structural displacement monitoring by data fusion of a consumer-grade camera and accelerometers

The displacement responses of bridge structures are critical both for structural health monitoring and structural safety evaluation. Using vision-based sensors to measure structural displacement responses is an effective method and has attracted considerable attention owing to its advantages of achieving non-contact, time-saving, and full-field measurements. However, there are several obstacles, including the limited frame rate, insufficient accuracy in large-scale measurements, and camera instability, which prevent the application of vision-based sensors. To overcome the shortcomings and improve the accuracies of the vision-based sensors, a displacement monitoring system combined with a consumer-grade camera and accelerometers was developed. Accelerometers attached to the target structures were used to reconstruct the dynamic displacement for data fusion with the vision-based displacement, while an accelerometer attached to the camera was used for camera vibration cancellation. Dynamic loading tests on a self–anchored suspension model bridge were conducted and results of the proposed system, the linear variable differential transformers, and a conventional vision-based system were compared. Field tests on a steel–concrete composite continuous beam bridge and a cable-stayed railway bridge were conducted, and the potential of the proposed system for use in real structures was validated.

Automated Motion Heatmap Generation for Bridge Navigation Watch Monitoring System

Abstract Most ship collisions and grounding accidents are due to errors made by watchkeeping personnel (WP) on the bridge. International Maritime Organization (IMO) adopts the resolution on the Bridge Navigation Watch Alarm System (BNWAS) detecting operator disability to avert these accidents. The defined system in the resolution is very basic and vulnerable to abuse. There is a need for a more advanced system of monitoring the behaviour of WP to mitigate watchkeeping errors. In this research, a Bridge Navigation Watch Monitoring System (BNWMS) is suggested to achieve this task. Architecture is proposed to train a model for BNWMS. The literature reveals that vision-based sensors can produce relevant input data required for model training. 2D body poses belonging to the same person are estimated from multiple camera views by using a deep learning-based pose estimation algorithm. Estimated 2D poses are projected into 3D space with a maximum 8 mm error by utilising multiple view computer vision techniques. Finally, the obtained 3D poses are plotted on a bird’s-eye view bridge plan to calculate a heatmap of body motions capturing temporal, as well as spatial, information. The results show that motion heatmaps present significant information about the behaviour of WP within a defined time interval. This automated motion heatmap generation is a novel approach that provides input data for the suggested BNWMS.