Vision-based Method Research Articles

Robust detection of headland boundary in paddy fields from continuous RGB-D images using hybrid deep neural networks

Accurate and robust headland boundary detection in the field is crucial for formulating turning strategies for agro-machinery. The headland area is challenging to be detected because of its similar appearance to the farmland, coupled with complex field conditions such as high weed pressure and discontinuous headland edges. In this study, a machine vision-based method for headland boundary detection was proposed, which introduced depth information in addition to RGB images to improve its detection accuracy. A deep learning-based network combining convolutional neural network (CNN) and recurrent neural network (RNN) was constructed for headland semantic segmentation. An interactive attention module (IAM) was proposed to fuse complementary information in RGB-D images adaptively. A time series information processing module (TPM) composed of a set of bidirectional convolution long short-term memories (ConvLSTMs) was used to extract interrelated information from consecutive images. Image preprocessing techniques and a proposed distance-based boundary point clustering algorithm were applied to the headland segmentation mask to obtain the boundary line on the working side of the agro-machinery. Ablation study results confirmed the effectiveness of IAM and TPM in improving the headland segmentation performance. The network presented achieved excellent headland segmentation performance, with a mean intersection over union (mIoU) of up to 95.7%. The mean deviation of boundary line extraction on 256*144 resolution images was 3.57 pixels. The detection rate reached 25.8 frames per second (FPS), which could provide real-time reference lines for agro-machinery turning path planning.

A lightweight model for blade tip image enhancement in helicopter rotor motion parameter measurement system

The helicopter rotor torsion angle is an important motion parameter of a helicopter rotor, which is helpful for the design of helicopter rotor system. The stereoscopic vision-based method measures the torsion angle by computing the directions of the blade tips in the blade tip images. However, the captured blade tip images suffer from blurring and underexposure, which affect the measurement accuracy. To this end, a lightweight deep model is proposed to improve the image quality. In addition, a blade tip image dataset is proposed to train the network. Corresponding experiments demonstrate the effectiveness of the proposed model. Compared to competitive methods, our model stands out with fewer parameters and faster inference time.

An approach for damage detection in noisy environments using DOG multi-scale space and fractal dimension

ABSTRACT The most prominent features of fractal dimension analysis are high sensitivity to damage and instant determination of damage location. However, an intrinsic deficiency is that it is difficult to identify damage in a structure while suppressing noise simultaneously. In addition, the vibration measurement with low-spatial-resolution is also the main factor restricting the accurate damage location. In this paper, a novel damage detection method based on fractal dimension is proposed, in order to provide the high-spatial-resolution vibration information, the phase-based optical flow is introduced, which is a vision-based measurement method that can calculate the structural vibration with high-spatial-resolution. Then, to detect damage in a structure while suppressing noise, an advanced signal analysis method is proposed, with this method, the fractal dimension (FRAC) is mapped into Difference of Gaussian (DOG) multi-scale space, producing the FRAC-DOG to reveal damage-induced singularities in mode shapes. Finally, a data fusion technique is applied to the proposed method to provide a reasonable damage detection result that maximises the elimination of uncertainty. The numerical and experimental results demonstrate that the proposed method can detect damage with high-precision in noisy environments by comparing it with the existing damage detection methods.

A Deep-Learning Approach for Identifying a Drunk Person Using Gait Recognition

Various accidents caused by alcohol consumption have recently increased in prevalence and have become a huge social problem. There have been efforts to identify drunk individuals using mobile devices; however, it is difficult to apply this method to a large number of people. A promising approach that does not involve wearing any sensors or subject cooperation is a markerless, vision-based method that only requires a camera to classify a drunk gait. Herein, we first propose a markerless, vision-based method to determine whether a human is drunk or not based on his or her gait pattern. We employed a convolutional neural network to analyze gait patterns with image augmentation depending on gait energy images. Gait images captured through a camera allow a complex neural network to detect the human body shape accurately. It is necessary for removing the background behind human shape in the gait image because it disrupts the detection algorithm. A process of conversion into gait energy images and augmenting image data is then applied to the dataset of the gait images. A total of 20 participants participated in the experiment. They were required to walk along a line both with and without wearing the Drunk Busters Goggles, which were intended to collect sober and drunk gait images. Validation accuracy for the recognition of a drunk state in 20 persons was approximately 74.94% under optimal conditions. If the present approach fulfills its promise, we can prevent safety accidents due to alcohol, thus decreasing its burden on industries and society.

A vision-based calibration method for aero-engine blade-robotic grinding system

A vision-based calibration method for aero-engine blade-robotic grinding system

Target-free vision-based approach for modal identification of a simply-supported bridge

Accurate and efficient vibration response measurement is one of the key tasks in structural health monitoring (SHM) of civil engineering structures. Normally, dynamic responses such as displacement responses, are acquired by physical sensors and data acquisition systems installed on the structures. However, the installation of sensors may be time-consuming and costly. In recent decades, computer vision-based methods have been introduced into civil engineering to provide alternatives for traditional displacement measurements. However, the camera specifications, such as limitations on field of view (FOV) and relatively low resolution of economic lens, have prevented the vision-based method from being widely used for vibration displacement measurement on relatively large-scale structures. This paper proposes a target-free vision-based approach for dynamic displacement measurement and modal identification of simply-supported bridges using a single camera. In this study, a series of videos are captured in separate experimental tests using a consumer-class GoPro on a long-span beam subjected to a moving load. The radial distortion of videos caused by the wide-angle lens is offset by distortion adaption, and the scale factor is fitted using a static video shooting experiment. The tracking process consists of three basic steps, namely, region of interest (ROI) selection, natural feature extraction and features tracking. The results are compared with those measured by linear variable differential transducer (LVDT), demonstrating a high accuracy by the proposed approach. Vibration characteristics, including natural frequencies and mode shapes of the bridge, are identified from the measured displacement. The overall mode shapes are obtained by combining the mode shapes identified from different segmental videos of the bridge obtained by roving the single camera along the bridge for measurements. Comparisons of the natural frequencies and mode shapes between the results from the proposed approach and LVDTs demonstrate the effectiveness and accuracy of the proposed approach for displacement measurement and modal identification of long bridges under moving loads.

Automatic Inspection of Bridge Bolts Using Unmanned Aerial Vision and Adaptive Scale Unification-Based Deep Learning

Bolted connections are essential components that require regular inspection to ensure bridge safety. Existing methods mainly rely on traditional artificial vision-based inspection, which is inefficient due to the many bolts of bridges. A vision-based method using deep learning and unmanned aerial vision is proposed to automatically analyze the bridge bolts’ condition. The contributions are as follows: (1) Addressing the problems that motion blur often exists in videos captured by unmanned ariel systems (UASs) with high moving speed, and that bolt damage is hard to accurately detect due to the few pixels a single bolt occupies, a bolt image preprocessing method, including image deblurring based on inverse filtering with camera motion model and adaptive scaling based on super-resolution, is proposed to eliminate the motion blur of bolt images and segment them into subimages with uniform bolt size. (2) Addressing the problem that directly applying an object detection network for both bolt detection and classification may lead to the wrong identification of bolt damage, a two-stage detection method is proposed to divide bolt inspection into bolt object segmentation and damage classification. The proposed method was verified on an in-service bridge to detect bolts and classify them into normal bolts, corrosion bolts, and loose bolts. The results show that the proposed method can effectively eliminate the inherent defects of data acquired by UAS and accurately classify the bolt defects, verifying the practicability and high precision of the proposed method.

Vision-based thermal drift monitoring method for machine tools

A method is presented to measure machine tool thermal drift for error compensation. A wireless microscope within a tool holder in the spindle is used to capture videos of image targets attached to the worktable. For each target, one video is captured during spindle rotation orthogonal to the worktable and another video is captured during axis translation orthogonal to the worktable. Data are collected periodically so that the three-dimensional thermal error at each target location is determined via image analysis. Experiments verify that the method measures micrometer-level tool-to-workpiece thermal drift for error model development and thermal drift compensation.

Bolts looseness detection in construction machinery

Bolt connections have increasingly wide applications in the mechanical industry, such as civil construction, construction machinery, etc. However, along with the vibration of the mechanical operation, bolt looseness is unavoidable and thus can lead to severe consequences. Therefore, the detection of loose bolts deserves an in-depth study. However, conventional bolt looseness detection techniques require a lot of time and labor, and direct contact sensor-based detection techniques are sensitive to interference from the surroundings. To tackle these challenges mentioned above, this work proposes a vision-based bolt looseness detection method, with the advantages of low cost, easy deployment, and non-contact. Specifically, an innovative attention mechanism is introduced into the Yolov4 framework to improve the accuracy and speed of bolt detection. Each bolt image is cropped by the recognition result of the detector, then the bolt contour is extracted by edge detection, and the bolt is determined to be loose or not based on the angle of bolt rotation. Comprehensive experiments in real-world settings verify the effectiveness and efficiency of the proposed method.

An Intelligent Vision-Based Nutritional Assessment Method for Handheld Food Items

Dietary assessment has proven to be an important tool to evaluate the dietary intake of patients with diabetes and obesity. The traditional approach of accessing the dietary intake is to conduct a 24-hour dietary recall (24HR), a structured interview designed to obtain information on food categories and volume consumed by the participants. Due to the unconscious biases in this kind of self-reporting approach, many research studies have explored the use of vision-based approaches to provide accurate and objective assessments. Despite the promising results of food recognition by deep neural networks, there still exist several hurdles in deep learning-based food volume estimation ranging from domain shift between synthetic and raw 3D models, shape completion ambiguity and, most importantly, lack of large-scale paired training dataset. Therefore, this paper proposed an intelligent nutritional assessment approach via weakly-supervised point cloud completion. It aims to close the reality gap in 3D point cloud completion tasks and address the targeted challenges by proposing the Point Cloud Auxiliary Classifier GAN (PC-ACGAN), a novel and robust point cloud completion architecture that can be trained in a weakly-supervised manner. Then the volume can be easily estimated from the completed representation of the food. In addition to advantages with regard to efficiency, another major merit of our system is that it can be used to estimate the volume of handheld food items without requiring the constraints including placing the food items on a table or next to fiducial markers, which facilitate the implementation on both wearable and handheld cameras. Comprehensive experiments have been carried out on major benchmark datasets and self-constructed volume-annotated dataset respectively, in which the proposed PC-ACGAN demonstrates comparable results with several strong fully-supervised baseline methods and shows superior completion ability in handling food volume estimation.

Vision-Based Autonomous Landing of a Hybrid Robot on a Powerline

In recent years, various types of inspection robots have been developed to automate powerline inspection. The hybrid robot combines the advantages of climbing and flying robots and has a promising prospect in powerline inspection. But landing a hybrid robot on the target powerline among multiple ones is challenging. Flights require robust detection of powerlines and stable tracking of the target powerline. We propose a complete solution for the autonomous landing of a hybrid robot on a powerline. First, a special feature extraction operator and the corresponding density-based feature recognition algorithm are designed to detect multiscale powerlines. Second, a binocular vision-based depth estimation method for the landing point in the powerline is described. Third, two spatio-temporal dictionaries are established to track the target one in multiple powerlines. Meanwhile, landing strategies and control methods are presented to achieve a stable landing task. Finally, a hybrid robot is designed to validate the proposed method. The experiment results demonstrate the accuracy of the powerline detection and depth estimation algorithm, as well as the effectiveness of the robot in tracking and landing tasks.

Application and Practice of Artificial Intelligence Technology in Interior Design

Abstract In order to enhance the technical innovation of interior design, this paper proposes an interior design approach based on artificial intelligence technology to optimize the overall scheme of interior design. The shape and contour features of different kinds of objects in the interior are modeled by visual information design, the information components of the visual images are determined using correlation scale analysis, and the maximum grayscale values are marked to implement 3D visual feature reconstruction. The reconstruction process is based on accurately aligned point cloud data, and parametric surface reconstruction is implemented for the target objects to obtain a more accurate 3D interior design model and make the interior scene simulation more realistic. In order to verify the effectiveness of the application of the interior design method based on artificial intelligence technology, simulation design experiments are conducted. The results show that the multi-element fusion (TBG) consideration characteristics of the proposed method are more uniformly distributed. The visual saturation and stereo image design levels reach 98% and 88.51%, respectively. The average error result is 0.06, which are both better than the 3D vision-based interior design method and the multi-element fusion-based interior design method, and have higher design reliability. It can be seen that the interior design method based on artificial intelligence technology helps to enhance the effect of intelligent application of interior design and promotes the optimization and upgrading of the interior design working experience.

Vision-Based Method to Measure the Synchronization Level of Jumping Crowds

The prediction of the dynamic loads produced by groups of people is a crucial aspect for the design of stadiums or entertaining venues. This is because the coordinated motion of lively crowds may induce severe vibration levels in the structure, which can become critical for both human comfort and structural safety. However, the available information on this topic are very limited. Human loads often rely on deterministic models which do not consider the interaction and the coordination achieved by the participants or try to account for them through empirical assumptions. Therefore, they could find very little correspondence in realistic scenarios. This paper aims to close this gap by introducing a vision-based technique able to directly measure the crowd loading and quantify the synchronization level between the individuals. Starting from a sequence of images of a jumping crowd, Digital Image Correlation (DIC) is used extract the vertical velocity of different regions occupied by the participants; then, the vertical force time record is estimated. Finally, the comparison between the actual force signals and their envelopes allows to estimate the crowd synchronization over time. The method has been successfully validated with two field tests on the grandstands of the Giuseppe Meazza stadium in Milan, demonstrating its ability to reliably estimate the synchronization level reached by the participants.

Surface detection method of glass fibre composites based on computer vision

Considering the high cost, low efficiency and poor real-time performance of manual inspection methods in detecting surface defects such as glass fibre imprints, resin build-up and wrinkles, in this paper, a machine vision-based method is proposed to detect surface defects of glass fibre composites. The method designs an automatic inspection platform using two high-resolution line scan cameras for image acquisition. The eight directional templates of the Kirsch operator are used to convolve the derivatives of the image pixel points respectively, and the largest template is selected to determine its edge direction, and the detection of surface defects is achieved by combining with the Canny operator. The experimental results show that the proposed algorithm can well suppress noise interference, improve the accuracy of edge localisation and detection, and well retain edge information while avoiding pseudo-edges.

A fusion estimation of the peak tire–road friction coefficient based on road images and dynamic information

To accurately acquire the peak tire–road friction coefficient, a fusion estimation framework combining vision and vehicle dynamic information is established. First, information for the road ahead is collected in advance from an image captured by a camera, and the road type with its typical range of tire–road friction coefficients is identified with a lightweight convolutional neural network. Then, an unscented Kalman filter (UKF) method is established to estimate the tire–road friction coefficient value directly according to the dynamic vehicle states. Next, the results from the road-type recognition and dynamic estimation methods are spatiotemporally synchronized. Finally, a confidence-based vision and vehicle dynamic fusion strategy is proposed to obtain an accurate peak tire–road friction coefficient. The virtual and real vehicle test results suggest that the proposed fusion estimation strategy can accurately determine the peak tire–road friction coefficient. The proposed strategy can more precisely acquire the tire–road friction coefficient than can the general vision-based estimation method and is superior to the dynamic-based estimation method in that it eliminates the need for sufficient tire excitation to some extent.

Research on Parking Space Status Recognition Method Based on Computer Vision

To improve the utilization rate of parking space resources and reduce the cost of installing and maintaining sensor recognition, this paper proposed an improved computer vision-based parking space status recognition method. The overall recognition accuracy was improved by graying the video, filtering smoothing noise reduction, image enhancement pre-processing, introducing texture feature extraction method based on LBP operator, improving the background difference method, and then, we used a perceptual hash algorithm to calculate the Hamming distance between the background image and the hash string of the current frame of the video, excluding the influence of light and pedestrian on recognition accuracy. Finally, a parking space status recognition system is developed relied on the Python environment, and parking spaces are recognized in three environmental states: with direct light, without direct light, and in rain and snow. The overall average accuracy of the experimental results was 97.2%, which verifies the accuracy of the model.

Machine vision-based transverse vibration measurement of diamond wire

Diamond wire sawing is one of the key technologies for wafering in the semiconductor industry. The transverse vibration of the wire during sawing will generate additional displacement that not only reduces the as-cut wafer quality but increases the critical adhesion gap between the wires, which is an important factor limiting the development of diamond wire slicing technology toward narrower kerf loss and thinner as-cut wafer. The vibration measurement of the diamond wire provides an indispensable test basis for its vibration control. Therefore, a machine vision-based transverse vibration measurement method of diamond wire is developed in this paper. The approximate edge position of the wire in the image midline is obtained to crop the region of interest in the image by using the maximum continuous same-directional pixel gradient algorithm and thus reduce the computing time. The wire edges in the image are accurately extracted by an adaptive image segmentation method proposed based on the image histogram. According to the surface topography characteristics of diamond wire, a feature extraction algorithm based on the multiple inward line fitting using the least square method is proposed, which can effectively detect the position of the inner diameter boundary of the diamond wire. Accordingly, the equation of the central axis of the wire in the image is obtained to realize the transverse vibration detection of the diamond wire. To conclude, a diamond wire vibration measurement experiment is conducted and the machine vision detection results are compared with the vibration displacement measured by the accelerometer at the same position of the wire end. The results show that the measurement error of the machine vision vibration inspection is generally small for the different types of diamond wires and the different frequencies of the wire transverse vibration, which means that the measurement method proposed in this paper has good accuracy, and applicability, and robustness.

Real-time automated detection of older adults' hand gestures in home and clinical settings.

The online version contains supplementary material available at 10.1007/s00521-022-08090-8.

Stereo vision-based Kinematic calibration method for the Stewart platforms.

Accuracy is the most important index for the industrial applications of the Stewart platform, which can be guaranteed by the kinematic calibration method to improve the motion orbit performance of this platform. In order to improve the effectiveness of the least squares algorithm and the identified accuracy of the platform's geometric parameter errors, an applicab-le dimensionless error model based on the structural characteristics of the Stewart platform is investigated. Moreover, a novel stereo vision-based measurement method is proposed, which can measure the 6-degree-of-freedom (DOF) pose of the moving platform. On this basis, an identification simulation is schemed to validate the efficiency of the dimensionless error model, and the kinematic calibration experiment is carried out on a prototype. The experimental results demonstrate that the position error is decreased to 0.261 mm with an improved accuracy of 89.720%, the orientation error is decreased to 0.051° with an improved accuracy of 90.351%.

Vision-based inspection of prefabricated components using camera poses: Addressing inherent limitations of image-based 3D reconstruction

Modular construction can lead to additional cost overruns and delays when a defect is found on the construction site and is not easily repairable. Researchers have developed various methods that use image-based 3D reconstruction for quality assessment, but they have inherent limitations, such as inconsistency and dealing with surfaces with reflectivity and limited visual features. Therefore, this paper presents a vision-based quality assessment method using cameras for prefabricated components by addressing these limitations. Specifically, this paper proposes a novel quality inspection method with sub-millimeter accuracy using cameras focused on leveraging camera poses (as opposed to 3D point clouds that are often not consistent in quality) from the image-based 3D reconstruction. The 3D point estimation by computing triangulation was used for achieving accurate measurement. The proposed method is validated using six different variances and two case studies – an aluminum pipe with a reflective surface and a fabricated concrete column. The results demonstrate the accuracy and effectiveness of the proposed method.