Contact-type Sensor Research Articles

Smart structural health monitoring using computer vision and edge computing

Structural health monitoring (SHM) provides real-time data on the condition and performance of infrastructure, enabling timely and cost-effective maintenance interventions, and hence enhanced safety and extended service life. The computer vision-based non-contact sensor has emerged as a promising alternative to conventional contact-type sensors for structural displacement measurement and SHM. Many of the currently reported vision-based structural displacement measurement systems typically temporarily set up a video camera from a distance to the structure. The collected images or videos are usually stored locally and post-processed offline to obtain structural displacement responses, which is cumbersome and limited to short-term SHM applications. The recent development of technologies empowered by the Internet of Things (IoT) and edge computing has enabled real-time video processing and analysis at the source, minimizing latency, reducing bandwidth requirements, and enabling prompt decision-making, thereby enhancing efficiency and responsiveness compared to traditional offline video recording and processing systems. In this paper, an edge computing vision-based displacement measurement system (EdgeCVDMS) is developed. Video recording, processing, and displacement response identification are entirely performed on an edge device integrated with the vision-based displacement tracking algorithm, thereby greatly reducing the amount of data transmitted to the cloud server. The feasibility and applicability of the developed sensing system are experimentally validated on a laboratory-scaled transmission tower structure. The proposed EdgeCVDMS is cost-effective, easily deployable, and of great potential to be applied for the condition assessment of a larger population of aging civil infrastructure.

Measuring stress of strand using magnetic barkhausen noise measured by solenoid-type sensor

ABSTRACT This paper presents a new approach on non-destructive stress estimation that involves measuring the magnetic Barkhausen noise (MBN) on prestressed strands using a solenoid-type sensor. Using a contact-type sensor on infrastructure is difficult because prestressed strands are wrapped with protective materials and cannot be directly touched. To test the feasibility of the new idea, a solenoid-type MBN sensor is developed that measures the stress of a seven-wire strand. The sensor consists of a magnetic-field inducing coil and a sensing coil, designed to generate a varying magnetic field and measure the change in magnetic flux, respectively. An infinite impulse response filter is recommended to obtain the MBN from the measured signal, and the root mean square (RMS) is introduced to determine the level of the MBN. An indoor test is performed on a seven-wire strand with a diameter of 15.2 mm. The test shows that the MBN’s RMS decreases as the tensile stress increases. The relationship between the MBN’s RMS and the strand’s stress exhibits a clear trend similar to those observed in previous studies using contact-type sensors. The test results indicate that it is possible to estimate the stress of a strand inside a duct using the solenoid-type MBN sensor.

Temporal Stability of Management Zone Patterns: Case Study with Contact and Non-Contact Soil Electrical Conductivity Sensors in Dryland Pastures.

Precision agriculture (PA) intends to validate technological tools that capture soil and crop spatial variability, which constitute the basis for the establishment of differentiated management zones (MZs). Soil apparent electrical conductivity (ECa) sensors are commonly used to survey soil spatial variability. It is essential for surveys to have temporal stability to ensure correct medium- and long-term decisions. The aim of this study was to assess the temporal stability of MZ patterns using different types of ECa sensors, namely an ECa contact-type sensor (Veris 2000 XA, Veris Technologies, Salina, KS, USA) and an electromagnetic induction sensor (EM-38, Geonics Ltd., Mississauga, ON, Canada). These sensors were used in four fields of dryland pastures in the Alentejo region of Portugal. The first survey was carried out in October 2018, and the second was carried out in September 2020. Data processing involved synchronizing the geographic coordinates obtained using the two types of sensors in each location and establishing MZs based on a geostatistical analysis of elevation and ECa data. Although the basic technologies have different principles (contact versus non-contact sensors), the surveys were carried out at different soil moisture conditions and were temporarily separated (about 2 years); the ECa measurements showed statistically significant correlations in all experimental fields (correlation coefficients between 0.449 and 0.618), which were reflected in the spatially stable patterns of the MZ maps (averaging 52% of the total area across the four experimental fields). These results provide perspectives for future developments, which will need to occur in the creation of algorithms that allow the spatial variability and temporal stability of ECa to be validated through smart soil sampling and analysis to generate recommendations for sustained soil amendment or fertilization.

Monitoring respiratory state from near-infrared face video images

In this paper, we propose an algorithm for estimating respiratory state using near-infrared facial video images. Estimation of respiratory state is an important indicator for early detection of respiratory diseases. In particular, there is a demand for monitoring respiratory state during the night. One method of monitoring respiratory state is to use contact-type sensors. However, this method requires the installation of many sensors and a visit to a hospital, which place a burden on patients. Therefore, we propose to acquire respiratory-induced features from near-infrared face video images and investigate their similarity to measurements obtained with a respirometer for non-contact monitoring of respiratory state in the dark. Respiratory-induced features were obtained from pulse wave signals extracted from the face video images. The results showed correlations in several respiratory states. This study opens some perspectives in non-contact monitoring of respiratory states.

Computer Vision-Based Pier Settlement Displacement Measurement of a Multispan Continuous Concrete Highway Bridge under Complex Construction Environments

Various concrete bridges have been built across oceans, valleys, and mountains; however, the settlement displacement of bridge piers caused by environmental changes or self-weight during construction phases often leads to uneven stresses, cracking, and eventual collapse. To address the labor-intensive and high-cost issues of pier displacement monitoring using contact-type sensors, this paper proposes an automatic vision-based method for measuring pier settlement displacement under complex construction environments, such as complex image backgrounds, varying ambient light, and camera movement. In the proposed method, a deep learning network was first employed to eliminate the adverse effect of complex image backgrounds and varying ambient light on the accuracy of target detection; then, an adaptive displacement extraction algorithm without a human-computer interaction process was developed to automatically extract the center coordinates of targets attaching to the bridge piers and reference platform; finally, the pier settlement displacement was calculated by using the relative displacements obtained by a dual camera system to eliminate the measurement error caused by camera translation and rotation movements. Laboratory tests of a cantilever beam and field tests of a continuous multispan concrete girder highway bridge under construction have successfully validated the effectiveness and robustness of the developed methodology. The results obtained in this paper can provide some insights for engineers in applying computer vision technology for the real-time monitoring of bridge displacements.

A non-contact vehicle weighing approach based on bridge weigh-in-motion framework and computer vision techniques

Bridge weigh-in-motion (BWIM) system plays increasingly important role in vehicle load monitoring and bridge maintenance due to advantages of high efficiency and little impact on traffic. However, the contact-type sensors in a traditional BWIM system should be installed in advance and is environment-sensitive, which hinders its application. To this end, a vision-based contactless BWIM (cBWIM) system is proposed. The core idea of the current method is to replace the traditional contact-type sensors by two consumer cameras to realize real-time monitoring of bridge displacement and vehicle trajectory. The cBWIM system consists of three modules, including the axle weight identification algorithm, the bridge displacement monitoring system and the vehicle trajectory tracking system. The influence line-based Moses algorithm is utilized for identification of axle weight. For bridge displacement monitoring, the minimum bounding rectangle-based similarity matching method is proposed to achieve high accuracy measurement of displacement and real-time data processing. For vehicle trajectory tracking, a deep learning-based YOLOv5s algorithm is adopted as a detector, combined with DeepSort algorithm to realize real-time tracking of vehicle position. The cBWIM system is applicable for non-uniform motion and vehicle occlusion cases. Laboratory tests covering various cases (i.e. uniform motion, non-uniform motion and visual occlusion) and field measurement on a full-scale bridge have demonstrated that the proposed method is effective and reliable and can provide alternative tool to the current vehicle weighing methods.

Interpretation of Bridge Health Monitoring Data from Satellite InSAR Technology

This paper presents a study on applying satellite Interferometric Synthetic Aperture Radar (InSAR) technology for the remote monitoring of road bridges and interpreting the results from a structural standpoint. The motivation behind this study arises from the widespread deterioration observed in many road bridges worldwide, leading to the need for large-scale, economic, and effective structural health monitoring (SHM) techniques. While traditional contact-type sensors have cost sustainability limitations, remote sensing techniques, including satellite-based InSAR, offer interesting alternative solutions. The objective of this study is three-fold: (i) to process InSAR data specifically for road bridges in operational conditions through the Multi-Temporal InSAR technique and extract displacement time series of reflective targets on their decks; (ii) to interpret the observed millimetric bridge displacements to verify the consistency with expected response to environmental loads and the possibility to detect unexpected behaviours; and (iii) to investigate the correlation between bridge displacements and environmental loads as temperature and river water flow variations. The study focuses on the multi-span prestressed concrete A22 Po River Bridge in Italy, utilising a dataset of X-Band HIMAGE mode Stripmap images acquired over eight years by the satellite constellation COSMO-SkyMed. The study demonstrates the effectiveness of InSAR-based SHM in detecting temperature-induced displacements and identifying different bridge spans simply by studying the sign of the correlation between displacements and temperature variation. It also reveals an unexpected behaviour in a portion of the bridge retrofitted to prevent scour issues a few years before the dataset start date. Furthermore, the correlation between pier displacements and river level variations underscores the importance of considering environmental factors and the geotechnical characteristics of the foundation soils in bridge monitoring. The results obtained from this study are significant with a view to using this satellite InSAR-based monitoring for early detection of anomalous bridge behaviour on a large scale.

A Feasibility Study on Extension of Measurement Distance in Vision Sensor Using Super-Resolution for Dynamic Response Measurement.

The current civil infrastructure conditions can be assessed through the measurement of displacement using conventional contact-type sensors. To address the disadvantages of traditional sensors, vision-based sensor measurement systems have been derived in numerous studies and proven as an alternative to traditional sensors. Despite the benefits of the vision sensor, it is well known that the accuracy of the vision-based displacement measurement is largely dependent on the camera extrinsic or intrinsic parameters. In this study, the feasibility study of a deep learning-based single image super-resolution (SISR) technique in a vision-based sensor system is conducted to alleviate the low spatial resolution of image frames at long measurement distance ranges. Additionally, its robustness is evaluated using shaking table tests. As a result, it is confirmed that the SISR can reconstruct definite images of natural targets resulting in an extension of the measurement distance range. Additionally, it is determined that the SISR mitigates displacement measurement error in the vision sensor-based measurement system. Based on this fundamental study of SISR in the feature point-based measurement system, further analysis such as modal analysis, damage detection, and so forth should be continued in order to explore the functionality of SR images by applying low-resolution displacement measurement footage.

Coiling an optical fiber for long-range dynamic displacement and force sensing

Optical fiber sensors have been applied on detecting various physical parameters. In this work, inspired by the mechanism of tendril coiling, we designed a spring-like sensor using optical fibers based on macro bending loss. The proposed optical-fiber spring has good elasticity like a mechanical spring and thus can be used to measure large displacement. As a contact-type sensor, the fabricated optical-fiber spring can measure dynamic displacement, whose range is 4 orders of magnitude higher than that of the point-wise fiber Bragg grating displacement sensor. In addition to the point-wise displacement sensing, we further integrated the optical-fiber spring with a Kresling origami to form a force sensor.

Vision-based real-time structural vibration measurement through deep-learning-based detection and tracking methods

Structural vibration measurement is crucial in structural health monitoring and structural laboratory tests. Traditional contact type sensors are usually required to be attached to the test specimens, which may be difficult to install, and may affect the structural properties and response. Non-contact type wireless sensors are usually expensive and require specialized workers to install and operate. In recent years, vision-based tracking methods for structural vibration measurement have gained increasing interests due to their high accuracy, non-contact feature and low cost. However, traditional vision-based tracking algorithms are susceptible to external environmental conditions such as illumination and background noise. In this paper, two real-time methods, YOLOv3-tiny and YOLOv3-tiny-KLT, are proposed to track structural motions. In the first method, YOLOv3-tiny is established based on the YOLOv3 architecture to localize customized markers where structural displacements are directly determined from the bounding boxes generated. The second method, YOLOv3-tiny-KLT, is a more advanced method which combines the YOLOv3-tiny detector and the traditional KLT tracking algorithm. The pretrained YOLOv3-tiny is deployed to localize the targets automatically, which will then be tracked by Kanade‐Lucas‐Tomasi algorithm. YOLOv3-tiny is intended to provide baseline vibration measurement when the KLT tracking gets lost. The proposed methods were implemented for the videos of shake table tests on a two-storey steel structure. Parametric studies were conducted for the YOLOv3-tiny-KLT method to examine its sensitivity to the tracking parameters. The results show that the proposed method is capable of achieving real-time speed and high accuracy, when compared with the traditional displacement sensors including linear variable differential transducer (LVDT) and String Pots. It is also found that the combined YOLOv3-tiny-KLT approach achieves higher accuracy than YOLOv3-tiny only method, and higher robustness than KLT only method against illumination changes and background noise.

Vision-based model updating and evaluation of miter gates on inland waterways

Miter gates are critical components of inland waterways infrastructure, typically used as both the door for and damming surface of locks. Therefore, knowing the condition of miter gates is especially important to ensure their continued operation. Measured displacements of the gates under load are known to be informative of their condition. Such measurements can also be used to update numerical models of the structure that can serve as part of a digital twin, allowing for more comprehensive management of the infrastructure. However, model updating of large-scale infrastructure requires sufficient displacement data that are often challenging to obtain, due in part to inaccessibility and a lack of a viable reference location for contact-type sensors. Vision-based displacement measurements have been proposed to inform a model-updating scheme of the miter gates; however, several challenges have prevented field implementation of these technologies. First, determining the appropriate location and orientation of cameras is critical to obtaining useful displacement information for model updating. In addition, environmental factors that can negatively affect vision-based structural displacement measurements such as lighting changes and camera motion need to be remediated. Moreover, interpreting displacement from photographic imagery is abstract because 2D images can only capture a projection of displacements of the 3D structure or the 3D numerical models. This research addresses each of the challenges through the use of a graphics-based digital twin (GBDT). Prior to conducting the field survey, optimal camera locations and orientations are explored. Additionally, a robust optical flow method is developed to mitigate the negative influence from environment factors on displacement measurements. Finally, displacement from the photographic survey is interpreted by projecting displacement from the numerical models onto the 2D image plane using the GBDT; the difference between these displacements is used to inform a heuristic-based model updating strategy. The proposed approach is demonstrated on the miter gates at The Dalles Lock and Dam. This initial study focuses on determining degradation parameters to which the displacement measurements of the miter gate are most sensitive, laying the groundwork for future rigorous model updating to provide the necessary information regarding repairs and retrofits that would otherwise be difficult to obtain. The main novelty and contribution of this research lie in how the practical challenges of field implementation of existing technologies regarding vision-based model updating are effectively resolved, thus bridging the gap between theory and application to large-scale civil infrastructure.

Acoustic-Based Machine Condition Monitoring—Methods and Challenges

The traditional means of monitoring the health of industrial systems involves the use of vibration and performance monitoring techniques amongst others. In these approaches, contact-type sensors, such as accelerometer, proximity probe, pressure transducer and temperature transducer, are installed on the machine to monitor its operational health parameters. However, these methods fall short when additional sensors cannot be installed on the machine due to cost, space constraint or sensor reliability concerns. On the other hand, the use of acoustic-based monitoring technique provides an improved alternative, as acoustic sensors (e.g., microphones) can be implemented quickly and cheaply in various scenarios and do not require physical contact with the machine. The collected acoustic signals contain relevant operating health information about the machine; yet they can be sensitive to background noise and changes in machine operating condition. These challenges are being addressed from the industrial applicability perspective for acoustic-based machine condition monitoring. This paper presents the development in methodology for acoustic-based fault diagnostic techniques and highlights the challenges encountered when analyzing sound for machine condition monitoring.

Vibration signal separation of rotating cylindrical structure through target-less photogrammetric approach

Vibration analysis is one of the most widely used methodologies for structural health monitoring. For vibration monitoring of structures, contact-type sensors, such as gap sensors, are commonly utilized. However, these devices can add mass loading to a lightweight structure, resulting in degradation in performance of the structure. As an alternative, photogrammetric approaches, such as digital image correlation and point-tracking techniques have been proposed for vibration measurement. Despite the many advantages of image-based techniques applied by non-contact-type sensors, these are ineffective in detecting vibrations at the outer edge of a rotating cylinder-shaped structure because tracking of the speckles or targets on the backside of a structure invisible to a camera is impossible. Furthermore, vibrations occurring in structures are usually attributed to multiple factors; to identify these factors individually, the vibration signals should be detected separately. In this study, we introduce a noncontact-based vibration signal separation technique that can be used to recognize multiple factors that cause a structure to vibrate at different frequency rates. For this purpose, the vibration occurring in a structure is first detected using an edge-based vibration measurement technique. A phase-based motion magnification technique is then used to generate multiple magnified videos at different dominant frequencies. Subsequently, vibrations were detected in multiple magnified videos to separate vibration signals. To demonstrate the effectiveness of the proposed method, quantitative and qualitative assessments were conducted on both simulation and real video datasets containing vibrations with multiple frequency rates. The results generated by this method can be utilized for health monitoring of various types of structures on which targets, necessary for the general photogrammetry approach for monitoring purposes, are not attached. Furthermore, the magnified videos generated in this study can be used for the visualization and mode identification of existing vibration signals at different frequencies in a structure.

Development of a low-cost vision-based real-time displacement system using Raspberry Pi

Vision-based real-time displacement measurement systems can measure the structural displacements without contact and output the results immediately, overcoming some drawbacks of traditional contact-type sensors and off-line vision sensors in field applications. However, high cost and complexity of installation restrict the widespread application of the off-the-shelf vision-based real-time displacement systems. In this work, a fast image processing software is developed for operation in single board computers (SBC), firstly, based which a low-cost and portable vision-based real-time displacement measurement system using Raspberry Pi is constructed for civil structural health monitoring. To evaluate the performance of the system in field applications, a new method for validation of on-site measurement accuracy is proposed. The experimental results show that the constructed system accurately measured multipoint structural displacements of a long-span suspension bridge in real-time, with at least 1/17 pixel accuracy, providing a cost-efficient, convenient, high-precision, and contactless approach to obtain the displacement responses of infrastructures in the field.

Application of infrared thermal imaging technique in in-situ temperature field measurement of hydrate-bearing sediment under thermal stimulation

Natural gas hydrate is a green energy with great development prospects. The exploitation of natural gas hydrate has been attracting much attention. The temperature field distribution of hydrate-bearing sediment is a crucial indication on the heat or mass transfer characteristics and thermal properties of hydrate-bearing sediment during hydrate exploitation. However, to date, the acquisition means of temperatures of hydrate-bearing sediment have been limited to the discrete temperature measurement by using the contact-type temperature sensors inserted into hydrate sediment. The continuous evolution of temperature field of hydrate-bearing sediment can be hardly achieved. Thus, in this study, a new method applying the non-contact and non-destructive infrared thermal imager was developed and applied in the in-situ measurement of the continuous evolution of temperature field of hydrate-bearing sediment. The results confirm the feasibility of application of the infrared thermal imaging technique in the hydrate-related research. The complete continuous temperature fields of hydrate-bearing sediment and non-hydrate-bearing sediment under thermal stimulation were imaged and compared. The results indicate that the temperature field distribution of hydrate sediment is predominately controlled by the effective thermal conductivity of hydrate-bearing sediment. In addition, the effects of the type of hydrate-bearing sediment, heating power and particle size of sediment were investigated.

Research on a Wireless Temperature Measuring Device Using Inductive Power

With the advancement of urban construction, more and more distribution network lines are changed from overhead lines to cable lines. Currently, the distribution network cable connection generally adopts the European or American cable elbow. If the temperature changes inside the cable elbow can be monitored in time, it will help to find the hidden fault of the cable elbow in time and improve the safe operation level of the power grid. In this paper, the online temperature measurement device of distribution network cable is discussed and researched, and a contact-type inductive wireless temperature measurement sensor suitable for the elbow joint of distribution network cable is designed. The sensor not only can accurately measure temperature, but also has the characteristics of a convenient communication network, safe and reliable information, and strong insulation ability, which is suitable for large-scale promotion and application.

Noncontact vision-based impact force reconstruction and spatial-temporal deflection tracking of a flexible barrier system under rockfall impact

Flexible barrier systems are significant for natural disasters (e.g., rockfalls, mudslides) prevention in mountainous areas. However, monitoring the dynamic behavior of flexible barrier systems subjected to rockfall impact is difficult in practice due to the unpredictability of disasters and the problem of easily broken contact-type sensors. Therefore, this paper proposed a noncontact vision-based methodology, in which deep learning and computer vision techniques are developed to reconstruct impact forces induced by falling rocks and to identify the spatial–temporal deflection of a flexible barrier system from remotely captured image sequences by a high-speed camera. The contribution of this paper lies in two aspects: (1) a lightweight neural network is trained to detect falling rocks from captured images for motion information extraction and impact force reconstruction; (2) a two-dimensional distribution map of velocity amplitude is constructed for tracking the full-field spatial deflection pattern of the flexible barrier system under rockfall impact, and multipoint temporal deflection curves of the studied system are also extracted for maximum elongation calculation. The effectiveness of the proposed method is validated by full-scale experiments of a three-span flexible barrier system under rockfall impact with energies of 250 kJ and 750 kJ. The results show that the reconstructed impact forces, tracked full-field spatial deflection pattern, and extracted multipoint temporal deflection curve by the proposed method agree well with reference values, verifying the correctness and robustness of the proposed method. The obtained results can be directly used for the reliable design and condition assessment of flexible barrier systems under rockfall disasters.

Vision-based interaction force estimation for robot grip motion without tactile/force sensor

Humans perceive an interaction force through the kinesthetic sense or tactile sense. When viewing the image, they estimate the interaction force on the basis of pseudo-haptics. The interaction force of a robot is traditionally measured using a contact-type tactile sensor or a force/torque (F/T) sensor. In this work, we propose a method for estimating the interaction force between a robot and objects during grasping and picking. The method is based on images, without involving an F/T sensor or a tactile sensor. For undeformable objects, more precise force estimation was achieved by simultaneously using RGB and depth images, the robot position, and electrical current. We propose a deep neural network structure that combines DenseNet and a Transformer encoder/decoder for predicting the interaction force. We verified proposed network with generated DB which has recorded the interaction with 41 objects. Additionally, we compared the results with changing the inputs of the network. Our model could estimate the interaction force from various input modalities for both known objects and unseen objects during its training. The results clearly indicate that the proposed method produces the best results compared with other models, with less than 3% error in estimating the interaction force.

Intelligent controller for robust orientation control of smart actuator based parallel manipulator

Flexible parallel manipulators can be used as an orienting device due to its precise positioning ability. The flexibility demands the use of non-conventional actuators and noncontact type/flexible sensors for feedback measurement. This study proposes a novel 2 Degree of Freedom (DoF) parallel manipulator actuated by shape memory alloy springs (SMA) which could be used as an orienting device for a table top Cartesian robot. To mitigate the use of contact type sensor, model based feedback is proposed to obtain SMA length for the feedback control. The inclusion of SMA actuators to parallel mechanism imposes control challenges due to nonlinearities, coupling effects, and disturbances. This work also proposes a novel control scheme based on concept of partition control law to control the SMA spring position without the use of any complex mathematical model. The proposed controller is a combination of two parts Fuzzy PID servo control and the time delay estimation (TDE) based auxiliary control. The fuzzy PID control handles the error tracking and the auxiliary control tries to cancel the nonlinearities and disturbances, thus enhancing the robustness. The performance of the proposed controller is tested for a complex trajectory under disturbances and the results are compared with existing nonlinear controllers which proves superior nature of the proposed controller.

Internet of Things in the Bathroom: Smart Health-Monitoring Bidet System

The Internet of things (IoT) helps our everyday lives such as by monitoring objects and tracking behaviors in various settings, but studies on enhancing the bathroom experience are rare. This article describes full details about development and implementation of a smart health-monitoring bidet based on our study published previously in the conference. A smart bidet system is designed to monitor the users’ health through several contact-type sensors, such as pressure, oxygen, and thermometer. The system is equipped with a built-in artificial intelligence software platform and is designed to detect anal and spinal diseases. The attached sensors normally operate under waterproof conditions: we tested their performances under X6 international protection marking conditions. These devices were designed to operate properly even in extremely waterproof conditions. The temperature, pressure, and oxygen sensors of the bidet system had error rates of about 4.1, 0.6, and 1.1 percent, respectively.