Vision-based Monitoring Research Articles

Non-invasive technique for real-time myocardial infarction detection using faster R-CNN

The medical history explores that Myocardial Infarction has been one of the leading factors of death in human beings since several decades globally. The researchers’ key tasks are to emerge a novel real-time health vision-based monitoring system with added measurement features like high accuracy, robust, reliable, low-cost, low power with high data security. The main purpose of this research is to bestow an advanced non-invasive algorithmic approach for detecting the chest pain posture and fall posture based vital signs of Myocardial Infarction and analyzing the performance of a Faster Region-based Convolution Neural Network algorithm. This object detection computer vision technique is simulated for 3000 three-dimensional real-life indoor environment RGB color images for two datasets Nanyang Technological University Red Blue Green, and Depth dataset and private dataset-RMS trained datasets using TensorFlow object detection Application Programming Interface. The 3D RGB Images of NTU RGB database used for Vital Signs of Myocardial Infarction performance analysis is an improved approach. The simulation results have been compared with the existing works. The demonstrated results of ResNet-101 Faster RCNN showed the evaluated metric values: high mean precision and average recall value is a major contribution in this work.

A computer vision-based, in-situ springback monitoring technique for bending of large profiles

Bending processes have various advantages, such as less processing time, lower number of tooling parts, and cost compared to other manufacturing processes. However, one of the disadvantages of a bending process is the inevitable springback problem, which entails geometrical inaccuracy. Many researchers have made attempts to effectively measure springback in-line to control product quality and compensate for variability. While measurement tools and machines are available to measure springback, they might not be able to accommodate large products due to the size limit of measurement devices. Nevertheless, sensor-based monitoring is becoming critical to control product quality and to move towards Industry 4.0. In this paper, an in-situ springback monitoring technique for bending of large-size profiles is proposed to overcome the measurement restrictions for such profiles. A computer vision technique with the circular Hough transform was used to evaluate springback. The marked points on a profile were used to track the deformation of the workpiece. However, a weakness with image processing is to recognize the points from the complex background. Instead of employing global search for the points in an image frame, the marked points were detected by locally setting regions based on forming parameters such as a bending angle and stretching level. Springback was calculated by the change of position of those points. The results of springback monitoring were validated with the physically measured data from experiments. Based on this measurement technique, the feasibility of a computer vision-based springback monitoring in large-size profile bending is discussed in detail.

Additional Cover

The cover image is based on the Research Article Computer vision-based monitoring of the 3-D structural deformation of an ancient structure induced by shield tunneling construction by Xiao-Wei Ye et al., https://doi.org/10.1002/stc.2702.

A few-shot learning approach for database-free vision-based monitoring on construction sites

This paper proposes a few-shot learning approach that can successfully learn and detect new construction objects when only a few training data are given. The proposed approach includes few-shot model design and meta-learning processes. To validate the approach, the authors conducted experiments using a popular construction benchmark dataset, AIMDataset. Even if only 20 training images were provided to a new construction object, the few-shot learning could build an object detection model with the mean Average Precision of 73.1% on average, whereas the performance of the existing supervised learning was limited to 36.5%. The results imply that the proposed approach can successfully learn and detect new types of construction objects only with few labeled images given, enabling to reduce the number of training images while maximizing the model performance. It would be then possible to save human efforts required for data labeling and enhance the practicality of vision-based construction monitoring systems.

Mast Arm Monitoring via Traffic Camera Footage: A Pixel-Based Modal Analysis Approach

Traffic signal mast arm structures must be regularly inspected for cracking, bolt loosening, and other signs of deterioration. Due to large inventories, physical inspections and/or dedicated monitoring systems can be prohibitively time-consuming and expensive to implement at a large scale. However, the growing use of vision-based methods for structural monitoring applications introduces the possibility of leveraging video footage from existing traffic cameras for this purpose. The extraction of dynamic properties (i.e., natural frequencies and damping) from this footage could be employed in detecting possible signs of deterioration. This study presents a vision-based monitoring method which uses a single traffic camera to identify the modal properties of the supporting traffic signal mast arm. This was achieved via operational modal analysis on pixel displacements obtained from a traffic camera mounted on a traffic signal mast arm in Norfolk, VA, monitored during July, 2019. First, sub-pixel displacements were extracted frame-by-frame using weighted centroid tracking of pavement markings. Then, covariance-driven stochastic subspace identification (SSI-Cov) was employed to extract the mast arm fundamental frequencies, damping ratios, and mode shapes. For validation of the vision-based results, SSI-Cov was also applied to acceleration data recorded by two high-sensitivity accelerometers mounted on the structure. In total, the processing was carried out on four different videos and ten acceleration datasets. The vision-based method was able to reliably identify the fundamental frequencies of the structure (Δf < 0.005 Hz mean difference). The associated damping ratios were consistently overestimated but still close in structural terms (Δζ < 0.7% mean difference).

Light-DehazeNet: A Novel Lightweight CNN Architecture for Single Image Dehazing.

Due to the rapid development of artificial intelligence technology, industrial sectors are revolutionizing in automation, reliability, and robustness, thereby significantly increasing quality and productivity. Most of the surveillance and industrial sectors are monitored by visual sensor networks capturing different surrounding environment images. However, during tempestuous weather conditions, the visual quality of the images is reduced due to contaminated suspended atmospheric particles that affect the overall surveillance systems. To tackle these challenges, this article presents a computationally efficient lightweight convolutional neural network referred to as Light-DehazeNet (LD-Net) for the reconstruction of hazy images. Unlike other learning-based approaches, which separately measure the transmission map and the atmospheric light, our proposed LD-Net jointly estimates both the transmission map and the atmospheric light using a transformed atmospheric scattering model. Furthermore, a color visibility restoration method is proposed to evade the color distortion in the dehaze image. Finally, we conduct extensive experiments using synthetic and natural hazy images. The quantitative and qualitative evaluation on different benchmark hazy datasets verify the superiority of the proposed method over other state-of-the-art image dehazing techniques. Moreover, additional experimentation validates the applicability of the proposed method in the object detection tasks. Considering the lightweight architecture with minimal computational cost, the proposed system is encouraged to be incorporated as an integral part of the vision-based monitoring systems to improve the overall performance.

A Vision-Based Monitoring Method for the Looseness of High-Strength Bolt

Bolts are widely applied to the connections of structural components. Under the actions of alternative forces induced by the external loads, the bolts are commonly inevitable to be loosening in engineering, which will be a threat to structural safety. To avoid the reduction of the pretightening force, it is vital to examine the status of bolts periodically. So far, manual inspection with a torque wrench is the most frequently used approach, even though it is time-consuming, labor-intensive, and low frequency of data acquisitions for structural maintenance. Due to the looseness of bolts which commonly follows with relative rotation between bolts and nuts, in this study, a vision-based bolt monitoring system with an Internet of Things (IoT) device is proposed. Specifically, to observe the relative rotation between bolts and nuts, a novel barcode marker (termed PAC-code) is introduced at first. As followed, a corresponding smart device embedded with PAC-code decoding algorithm is also described for the further long-term monitoring of bolt looseness. Finally, a laboratory-based experiment was carried out to validate the reliability and precision of our system. The results indicate that the proposed system is sensitive in the identification of angular changes and can be used to monitor the looseness of bolts as precise as 0.1°. This technique is not only easy to be deployed with high economic efficiency in engineering but also meaningful for the data acquisition in bolts looseness-proof research.

Vision-Based Vibration Monitoring of Structures and Infrastructures: An Overview of Recent Applications

Contactless structural monitoring has in recent years seen a growing number of applications in civil engineering. Indeed, the elimination of physical installations of sensors is very attractive, especially for structures that might not be easily or safely accessible, yet requiring the experimental evaluation of their conditions, for example following extreme events such as strong earthquakes, explosions, and floods. Among contactless technologies, vision-based monitoring is possibly the solution that has attracted most of the interest of civil engineers, given that the advantages of contactless monitoring can be potentially obtained thorough simple and low-cost consumer-grade instrumentations. The objective of this review article is to provide an introductory discussion of the latest applications of vision-based vibration monitoring of structures and infrastructures through an overview of the results achieved in full-scale field tests, as documented in the published technical literature. In this way, engineers new to vision-based monitoring and stakeholders interested in the possibilities of contactless monitoring in civil engineering could have an outline of up-to-date achievements to support a first evaluation of the feasibility and convenience for future monitoring tasks.

Modal identification of building structures using vision-based measurements from multiple interior surveillance cameras

Surveillance cameras are ubiquitous within modern building systems, providing a wealth of data that, to date, has not been utilized for structural health monitoring and condition assessment. In this paper, a vision-based vibration-monitoring technique is introduced, which leverages existing cameras within building structures to extract modal information and assess for damage. Cameras already installed within buildings, such as interior surveillance cameras, are used to measure structural responses (inter-story drifts and rotations) by tracking the cameras’ motion relative to the floor below. By synchronizing vision-based inter-story measurements captured at multiple stories, a complete three-dimensional representation of the building’s motion is recovered. Output-only system identification (frequency domain decomposition) is applied to these vision-based measurements to extract modal information—frequencies and mode shapes. An experimental parametric study on a lab-scale three-story structure is conducted to assess the efficacy of the vision-based monitoring technique for structural modal identification. The primary set of structure/camera geometries considered is representative of actual surveillance camera installations. Broadband white noise is used to excite the structure uni- and bidirectionally, and the resulting ambient and transient responses are measured. The accuracy of the modal information extracted by means of the proposed vision-based technique is assessed through comparison with modal data from accelerometer-based measurements, quantified by the modal assurance criteria. Moreover, the precision, robustness, and repeatability of the method as well as its shortcomings are established and are described in this paper.

Visual Analytics for Operation-Level Construction Monitoring and Documentation: State-of-the-Art Technologies, Research Challenges, and Future Directions

Operation-level vision-based monitoring and documentation has drawn significant attention from construction practitioners and researchers. To automate the operation-level monitoring of construction and built environments, there have been much effort to develop computer vision technologies. Despite their encouraging findings, it remains a major challenge to exploit technologies in real construction projects, implying that there are knowledge gaps in practice and theory. To fill such knowledge gaps, this study thoroughly reviews 119 papers on operation-level vision-based construction monitoring, published in mainstream construction informatics journals. Existing research papers can be categorized into three sequential technologies: (1) camera placement for operation-level construction monitoring, (2) single-camera-based construction monitoring and documentation, and (3) multi-camera-based onsite information integration and construction monitoring. For each technology, state-of-the-art algorithms, open challenges, and future directions are discussed.

Automatic Video Annotation of Human Health Care Action via Clustering

Vision-based activity monitoring provides applications that revolutionized the e-health sector. Considering the potential of crowdsourcing data, to develop large scale applications, the researchers are working on consolidating smart hospital with crowd sourcing data. For creating a meaningful pattern from such huge data, a key challenge is that it needs to be annotated. Especially, the annotation of medical images plays an important role in providing pervasive health services. Although, multiple image annotation methods such as manual and semi-supervised exist. However, high cost and computation time remains a major issue. To overcome the abovementioned issues, a methodology is proposed for automatic annotation of images. The proposed approach is based on three tires namely frame extraction, interest point's generation, and clustering. Since the medical imaging lacks an appropriate dataset for our experimentation. Consequently, we have introduced a new dataset of Human Health care Actions (HHA). The data set comprises of videos related to multiple medical emergencies such as allergy reactions, burn, asthma, brain injury, bleeding, poisoning, heart attack, choking and spinal injury. We have also proposed an evaluation model to assess the effectiveness of the proposed methodology. The promising results of the proposed technique indicate the effectiveness of 78% in terms of Adjusted Rand Index. Furthermore, to investigate the effectiveness of the proposed technique, a comparison is made, by training the neural network classifier with annotated labels generated by proposed methodology and other existing techniques such as semi-supervised and manual methods. The overall precision of the proposed methodology is 0.75 (i.e., 75%) and semi-supervised learning is 0.69 (69%).

Towards database-free vision-based monitoring on construction sites: A deep active learning approach

In order to achieve database-free (DB-free) vision-based monitoring on construction sites, this paper proposes a deep active learning approach that automatically evaluates the uncertainty of unlabeled training data, selects the most meaningful-to-learn instances, and eventually trains a deep learning model with the selected data. The proposed approach thus involves three sequential processes: (1) uncertainty evaluation of unlabeled data, (2) training data sampling and user-interactive labeling, and (3) model design and training. Two experiments were performed to validate the proposed method and confirm the positive effects of active learning: one experiment with active learning and the other without active learning (i.e., with random learning). In the experiments, the research team used a total of 17,000 images collected from actual construction sites. To achieve 80% mean Average Precision (mAP) for construction object detection, the random learning method required 720 training images, whereas only 180 images were sufficient when exploiting active learning. Moreover, the active learning could build a deep learning model with the mAP of 93.0%, while that of the random learning approach was limited to 89.1%. These results demonstrate the potential of the proposed method and highlight the considerable positive impacts of uncertainty-based data sampling on the model's performance. This research can improve the practicality of vision-based monitoring on construction sites, and the findings of this study can provide valuable insights and new research directions for construction researchers.

Malicious attacks detection in crowded areas using deep learning-based approach

With the increasing need to ensure people's safety in crowded areas, the development of a systematic anomaly detection mechanism is becoming indispensable. Here are a few examples of recent malicious attacks targeting crowded areas in big cities: in 2016, a truck driver attacked and killed 84 persons walking in the promenade in Nice, France; and on 19 December, 2016, a truck was deliberately driven into the Christmas market, in Berlin, Germany, killing 12 people and injuring 56 others. These attacks demonstrate the need for efficient monitoring systems to avoid such devastating attacks. To do so, early detection and prevention abilities are vital. Detecting and localizing abnormal events in crowded scenes is important and has significant implications in video surveillance applications. Video surveillance can be challenging, as abnormal events can be unpredictable and changing, based on the context. Accurately detecting and localizing anomalies in videos is a powerful tool that can help to improve security and understand the behavior of anomalies. In this paper, we present an automated visionbased monitoring scheme specifically designed for atypical event-detection and localization in crowded areas.

Vision-Based Monitoring of Post-Tensioned Diagonals on Miter Lock Gate

AbstractMiter gates are structures that act as both the damming surface and doorway of lock chambers and are commonly found on American rivers. Miter gates are assumed to have very little torsional...

AQ360: UAV-Aided Air Quality Monitoring by 360-Degree Aerial Panoramic Images in Urban Areas

Driven by the increasingly serious air pollution problem, nowadays different systems can be used to achieve the monitoring task of air quality index (AQI) in urban areas. In this article, we design a novel unmanned aerial vehicle-aided (UAV-aided) AQI monitoring system, called AQ360, which detects the air quality level from the 360-degree aerial panoramic images taken by the onboard camera. Specifically, we first present our own AQI recognition approach based on the physical form of the haze pictures, where the AQI is jointly decided by the images captured along six directions over the target location. Then, we study the UAV placement problem of selecting UAV's flight altitude and 2-D coordinates during the monitoring process. The objective is to save the system energy consumption while maintaining the accuracy of estimating AQI distribution. For practical considerations, we implement and evaluate the proposed system in real-world scenarios. The results show that our system can provide a lower AQI recognition error compared with existing vision-based monitoring approaches, and energy consumption is also reduced when applying for large-area tasks.

Data-Driven Structural Health Monitoring and Damage Detection through Deep Learning: State-of-the-Art Review.

Data-driven methods in structural health monitoring (SHM) is gaining popularity due to recent technological advancements in sensors, as well as high-speed internet and cloud-based computation. Since the introduction of deep learning (DL) in civil engineering, particularly in SHM, this emerging and promising tool has attracted significant attention among researchers. The main goal of this paper is to review the latest publications in SHM using emerging DL-based methods and provide readers with an overall understanding of various SHM applications. After a brief introduction, an overview of various DL methods (e.g., deep neural networks, transfer learning, etc.) is presented. The procedure and application of vibration-based, vision-based monitoring, along with some of the recent technologies used for SHM, such as sensors, unmanned aerial vehicles (UAVs), etc. are discussed. The review concludes with prospects and potential limitations of DL-based methods in SHM applications.

Evidence-driven sound detection for prenotification and identification of construction safety hazards and accidents

As the construction industry experiences a high rate of casualties and significant economic loss associated with accidents, safety has always been a primary concern. In response, several studies have attempted to develop new approaches and state-of-the-art technology for conducting autonomous safety surveillance of construction work zones such as vision-based monitoring. The current and proposed methods including human inspection, however, are limited to consistent and real-time monitoring and rapid event recognition of construction safety issues. In addition, the health and safety risks inherent in construction projects make it challenging for construction workers to be aware of possible safety risks and hazards according to daily planned work activities. To address the urgent demand of the industry to improve worker safety, this study involves the development of an audio-based event detection system to provide daily safety issues to laborers and through the rapid identification of construction accidents. As an evidence-driven approach, the proposed framework incorporates the occupational injury and illness manual data, consisting of historical construction accident data classified by types of sources and events, into an audio-based safety event detection framework. This evidence-driven framework integrated with a daily project schedule can automatically provide construction workers with prenotifications regarding safety hazards at a pertinent work zone as well as consistently contribute to enhanced construction safety monitoring by audio-based event detection. By using a machine learning algorithm, the framework can clearly categorize the narrowed-down sound training data according to a daily project schedule and dynamically restrict sound classification types in advance. The proposed framework is expected to contribute to an emerging knowledge base for integrating an automated safety surveillance system into occupational accident data, significantly improving the accuracy of audio-based event detection.

A Vision-Based System for Monitoring Elderly People at Home

Assisted living technologies can be of great importance for taking care of elderly people and helping them to live independently. In this work, we propose a monitoring system designed to be as unobtrusive as possible, by exploiting computer vision techniques and visual sensors such as RGB cameras. We perform a thorough analysis of existing video datasets for action recognition, and show that no single dataset can be considered adequate in terms of classes or cardinality. We subsequently curate a taxonomy of human actions, derived from different sources in the literature, and provide the scientific community with considerations about the mutual exclusivity and commonalities of said actions. This leads us to collecting and publishing an aggregated dataset, called ALMOND (Assisted Living MONitoring Dataset), which we use as the training set for a vision-based monitoring approach.We rigorously evaluate our solution in terms of recognition accuracy using different state-of-the-art architectures, eventually reaching 97% on inference of basic poses, 83% on alerting situations, and 71% on daily life actions. We also provide a general methodology to estimate the maximum allowed distance between camera and monitored subject. Finally, we integrate the defined actions and the trained model into a computer-vision-based application, specifically designed for the objective of monitoring elderly people at their homes.

Machine Vision-Based Monitoring Methodology for the Fatigue Cracks in U-Rib-to-Deck Weld Seams

The orthotropic steel-box girder (OSG) is widely used in the construction of a large-scale bridges. Since cumulative damages caused by the heavy vehicles and initial flaws of welding, the bridges with OSGs frequently suffer from fatigue cracks, which are commonly distributed around U-ribs. Hence, the management of fatigue cracks is mandatory in practical engineering. Although some techniques have been adopted for the detection of cracks, the workflow is often labor-intensive, time-consuming, and of low-temporal resolution. Considering the optical visibility of a crack and the limitation of the shape of an over-welding-hole around the U-rib, a machine vision-based monitoring methodology for the fatigue cracks in U-rib-to-deck weld seams is proposed in this paper. To be specific, a specific Internet of Things (IoT) based image acquisition device is first developed and introduced to obtain precisely part-view images of a fatigue crack. As followed, a novel image rectification and stitching method based on a specified coded calibration board is innovated and described for generating a measurable panoramic fatigue crack image. Furthermore, a deep learning-based crack detection-segmentation integrated algorithm is developed to detect and segment the crack areas. Afterwards, a feature extraction procedure based on image processing is explored to obtain the morphological features of a crack, involving its area, length and width. Finally, a field experiment was carried out on a real steel suspension bridge. By comparing the measurements both from manual measuring and vision-based monitoring, the results indicate that the proposed methodology is very promising to monitor the fatigue cracks in U-rib-to-deck weld seams, and the root-mean-square errors in length and width measuring could be 3.0195 mm and 0.003 mm, respectively. This work is not only of practical value to the management and maintenance of the OSG bridges in engineering, but also critical for the researches on fatigue cracks propagation.

Artificial Intelligence Vision-Based Monitoring System for Ship Berthing

This paper proposes a novel artificial intelligence vision-based monitoring system (AVMS) for ship berthing. To dock a ship, it is necessary to accurately estimate the relative distance between the quay wall and the ship. However, maneuvering large ships near a port is a complicated and difficult procedure. Thus, tugboats push the ship and dock it at the berth under the supervision of a pilot, who receives distance information from a berthing aid system (BAS). The conventional BAS based on laser distance sensors, which is the most widely used approach, is high-priced and limited by the size of the ship. Additionally, if there is an obstacle between the ship and the berth, the distance cannot be measured, since it obscures the laser signal. To address this problem, we develop an AVMS sensor module composed of a low-priced camera, a differential global positioning system (DGPS) receiver, and an inertial measurement unit (IMU) with an algorithm to estimate the distance between ship and berth. To evaluate the performance of the proposed AVMS, field tests are performed at Ulsan port in Korea, and the results are compared with a conventional BAS. From the field test results, the AVMS provides highly accurate estimates and shows robust performance in poor weather conditions compared to the conventional BAS. The AVMS can measure the distance between ship and berth regardless of the size of the ship, since it has a wide field of view. In addition, it provides the pilot with real-time image information of the ship approaching the berth to obtain safe ship berthing.