Camera-based Methods Research Articles

Fusion in Dissimilarity Space Between RGB D and Skeleton for Person Re Identification

Person re-identification (Re-id) is one of the important tools of video surveillance systems, which aims to recognize an individual across the multiple disjoint sensors of a camera network. Despite the recent advances on RGB camera-based person re-identification methods under normal lighting conditions, Re-id researchers fail to take advantages of modern RGB-D sensor-based additional information (e.g. depth and skeleton information). When traditional RGB-based cameras fail to capture the video under poor illumination conditions, RGB-D sensor-based additional information can be advantageous to tackle these constraints. This work takes depth images and skeleton joint points as additional information along with RGB appearance cues and proposes a person re-identification method. We combine 4-channel RGB-D image features with skeleton information using score-level fusion strategy in dissimilarity space to increase re-identification accuracy. Moreover, our propose method overcomes the illumination problem because we use illumination invariant depth image and skeleton information. We carried out rigorous experiments on two publicly available RGBD-ID re-identification datasets and proved the use of combined features of 4-channel RGB-D images and skeleton information boost up the rank 1 recognition accuracy.

A Neonatal Phantom for Vital Signs Simulation.

Neonatal intensive care units provide vital medical support for premature infants. The key aspect in neonatal care is the continuous monitoring of vital signs measured using adhesive skin sensors. Since sensors can cause irritation of the skin and lead to infections, research focuses on contact-free, camera-based methods such as infrared thermography and photoplethysmography imaging. The development of image processing algorithms requires large datasets, but recording the necessary data from studies brings tremendous effort and costs. Therefore, realistic patient phantoms would be feasible to create a comprehensive dataset and validate image-based algorithms. This work describes the realization of a neonatal phantom which can simulate physiological vital parameters such as pulse rate and thermoregulation. It mimics the outer appearance of premature infants using a 3D printed base structure coated with several layers of modified, skin-colored silicone. A distribution of red and infrared LEDs in the scaffold enables the simulation of a PPG signal by mimicking pulsative light intensity changes on the skin. Additionally, the body temperature of the phantom is individually adjustable in several regions using heating elements. In the validation process for PPG simulation, the feasibility of setting different pulse frequencies and the variation of oxygen saturation levels was obtained. Furthermore, heating tests showed region-dependent temperature variations between 0.19 °C and 0.81 °C around the setpoint. In conclusion, the proposed neonatal phantom can be used to simulate a variety of vital parameters of preterm infants and, therefore, enables the implementation of image processing algorithms for the analysis of the medical state.

Comparison of noncontact camera-based methods for measuring pulse-rate of awake infants

Comparison of noncontact camera-based methods for measuring pulse-rate of awake infants

Evaluation of biases in remote photoplethysmography methods

This work investigates the estimation biases of remote photoplethysmography (rPPG) methods for pulse rate measurement across diverse demographics. Advances in photoplethysmography (PPG) and rPPG methods have enabled the development of contact and noncontact approaches for continuous monitoring and collection of patient health data. The contagious nature of viruses such as COVID-19 warrants noncontact methods for physiological signal estimation. However, these approaches are subject to estimation biases due to variations in environmental conditions and subject demographics. The performance of contact-based wearable sensors has been evaluated, using off-the-shelf devices across demographics. However, the measurement uncertainty of rPPG methods that estimate pulse rate has not been sufficiently tested across diverse demographic populations or environments. Quantifying the efficacy of rPPG methods in real-world conditions is critical in determining their potential viability as health monitoring solutions. Currently, publicly available face datasets accompanied by physiological measurements are typically captured in controlled laboratory settings, lacking diversity in subject skin tones, age, and cultural artifacts (e.g, bindi worn by Indian women). In this study, we collect pulse rate and facial video data from human subjects in India and Sierra Leone, in order to quantify the uncertainty in noncontact pulse rate estimation methods. The video data are used to estimate pulse rate using state-of-the-art rPPG camera-based methods, and compared against ground truth measurements captured using an FDA-approved contact-based pulse rate measurement device. Our study reveals that rPPG methods exhibit similar biases when compared with a contact-based device across demographic groups and environmental conditions. The mean difference between pulse rates measured by rPPG methods and the ground truth is found to be ~2% (1 beats per minute (b.p.m.)), signifying agreement of rPPG methods with the ground truth. We also find that rPPG methods show pulse rate variability of ~15% (11 b.p.m.), as compared to the ground truth. We investigate factors impacting rPPG methods and discuss solutions aimed at mitigating variance.

Automatic recognition of labor activity: a machine learning approach to capture activity physiological patterns using wearable sensors

PurposeThe first step toward developing an automated construction workers performance monitoring system is to initially establish a complete and competent activity recognition solution, which is still lacking. This study aims to propose a novel approach of using labor physiological data collected through wearable sensors as means of remote and automatic activity recognition.Design/methodology/approachA pilot study is conducted against three pre-fabrication stone construction workers throughout three full working shifts to test the ability of automatically recognizing the type of activities they perform in-site through their lively measured physiological signals (i.e. blood volume pulse, respiration rate, heart rate, galvanic skin response and skin temperature). The physiological data are broadcasted from wearable sensors to a tablet application developed for this particular purpose, and are therefore used to train and assess the performance of various machine-learning classifiers.FindingsA promising result of up to 88% accuracy level for activity recognition was achieved by using an artificial neural network classifier. Nonetheless, special care needs to be taken for some activities that evoke similar physiological patterns. It is expected that blending this method with other currently developed camera-based or kinetic-based methods would yield higher activity recognition accuracy levels.Originality/valueThe proposed method complements previously proposed labor tracking methods that focused on monitoring labor trajectories and postures, by using additional rich source of information from labors physiology, for real-time and remote activity recognition. Ultimately, this paves for an automated and comprehensive solution with which construction managers could monitor, control and collect rich real-time data about workers performance remotely.

Recent Advancements in Non-Destructive Testing Techniques for Structural Health Monitoring

Structural health monitoring (SHM) is an important aspect of the assessment of various structures and infrastructure, which involves inspection, monitoring, and maintenance to support economics, quality of life and sustainability in civil engineering. Currently, research has been conducted in order to develop non-destructive techniques for SHM to extend the lifespan of monitored structures. This paper will review and summarize the recent advancements in non-destructive testing techniques, namely, sweep frequency approach, ground penetrating radar, infrared technique, fiber optics sensors, camera-based methods, laser scanner techniques, acoustic emission and ultrasonic techniques. Although some of the techniques are widely and successfully utilized in civil engineering, there are still challenges that researchers are addressing. One of the common challenges within the techniques is interpretation, analysis and automation of obtained data, which requires highly skilled and specialized experts. Therefore, researchers are investigating and applying artificial intelligence, namely machine learning algorithms to address the challenges. In addition, researchers have combined multiple techniques in order to improve accuracy and acquire additional parameters to enhance the measurement processes. This study mainly focuses on the scope and recent advancements of the Non-destructive Testing (NDT) application for SHM of concrete, masonry, timber and steel structures.

Tuning the Baird Aromatic Triplet State Energy of Cyclooctatetraene to Maximize the Self-Healing Mechanism in Organic Fluorophores

Bright, photostable and non-toxic fluorescent contrast agents are critical for biological imaging. ‘Self-healing’ dyes, in which triplet states are intramolecularly quenched, enable fluorescence imaging by increasing fluorophore brightness and longevity, while simultaneously reducing the generation of reactive oxygen species that promote phototoxicity. Here, we systematically examine the self-healing mechanism in cyanine-class organic fluorophores spanning the visible spectrum. We show that the Baird aromatic triplet state energy of cyclooctatetraene can be physically altered to achieve order of magnitude enhancements in fluorophore brightness and signal-to-noise ratio both in the presence and absence of oxygen. We leverage these advances to achieve direct measurements of large-scale conformational dynamics within single molecules at sub-millisecond resolution using wide-field illumination and camera-based detection methods. These findings demonstrate the capacity to image functionally relevant conformational processes in biological systems in the kilohertz regime at physiological oxygen concentrations and shed important light on the multivariate parameters critical to self-healing organic fluorophore design (Pati et al., PNAS, 2020).

Noncontact Physiological Measurement Using a Camera: A Technical Review and Future Directions.

Using a camera as an optical sensor to monitor physiological parameters has garnered considerable research interest in biomedical engineering in recent decades. Researchers have explored the use of a camera for monitoring a variety of physiological waveforms, together with the vital signs carried by these waveforms. Most of the obtained waveforms are related to the human respiratory and cardiovascular systems, and in addition of being indicative of overall health, they can also detect early signs of certain diseases. While using a camera for noncontact physiological signal monitoring offers the advantages of low cost and operational ease, it also has the disadvantages such as vulnerability to motion and lack of burden-free calibration solutions in some use cases. This study presents an overview of the existing camera-based methods that have been reported in recent years. It introduces the physiological principles behind these methods, signal acquisition approaches, various types of acquired signals, data processing algorithms, and application scenarios of these methods. It also discusses the technological gaps between the camera-based methods and traditional medical techniques, which are mostly contact-based. Furthermore, we present the manner in which noncontact physiological signal monitoring use has been extended, particularly over the recent years, to more day-to-day aspects of individuals' lives, so as to go beyond the more conventional use case scenarios. We also report on the development of novel approaches that facilitate easier measurement of less often monitored and recorded physiological signals. These have the potential of ushering a host of new medical and lifestyle applications. We hope this study can provide useful information to the researchers in the noncontact physiological signal measurement community.

The Antarium: A Reconstructed Visual Reality Device for Ant Navigation Research.

We constructed a large projection device (the Antarium) with 20,000 UV-Blue-Green LEDs that allows us to present tethered ants with views of their natural foraging environment. The ants walk on an air-cushioned trackball, their movements are registered and can be fed back to the visual panorama. Views are generated in a 3D model of the ants’ environment so that they experience the changing visual world in the same way as they do when foraging naturally. The Antarium is a biscribed pentakis dodecahedron with 55 facets of identical isosceles triangles. The length of the base of the triangles is 368 mm resulting in a device that is roughly 1 m in diameter. Each triangle contains 361 blue/green LEDs and nine UV LEDs. The 55 triangles of the Antarium have 19,855 Green and Blue pixels and 495 UV pixels, covering 360° azimuth and elevation from −50° below the horizon to +90° above the horizon. The angular resolution is 1.5° for Green and Blue LEDs and 6.7° for UV LEDs, offering 65,536 intensity levels at a flicker frequency of more than 9,000 Hz and a framerate of 190 fps. Also, the direction and degree of polarisation of the UV LEDs can be adjusted through polarisers mounted on the axles of rotary actuators. We build 3D models of the natural foraging environment of ants using purely camera-based methods. We reconstruct panoramic scenes at any point within these models, by projecting panoramic images onto six virtual cameras which capture a cube-map of images to be projected by the LEDs of the Antarium. The Antarium is a unique instrument to investigate visual navigation in ants. In an open loop, it allows us to provide ants with familiar and unfamiliar views, with completely featureless visual scenes, or with scenes that are altered in spatial or spectral composition. In closed-loop, we can study the behavior of ants that are virtually displaced within their natural foraging environment. In the future, the Antarium can also be used to investigate the dynamics of navigational guidance and the neurophysiological basis of ant navigation in natural visual environments.

Joint vehicle detection and distance prediction via monocular depth estimation

Vehicle detection and distance estimation are critical components of driver assistance system and self-driving system, and considerable different frameworks have been investigated such as radar, laser and camera-based. Among them, camera-based vehicle detection and distance estimation have an obvious advantage over other systems in that it needs lower cost. However, existing camera-based methods are not robust enough under complex driving scenes. In this work, an end-to-end deep convolutional neural network framework is proposed to jointly detect vehicles and estimate vehicle distance efficiently. Specifically, a monocular depth estimation method is designed to transform the RGB appearance information into depth modality information. Then the vehicle detection module takes the RGB and depth image as inputs to improve the detection performance. Finally, the distance estimation module employs the detection results and the estimated depth information to predict the distance more precisely. The whole network can be trained in an end-to-end manner with the multi-task loss function. The proposed framework is evaluated on the public vehicle detection benchmark KITTI to show the effectiveness of the proposed framework. Moreover, the performance of three proposed sub-modules are also analysed separately to give a more comprehensive evaluation of the designed framework.

American Sign Language Recognition Using an FMCW Wireless Sensor (Student Abstract)

In today's digital world, rapid technological advancements continue to lessen the burden of tasks for individuals. Among these tasks is communication across perceived language barriers. Indeed, increased attention has been drawn to American Sign Language (ASL) recognition in recent years. Camera-based and motion detection-based methods have been researched extensively; however, there remains a divide in communication between ASL users and non-users. Therefore, this research team proposes the use of a novel wireless sensor (Frequency-Modulated Continuous-Wave Radar) to help bridge the gap in communication. In short, this device sends out signals that detect the user's body positioning in space. These signals then reflect off the body and back to the sensor, developing thousands of cloud points per second, indicating where the body is positioned in space. These cloud points can then be examined for movement over multiple consecutive time frames using a cell division algorithm, ultimately showing how the body moves through space as it completes a single gesture or sentence. At the end of the project, 95% accuracy was achieved in one-object prediction as well as 80% accuracy on cross-object prediction with 30% other objects' data introduced on 19 commonly used gestures. There are 30 samples for each gesture per person from three persons.

Multi-Sensor Fusion in Dynamic Environment using Evidential Grid Mapping

The performance of autonomous agents in both commercial and consumer applications increases along with their situational awareness. Tasks such as obstacle avoidance, agent to agent interaction, and path planning are directly dependent upon their ability to convert sensor readings into scene understanding. Central to this is the ability to detect and recognize objects. Many object detection methodologies operate on a single modality such as vision or LiDAR. Camera-based object detection models benefit from an abundance of feature-rich information for classifying different types of objects. LiDAR-based object detection models use sparse point clouds, where each point contains accurate 3D position of object surfaces. Camera-based methods lack accurate object to lens distance measurements, while LiDAR-based methods lack dense feature-rich details. By utilizing information from both camera and LiDAR sensors, advanced object detection and identification is possible. In this work, we introduce a deep learning framework for fusing these modalities and produce a robust real-time 3D bounding box object detection network. We demonstrate qualitative and quantitative analysis of the proposed fusion model on the popular KITTI dataset.

Pseudo real-time continuous measurements of particle preferential concentration in homogeneous isotropic turbulence

Preferential concentration is a phenomenon where particles in turbulent flow concentrate in regions of low vorticity and high strain rate, forming clusters. This phenomenon is prevalent in nature and various industrial environments. Previous measurement systems for detecting this phenomenon have been based mostly on discrete images, and is relatively slow because of the image post processing involved. We have developed a new fast-response continuous measurement system comprised of a photomultiplier tube (PMT), custom-designed lens tube, and a continuous wave Nd:YAG laser. Experiments were conducted in a chamber that creates homogeneous isotropic turbulence with no mean flow. Image-based Voronoi analysis and box counting analysis from a high-speed camera were compared against the new PMT-based method for three Stokes number cases of near 0, 2.2, and 10. As the Stokes number was varied, the new technique showed similar results compared to the conventional camera-based methods.

SLNet: Stereo face liveness detection via dynamic disparity-maps and convolutional neural network

Current state-of-the-art dual camera-based face liveness detection methods utilize either hand-crafted features, such as disparity, or deep texture features to classify a live face and face Presentation Attack (PA). However, these approaches limit the effectiveness of classifiers, particularly deep Convolutional Neural Networks (CNN) to unknown face PA in adverse scenarios. In contrast to these approaches, in this paper, we show that supervising a deep CNN classifier by learning disparity features using the existing CNN layers improves the performance and robustness of CNN to unknown types of face PA. For this purpose, we propose to supervise a CNN classifier by introducing a disparity layer within CNN to learn the dynamic disparity-maps. Subsequently, the rest of the convolutional layers, following the disparity layer, in the CNN are supervised using the learned dynamic disparity-maps for face liveness detection. We further propose a new video-based stereo face anti-spoofing database with various face PA and different imaging qualities. Experiments on the proposed stereo face anti-spoofing database are performed using various test case scenarios. The experimental results indicate that our proposed system shows promising performance and has good generalization ability.

Evaluation of inertial Sensors and optical measurement methods for head and shoulder angle motion analysis

Abstract This paper describes the advantages and disadvantages of inertial sensors and stationary camera-based measuring methods for the application of detecting head and shoulder angles. For this purpose three inertial sensor modules from Shimmer Sensing and a Kinect 2 camera system are used. The recorded data of both systems are converted into a comparable form. Based on an evaluation of the recorded measurement data, the inertial sensor and the camera-based methods are compared with regard to their suitability for detecting movements in the head and shoulder area.

Novel IoT-Based Privacy-Preserving Yoga Posture Recognition System Using Low-Resolution Infrared Sensors and Deep Learning

In recent years, the number of yoga practitioners has been drastically increased and there are more men and older people practice yoga than ever before. Internet of Things (IoT)-based yoga training system is needed for those who want to practice yoga at home. Some studies have proposed RGB/Kinect camera-based or wearable device-based yoga posture recognition methods with a high accuracy; however, the former has a privacy issue and the latter is impractical in the long-term application. Thus, this paper proposes an IoT-based privacy-preserving yoga posture recognition system employing a deep convolutional neural network (DCNN) and a low-resolution infrared sensor-based wireless sensor network (WSN). The WSN has three nodes ( ${x}$ , ${y}$ , and ${z}$ -axes) where each integrates $8\times 8$ pixels’ thermal sensor module and a Wi-Fi module for connecting the deep learning server. We invited 18 volunteers to perform 26 yoga postures for two sessions each lasted for 20 s. First, recorded sessions are saved as .csv files, then preprocessed and converted to grayscale posture images. Totally, 93 200 posture images are employed for the validation of the proposed DCNN models. The tenfold cross-validation results revealed that F1-scores of the models trained with xyz (all 3-axes) and ${y}$ (only ${y}$ -axis) posture images were 0.9989 and 0.9854, respectively. An average latency for a single posture image classification on the server was 107 ms. Thus, we conclude that the proposed IoT-based yoga posture recognition system has a great potential in the privacy-preserving yoga training system.

IMU-based Assessment of Ankle Inversion Kinematics and Orthosis Migration.

Ankle orthoses are used to prevent injuries during very fast inversion motions of the ankle. In order to provide proper protection, they must fit well and migrate as little as possible. Inertial measurement units (IMUs) have become a useful tool for accurate motion analysis and are frequently used for gait analysis. In the present paper, we examine orthosis migration and inversion kinematics of human ankles protected and unprotected by an orthosis. We present two test benches that were developed for these purposes, as well as a set of inertial sensor fusion methods that are used to determine kinematic parameters from the sensor readings. To avoid the common but restrictive assumption of a homogeneous magnetic field, we determine all motion parameters without the use of magnetometer readings. We conduct a measurement series to compare the proposed IMU-based method to alternative camera-based and goniometer-based methods. Using two different ankle orthosis prototypes, we demonstrate that the proposed IMU-based methods facilitate accurate assessment of orthosis migration and ankle inversion kinematics.

Objective 3D Printed Surface Quality Assessment Based on Entropy of Depth Maps.

A rapid development and growing popularity of additive manufacturing technology leads to new challenging tasks allowing not only a reliable monitoring of the progress of the 3D printing process but also the quality of the printed objects. The automatic objective assessment of the surface quality of the 3D printed objects proposed in the paper, which is based on the analysis of depth maps, allows for determining the quality of surfaces during printing for the devices equipped with the built-in 3D scanners. In the case of detected low quality, some corrections can be made or the printing process may be aborted to save the filament, time and energy. The application of the entropy analysis of the 3D scans allows evaluating the surface regularity independently on the color of the filament in contrast to many other possible methods based on the analysis of visible light images. The results obtained using the proposed approach are encouraging and further combination of the proposed approach with camera-based methods might be possible as well.

Is the high-speed camera-based method a plausible option for bar velocity assessment during resistance training?

This study aims to identify the measurement error associated with the mean movement velocity when using high-speed camera-based methods and video analysis during resistance training. Eleven resistance-trained men (26.0 ± 3.4 years) completed a progressive loading test in bench press exercise. Measurements from concentric mean velocity (MV), distance and time were obtained from a linear velocity transducer (T-Force) and videos recorded with high speed cameras on readily available smartphones (Samsung S6, Xiaomi A1, and iPhone X) and digital photo cameras (Casio FH20). Videos were examined using video analysis software (Kinovea). Despite the high correlations detected, the Bland-Altman analyses revealed that all high speed cameras produced substantial overestimation of barbell MV against high loads >60% 1RM (MV error = 0.06 ± 0.03 m·s−1 to 0.08 ± 0.04 m·s−1), but mainly against low loads <60% 1RM (MV error = 0.13 ± 0.06 m·s−1 to 0.20 ± 0.09 m·s−1). The maximum estimation error of the load being lifted (%1RM) was considerable both for low (8.5–12.7% 1RM) and high loads (13.9–22.6% 1RM). Among other practical limitations, the video-based system using different high-speed cameras and smartphone devices presents severe limitations when estimating mean concentric velocity, especially when recording low loads at high velocity.

In-Bed Pose Estimation: Deep Learning With Shallow Dataset.

This paper presents a robust human posture and body parts detection method under a specific application scenario known as in-bed pose estimation. Although the human pose estimation for various computer vision (CV) applications has been studied extensively in the last few decades, the in-bed pose estimation using camera-based vision methods has been ignored by the CV community because it is assumed to be identical to the general purpose pose estimation problems. However, the in-bed pose estimation has its own specialized aspects and comes with specific challenges, including the notable differences in lighting conditions throughout the day and having pose distribution different from the common human surveillance viewpoint. In this paper, we demonstrate that these challenges significantly reduce the effectiveness of the existing general purpose pose estimation models. In order to address the lighting variation challenge, the infrared selective (IRS) image acquisition technique is proposed to provide uniform quality data under various lighting conditions. In addition, to deal with the unconventional pose perspective, a 2- end histogram of oriented gradient (HOG) rectification method is presented. The deep learning framework proves to be the most effective model in human pose estimation; however, the lack of large public dataset for in-bed poses prevents us from using a large network from scratch. In this paper, we explored the idea of employing a pre-trained convolutional neural network (CNN) model trained on large public datasets of general human poses and fine-tuning the model using our own shallow (limited in size and different in perspective and color) in-bed IRS dataset. We developed an IRS imaging system and collected IRS image data from several realistic life-size mannequins in a simulated hospital room environment. A pre-trained CNN called convolutional pose machine (CPM) was fine-tuned for in-bed pose estimation by re-training its specific intermediate layers. Using the HOG rectification method, the pose estimation performance of CPM improved significantly by 26.4% in the probability of correct key-point (PCK) criteria at PCK0.1 compared to the model without such rectification. Even testing with only well aligned in-bed pose images, our fine-tuned model still surpassed the traditionally tuned CNN by another 16.6% increase in pose estimation accuracy.