Camera-based System Research Articles

Polarization Independent Optical Coherence Tomography

Optical coherence tomography (OCT) is a well established imaging modality for high-resolution three-dimensional imaging in clinical settings. While imaging, care must be taken to minimize the imaging artifacts related to the polarization differences between the sample and the reference signals. Current OCT systems adopt complicated mechanisms, such as the use of multiple detectors, polarization-maintaining fibers, polarization controllers to achieve polarization artifacts free sample images. Often the polarization controllers need readjustment which is not suitable for clinical settings. In this work, we demonstrate a simple approach that can minimize the polarization-related artifacts in the OCT systems. Polarization artifact-free images are acquired using two orthogonally polarized reference signals where the orthogonal polarization is achieved using a Faraday mirror. In the current approach, only a single detector is required which makes the current approach compatible with swept-source or camera-based OCT systems. Furthermore, no polarization controllers are used in the system which increases the system stability while minimizing the artifacts related to the sample birefringence, polarization change due to the sample scattering, and polarization change due to the optical fiber movements present in the system.

A Novel Camera-Based Approach to Increase the Quality, Objectivity and Efficiency of Aeronautical Meteorological Observations

Despite the general efforts of meteorological services to provide aeronautical observations at all ranges of airports automatically, for some meteorological variables, especially for the ones that highly rely on complex human perception (e.g., prevailing visibility and cloud coverage), reliable, fully automated observations cannot be ensured. This paper introduces novel possibilities to observe prevailing visibility and cloud coverage/height by means of a camera-based observation system that does not necessarily replace, but effectively and synergically amends the standard observations. We present human (and not automated) observations from a remote center that allows for an observer to report meteorological conditions remotely, only using images from cameras installed at the airport. The basic concept of the remote observer was developed within a previous SESAR project. The focus of our methods is set (1) on the quality of information with the occurrence of reduced visibility and enhanced cloud cover in inhomogeneous weather situations and (2) on a comparison of our approaches with those from local human observers. We conclude that for a correct estimation of the prevailing visibility, cloud coverage and cloud types, the automated sensors alone are inadequate; however, the camera-aided remote human approach to observations seems to be a promising supplement to eliminate the sensors’ deficiencies, in terms of the quality (e.g., high quality camera records; no more point measurements), objectivity (e.g., database of archived weather situations) and efficiency (e.g., no need to have an observer physically present at the airport). The possibility to provide observations remotely seems to be advantageous in the COVID-19 and post-COVID-19 era when the society must adapt to different levels of quarantine conditions, affecting and/or disabling standard work and travelling regimes.

A Vision-Based System for In-Sleep Upper-Body and Head Pose Classification.

Sleep quality is known to have a considerable impact on human health. Recent research shows that head and body pose play a vital role in affecting sleep quality. This paper presents a deep multi-task learning network to perform head and upper-body detection and pose classification during sleep. The proposed system has two major advantages: first, it detects and predicts upper-body pose and head pose simultaneously during sleep, and second, it is a contact-free home security camera-based monitoring system that can work on remote subjects, as it uses images captured by a home security camera. In addition, a synopsis of sleep postures is provided for analysis and diagnosis of sleep patterns. Experimental results show that our multi-task model achieves an average of 92.5% accuracy on challenging datasets, yields the best performance compared to the other methods, and obtains 91.7% accuracy on the real-life overnight sleep data. The proposed system can be applied reliably to extensive public sleep data with various covering conditions and is robust to real-life overnight sleep data.

MmEat: Millimeter wave-enabled environment-invariant eating behavior monitoring

Dietary habits are closely related to people’s health condition. Unhealthy diet can cause obesity, diabetes, heart diseases, as well as increase the risk of cancers. It is necessary to have a monitoring system that helps people keep tracking his/her eating behaviors. Traditional sensor-based and camera-based dietary monitoring systems either require users to wear dedicated devices or may potentially incur privacy concerns. WiFi-based methods, though yielding reasonably robust performance in certain cases, have major limitations. The wireless signals usually carry substantial information that is specific to the environment where eating activities are performed. To overcome these limitations, we propose mmEat, a millimeter wave-enabled environment-invariant eating behavior monitoring system. In particular, we propose an environment impact mitigation method by analyzing mmWave signals in Dopper-Range domain. To differentiate dietary activities with various utensils (i.e., eating with fork, fork and knife, spoon, chopsticks, bare hand) for fine-grained eating behavior monitoring, we construct Spatial–Temporal Heatmap by integrating multiple dimensional measurements. We further utilize an unsupervised learning-based 2D segmentation method and an eating period derivation algorithm to estimate time duration of each eating activity. Our system has the potential to infer the food categories and eating speed. Extensive experiments with over 1000 eating activities show that our system can achieve dietary activity recognition with an average accuracy of 97.5% and a false detection rate of 5%.

IoT Networks Assisted Baduanjin Auxiliary Training System Based on Depth Camera

Baduanjin auxiliary training system has insufficient feature definition for similar actions, which affects the recognition accuracy of effective action data. A depth camera-based Baduanjin auxiliary training system is designed. The depth camera is used as the core to design the peripheral circuit and develop the relevant firmware in the hardware part. TPS2552 is used to design the USB current limiting protection circuit, and a CrossLink chip is used to realize the interface and level conversion. In the software part, the human body 3D model of Baduanjin training is generated and the parameters of the human body model are obtained by establishing the mapping relationship between the model and the human body features in the image. Based on the automatic calibration of Baduanjin motion feature points by the depth camera, the motion posture was analyzed and summarized, the attitude auxiliary indicators were established, and the posture standard was judged by comparing the distance of the posture joints in the reconstructed 3D human body model. The system test results show that the recognition accuracy of Baduanjin auxiliary training system based on the depth camera is 96.25% for effective action data, which is 11.53%, 15.28%, and 13.06% higher than that of Kinect, motion intervention, and behavior recognition auxiliary training system, respectively. Therefore, the system can provide analysis and guidance for Baduanjin movement.

The Way to Modern Shutter Speed Measurement Methods: A Historical Overview.

Exposure time is a fundamental parameter for the photographer when the photo is composed, and the exact length of the exposure may be an essential determinant of performance in certain camera-based applications, e.g., optical camera communication (OCC) systems. There can be several reasons to measure the shutter speed of a camera: shutter speed may be checked at the time of manufacturing; it may be necessary to recheck in case of an elder camera model; it may be necessary to be measured if its exact value is not provided by the manufacturer; or a precise measurement may be necessary for a demanding application. In this paper various methods for shutter speed measurement are reviewed, presenting and analyzing methods that are still relevant today either for manufacturers, service personnel, amateur photographers, or the developers of camera-based systems. Each presented method is illustrated by real measurement results and the performance properties of the methods are also presented.

Underwater 3D Measurements with Advanced Camera Modelling

A novel concept of camera modelling for underwater 3D measurements based on stereo camera utilisation is introduced. The geometrical description of the ray course subject to refraction in underwater cameras is presented under assumption of conditions, which are typically satisfied or can be achieved approximately. Possibilities of simplification are shown, which allow an approximation of the ray course by classical pinhole modelling. It is shown how the expected measurement errors can be estimated, as well as its influence on the expected 3D measurement result. Final processing of the 3D measurement data according to the requirements regarding accuracy is performed using several kinds of refinement. For example, calibration parameters can be refined, or systematic errors can be decreased by subsequent compensation by suitable error correction functions. Experimental data of simulations and real measurements obtained by two different underwater 3D scanners are presented and discussed. If inverse image magnification is larger than about one hundred, remaining errors caused by refraction effects can be usually neglected and the classical pinhole model can be used for stereo camera-based underwater 3D measurement systems.

Contactless Vital Signs Monitoring From Videos Recorded With Digital Cameras: An Overview.

The measurement of physiological parameters is fundamental to assess the health status of an individual. The contactless monitoring of vital signs may provide benefits in various fields of application, from healthcare and clinical setting to occupational and sports scenarios. Recent research has been focused on the potentiality of camera-based systems working in the visible range (380–750 nm) for estimating vital signs by capturing subtle color changes or motions caused by physiological activities but invisible to human eyes. These quantities are typically extracted from videos framing some exposed body areas (e.g., face, torso, and hands) with adequate post-processing algorithms. In this review, we provided an overview of the physiological and technical aspects behind the estimation of vital signs like respiratory rate, heart rate, blood oxygen saturation, and blood pressure from digital images as well as the potential fields of application of these technologies. Per each vital sign, we provided the rationale for the measurement, a classification of the different techniques implemented for post-processing the original videos, and the main results obtained during various applications or in validation studies. The available evidence supports the premise of digital cameras as an unobtrusive and easy-to-use technology for physiological signs monitoring. Further research is needed to promote the advancements of the technology, allowing its application in a wide range of population and everyday life, fostering a biometrical holistic of the human body (BHOHB) approach.

Radar-Based Robust People Tracking and Consumer Applications

Robust people tracking in an indoor or outdoor environment is an important feature for fitness applications, surveillance systems for security, and assisted living for the elderly. Camera-based systems offer the state of the art approach but raise privacy concerns and consume a significant amount of power for the sensor and the processing. On the other hand, modern millimeter-wave Radar sensors provide a low-cost and low-power alternative with very good resolution in range, radial velocity and angle of arrival. In this work, we use two popular algorithms, for tracking people using Radar, and compare them with our novel method. We show that our method is robust in certain corner cases and is suitable for scenarios with many users in close proximity. In addition, we present two consumer applications that are possible with a Radar system, multi-user macro gesture recognition and hand tracking. We believe that the combination of above methods will open new gateways for consumer applications.

A Setup for Camera-Based Detection of Simulated Pathological States Using a Neonatal Phantom.

Premature infants are among the most vulnerable patients in a hospital. Due to numerous complications associated with immaturity, a continuous monitoring of vital signs with a high sensitivity and accuracy is required. Today, wired sensors are attached to the patient’s skin. However, adhesive electrodes can be potentially harmful as they can damage the very thin immature skin. Although unobtrusive monitoring systems using cameras show the potential to replace cable-based techniques, advanced image processing algorithms are data-driven and, therefore, need much data to be trained. Due to the low availability of public neonatal image data, a patient phantom could help to implement algorithms for the robust extraction of vital signs from video recordings. In this work, a camera-based system is presented and validated using a neonatal phantom, which enabled a simulation of common neonatal pathologies such as hypo-/hyperthermia and brady-/tachycardia. The implemented algorithm was able to continuously measure and analyze the heart rate via photoplethysmography imaging with a mean absolute error of 0.91 bpm, as well as the distribution of a neonate’s skin temperature with a mean absolute error of less than 0.55 °C. For accurate measurements, a temperature gain offset correction on the registered image from two infrared thermography cameras was performed. A deep learning-based keypoint detector was applied for temperature mapping and guidance for the feature extraction. The presented setup successfully detected several levels of hypo- and hyperthermia, an increased central-peripheral temperature difference, tachycardia and bradycardia.

IoT-Based Fumigation for Insect Repellent in Food Storages: Breaking the Trade-Off between Efficiency and Safety

Insect infestation in food can cause various health risks when ingested by humans, as well as damage to food itself. To tackle this, food safety can be secured by fumigating the food storage, using specific materials containing pesticides. However, because most fumigation is toxic to human health, there is a trade-off relationship between insect repellency and safety assurance. In this paper, to overcome this problem, first, organic fumigation is proposed, in which a relatively low-risk pyrethrin oil is developed. Second, a novel system which can remotely monitor and control fumigation using IoT is proposed for mitigating the fact that pyrethrin can also be dangerous when inhaled directly. Third, an insect repellent LED lamp system, which can replace insecticide through direct fumigation and ensure safety, has been proposed. Fourth, a camera-based human access detection system is developed for more efficient and safe controls during the fumigation. The performance of the proposed system has been verified through implemented test-bed, and it is revealed that the trade-off relationship between efficiency and safety can be overcome.

Objective scoring of footpad dermatitis in broiler chickens using image segmentation and a deep learning approach: camera-based scoring system

ABSTRACT 1. Footpad dermatitis (FPD) can be used as an important indicator of animal welfare and for economic evaluation; however, human scoring is subjective, biased and labour intensive. This paper proposes a novel deep learning approach that can automatically determine the severity of FPD based on images of chicken’s feet. 2. This approach first determined the areas of the FPD lesion, normal parts of each foot and the background, using a deep segmentation model. The proportion of the FPD for the chicken’s two feet was calculated by dividing the number of FPD pixels by the number of feet pixels. The proportion was then categorised using a five-point score for FPD. The approach was evaluated from 244 images of the left and right footpads using five-fold cross-validation. These images were collected at a commercial slaughter plant and scored by trained observers. 3. The result showed that this approach achieved an overall accuracy and a macro F1-score of 0.82. The per-class F1-scores from all FPD scores (scores 0 to 4) were similar (0.85, 0.80, 0,80, 0,80, and 0.87, respectively), which demonstrated that this approach performed equally well for all classes of scores. 4. The results suggested that image segmentation and a deep learning approach can be used to automate the process of scoring FPD based on chicken foot images, which can help to minimise the subjective bias inherent in manual scoring.

Single Camera-Based Remote Physical Therapy: Verification on a Large Video Dataset

In recent years, several systems have been developed to capture human motion in real-time using common RGB cameras. This approach has great potential to become widespread among the general public as it allows the remote evaluation of exercise at no additional cost. The concept of using these systems in rehabilitation in the home environment has been discussed, but no work has addressed the practical problem of detecting basic body parts under different sensing conditions on a large scale. In this study, we evaluate the ability of the OpenPose pose estimation algorithm to perform keypoint detection of anatomical landmarks under different conditions. We infer the quality of detection based on the keypoint confidence values reported by the OpenPose. We used more than two thousand unique exercises for the evaluation. We focus on the influence of the camera view and the influence of the position of the trainees, which are essential in terms of the use for home exercise. Our results show that the position of the trainee has the greatest effect, in the following increasing order of suitability across all camera views: lying position, position on the knees, sitting position, and standing position. On the other hand, the effect of the camera view was only marginal, showing that the side view is having slightly worse results. The results might also indicate that the quality of detection of lower body joints is lower across all conditions than the quality of detection of upper body joints. In this practical overview, we present the possibilities and limitations of current camera-based systems in telerehabilitation.

Validation of wearable inertial sensor-based gait analysis system for measurement of spatiotemporal parameters and lower extremity joint kinematics in sagittal plane.

Wearable inertial sensor-based motion analysis systems are promising alternatives to standard camera-based motion capture systems for the measurement of gait parameters and joint kinematics. These wearable sensors, unlike camera-based gold standard systems, find usefulness in outdoor natural environment along with confined indoor laboratory-based environment due to miniature size and wireless data transmission. This study reports validation of our developed (i-Sens) wearable motion analysis system against standard motion capture system. Gait analysis was performed at self-selected speed on non-disabled volunteers in indoor (n = 15) and outdoor (n = 8) environments. Two i-Sens units were placed at the level of knee and hip along with passive markers (for indoor study only) for simultaneous 3D motion capture using a motion capture system. Mean absolute percentage error (MAPE) was computed for spatiotemporal parameters from the i-Sens system versus the motion capture system as a true reference. Mean and standard deviation of kinematic data for a gait cycle were plotted for both systems against normative data. Joint kinematics data were analyzed to compute the root mean squared error (RMSE) and Pearson's correlation coefficient. Kinematic plots indicate a high degree of accuracy of the i-Sens system with the reference system. Excellent positive correlation was observed between the two systems in terms of hip and knee joint angles (Indoor: hip 3.98° ± 1.03°, knee 6.48° ± 1.91°, Outdoor: hip 3.94° ± 0.78°, knee 5.82° ± 0.99°) with low RMSE. Reliability characteristics (defined using standard statistical thresholds of MAPE) of stride length, cadence, walking speed in both outdoor and indoor environment were well within the "Good" category. The i-Sens system has emerged as a potentially cost-effective, valid, accurate, and reliable alternative to expensive, standard motion capture systems for gait analysis. Further clinical trials using the i-Sens system are warranted on participants across different age groups.

Development of a Robust, Simple, and Affordable Human Gait Analysis System Using Bottom-Up Pose Estimation With a Smartphone Camera.

Gait analysis is used in many fields such as Medical Diagnostics, Osteopathic medicine, Comparative and Sports-related biomechanics, etc. The most commonly used system for capturing gait is the advanced video camera-based passive marker system such as VICON. However, such systems are expensive, and reflective markers on subjects can be intrusive and time-consuming. Moreover, the setup of markers for certain rehabilitation patients, such as people with stroke or spinal cord injuries, could be difficult. Recently, some markerless systems were introduced to overcome the challenges of marker-based systems. However, current markerless systems have low accuracy and pose other challenges in gait analysis with people in long clothing, hiding the gait kinematics. The present work attempts to make an affordable, easy-to-use, accurate gait analysis system while addressing all the mentioned issues. The system in this study uses images from a video taken with a smartphone camera (800 × 600 pixels at an average rate of 30 frames per second). The system uses OpenPose, a 2D real-time multi-person keypoint detection technique. The system learns to associate body parts with individuals in the image using Convolutional Neural Networks (CNNs). This bottom-up system achieves high accuracy and real-time performance, regardless of the number of people in the image. The proposed system is called the “OpenPose based Markerless Gait Analysis System” (OMGait). Ankle, knee, and hip flexion/extension angle values were measured using OMGait in 16 healthy volunteers under different lighting and clothing conditions. The measured kinematic values were compared with a standard video camera based normative dataset and data from a markerless MS Kinect system. The mean absolute error value of the joint angles from the proposed system was less than 90 for different lighting conditions and less than 110 for different clothing conditions compared to the normative dataset. The proposed system is adequate in measuring the kinematic values of the ankle, knee, and hip. It also performs better than the markerless systems like MS Kinect that fail to measure the kinematics of ankle, knee, and hip joints under dark and bright light conditions and in subjects with long robe clothing.

Drowsiness Detection using EEG signals and Machine Learning Algorithms

Drowsiness is described as a state of reduced consciousness and vigilance accompanied by a desire or want to sleep. Driver tiredness is frequently detected using wearable sensors that track vehicle movement and camera-based systems that track driver behavior. Many alternative EEG-based drowsiness detection systems are developed due to the potential of electroencephalogram (EEG) signals to observe human mood and the ease with which they may be obtained. This paper applies Deep learning architecture like Convolutional Neural networks (CNN) and algorithms for the classification of EEG data for Drowsiness Detection. The key measures of video-based approaches include the detection of physical features; nevertheless, problems such as brightness limitations and practical challenges such as driver attention limits its usefulness. The main measure of video-based methods is the degree of closure of the eyelids; however, its success is limited by constraints like as brightness restrictions and practical challenges such as driver distraction. We have extracted statistical features and trained using various classifiers like Logistic Regression, Naïve Bayes, SVM, and K Nearest Neighbours and compared the accuracy using a deep learning CNN model. Results demonstrate that CNN achieved an accuracy of 94.75% by delegating feature extraction on itself. Upon comparing existing state–of–the–art drowsiness detection systems, the testing results reveal a higher detection capability. The results show that the the suggested method can be used to develop a reliable EEG-based driving drowsiness detection system.

A vision-based human-robot collaborative system for digital twin

Flexible and safe human-robot collaboration depends on accurately capturing the three-dimensional motion of humans and robots in the field of smart manufacturing. In this paper, a novel approach to developing a human-robot collaborative assembly system is proposed and applied to the field of digital twins. Within the context, a deep learning-based model is explored to develop a depth camera-based human recognition system for accurate prediction of key points for human skeletons model and high-precision human localisation in a human-robot collaborative setting. After the functional mapping of robot calibration, a collision warning module leverages coordinates of key human-robot points to facilitate efficient and safe human-robot collaborative assembly.

Young drivers’ pedestrian anti-collision braking operation data modelling for ADAS development

Smart cities and smart mobility come from intelligent systems designed by humans. Artificial Intelligence (AI) is contributing significantly to the development of these systems, and the automotive industry is the most prominent example of "smart" technology entering the market: there are Advanced Driver Assistance System (ADAS), Radar/LIDAR detection units and camera-based Computer Vision systems that can assess driving conditions. Actually, these technologies have become consumer goods and services in mass-produced vehicles to provide human drivers with tools for a more comfortable and safer driving. Nevertheless, they need to be further improved for progress in the transition to fully automated driving or simply to increase vehicle automation levels. To this end, it becomes imperative to accurately predict driver’s decisions, model human driving behaviors, and introduce more accurate risk assessment metrics. This paper presents a system that can learn to predict the future braking behavior of a driver in a typically urban vehicle-pedestrian conflict, i.e., when a pedestrian enters a zebra crossing from the curb and a vehicle is approaching. The algorithm proposes a sequential prediction of relevant operational indicators that continuously describe the encounter process. A car driving simulator was used to collect reliable data on braking behaviours of a cohort of 68 licensed university students, who faced the same urban scenario. The vehicle speed, steering wheel angle, and pedal activity were recorded as the participants approached the crosswalk, along with the azimuth angle of the pedestrian and the relative longitudinal distance between the vehicle and the pedestrian: the proposed system employs the vehicle information as human driving decisions and the pedestrian information as explanatory variables of the environmental state. In fact, the pedestrian’s polar coordinates are usually calculated by an on-board millimeter-wave radar which is typically used to perceive the environment around a vehicle. All mentioned information is represented in the form of time series data and is used to train a recurrent neural network in a supervised machine learning process. The main purpose of this research is to define a system of behavioral profiles in non-collision conditions that could be used for enhancing the existing intelligent driving systems, e.g., to reduce the number of warnings when the driver is not on a collision course with a pedestrian. Preliminary experiments reveal the feasibility of the proposed system.

An Approach to Progress Monitoring of Industrial Manual Processes Based on Camera Recordings and Object Interactions

Despite digitalization and automation, manual processes in factory operations still play a significant role. With the increasing complexity of tasks, it is advisable to support the employee during execution. Technological progress regarding the analysis of image data facilitates the use of camera-based systems for assistance functions. A core element for these assistance systems is the recognition of state or actions to achieve context-sensitive support.However, existing solutions are often developed for specific problems and are difficult to transfer to other processes. This is especially the case for manual processes with more and different sub-process steps such as setup and maintenance operations. High development effort even in case of minor process changes is the result. Therefore, the objective of this work is to develop a generic approach for progress monitoring of manual processes.The proposed approach is based on the understanding that manual processes can be considered as the manipulation of objects to achieve a defined state of their characteristic properties. Consequently, manual processes can be described by their features, like the positions or movements of objects or their relative position to each other. To realize a configurable approach a two-step procedure is proposed. The basis is the flexible recognition and localization of objects in camera recordings by machine learning methods. Subsequently, the information obtained can be utilized by a configurable logic to verify defined states.To develop the detailed concept, firstly, a method for analysis of complex manual processes in factory operation is conducted. The aim is to determine and categorize the objects relevant for optical recognition and state verification. Secondly, different machine learning approaches for object recognition and object localization are compared to evaluate their capability in capturing the feature states. The possible outputs of the machine learning approaches are used to create control elements for progress verification. Examples are the position of objects in relation to each other or the speed of movement. In the next step, a procedure for the process-specific selection of the control elements is developed. Finally, the approach is applied and discussed. A setup process of a turning machine and a deburring process serve as representative use cases.

Design of PTZ Camera-Based Multiview Monitoring System for Efficient Observation in Vessel Engine Room

Design of PTZ Camera-Based Multiview Monitoring System for Efficient Observation in Vessel Engine Room