Camera-based System Research Articles

JoulesEye

Smartphones and smartwatches have contributed significantly to fitness monitoring by providing real-time statistics, thanks to accurate tracking of physiological indices such as heart rate. However, the estimation of calories burned during exercise is inaccurate and cannot be used for medical diagnosis. In this work, we present JoulesEye, a smartphone thermal camera-based system that can accurately estimate calorie burn by monitoring respiration rate. We evaluated JoulesEye on 54 participants who performed high intensity cycling and running. The mean absolute percentage error (MAPE) of JoulesEye was 5.8%, which is significantly better than the MAPE of 37.6% observed with commercial smartwatch-based methods that only use heart rate. Finally, we show that an ultra-low-resolution thermal camera that is small enough to fit inside a watch or other wearables is sufficient for accurate calorie burn estimation. These results suggest that JoulesEye is a promising new method for accurate and reliable calorie burn estimation.

Design and Implementation of a Camera-Based Tracking System for MAV Using Deep Learning Algorithms

In recent years, the advancement of micro-aerial vehicles has been rapid, leading to their widespread utilization across various domains due to their adaptability and efficiency. This research paper focuses on the development of a camera-based tracking system specifically designed for low-cost drones. The primary objective of this study is to build up a system capable of detecting objects and locating them on a map in real time. Detection and positioning are achieved solely through the utilization of the drone’s camera and sensors. To accomplish this goal, several deep learning algorithms are assessed and adopted because of their suitability with the system. Object detection is based upon a single-shot detector architecture chosen for maximum computation speed, and the tracking is based upon the combination of deep neural-network-based features combined with an efficient sorting strategy. Subsequently, the developed system is evaluated using diverse metrics to determine its performance for detection and tracking. To further validate the approach, the system is employed in the real world to show its possible deployment. For this, two distinct scenarios were chosen to adjust the algorithms and system setup: a search and rescue scenario with user interaction and precise geolocalization of missing objects, and a livestock control scenario, showing the capability of surveying individual members and keeping track of number and area. The results demonstrate that the system is capable of operating in real time, and the evaluation verifies that the implemented system enables precise and reliable determination of detected object positions. The ablation studies prove that object identification through small variations in phenotypes is feasible with our approach.

An Improved Method for Broiler Weight Estimation Integrating Multi-Feature with Gradient Boosting Decision Tree

Broiler weighing is essential in the broiler farming industry. Camera-based systems can economically weigh various broiler types without expensive platforms. However, existing computer vision methods for weight estimation are less mature, as they focus on young broilers. In effect, the estimation error increases with the age of the broiler. To tackle this, this paper presents a novel framework. First, it employs Mask R-CNN for instance segmentation of depth images captured by 3D cameras. Next, once the images of either a single broiler or multiple broilers are segmented, the extended artificial features and the learned features extracted by Customized Resnet50 (C-Resnet50) are fused by a feature fusion module. Finally, the fused features are adopted to estimate the body weight of each broiler employing gradient boosting decision tree (GBDT). By integrating diverse features with GBTD, the proposed framework can effectively obtain the broiler instance among many depth images of multiple broilers in the visual field despite the complex background. Experimental results show that this framework significantly boosts accuracy and robustness. With an MAE of 0.093 kg and an R2 of 0.707 in a test set of 240 63-day-old bantam chicken images, it outperforms other methods.

Energy-saving potential of fresh air management using camera-based indoor occupancy positioning system in public open space

The energy consumption of fresh air systems constitutes a significant proportion of the energy consumption of heating, ventilation, and air-conditioning system therefore, energy savings for fresh air handling units are essential. Typically, fresh air is supplied to a room, regardless of whether the space is occupied. Owing to the development of machine vision technology, camera-based indoor occupancy positioning systems that can provide the correct number and location of indoor occupancies have been developed. Thus, energy conservation for fresh air-handling units can be achieved by not serving vacant areas or by assuming a design load in the working zones. In this study, an occupancy position-oriented fresh air (OPFA) system is proposed to supply fresh air more directly to an occupied zone, thus conserving energy. To determine the location of indoor occupants, a stereo camera-based indoor occupancy positioning system is developed, and its accuracy is verified to be within 50 cm; furthermore, it can operate at only 30 W. To determine the annual energy consumption of the OPFA system, the positions of occupants in a classroom are monitored and recorded for 8 months. Subsequently, the typical occupancy patterns of the areas of interest are extracted via hierarchical clustering analysis. The results show that the total occupied time in the classroom is only 50.6% of the opening time. The most typical usage scenario is a self-study scenario with fewer than three persons. Therefore, typical operation strategies for a fresh air handling unit can be designed based on these typical occupancy scenarios, which can potentially conserve more than one-half of the supplied fresh air volume. The performance of the system is numerically investigated and compared with that of a conventional system. The result shows that the system reduces the total fresh air volume and energy consumption by 60.2% and 67.7%, respectively. This study offers a paradigm for efficiently reducing the energy usage of fresh air systems in multipurpose buildings based on occupancy positions to identify typical indoor occupancy patterns, thus allowing the operation strategies of ventilation systems to be optimized.

AdvRain: Adversarial Raindrops to Attack Camera-Based Smart Vision Systems

Vision-based perception modules are increasingly deployed in many applications, especially autonomous vehicles and intelligent robots. These modules are being used to acquire information about the surroundings and identify obstacles. Hence, accurate detection and classification are essential to reach appropriate decisions and take appropriate and safe actions at all times. Current studies have demonstrated that “printed adversarial attacks”, known as physical adversarial attacks, can successfully mislead perception models such as object detectors and image classifiers. However, most of these physical attacks are based on noticeable and eye-catching patterns for generated perturbations making them identifiable/detectable by the human eye, in-field tests, or in test drives. In this paper, we propose a camera-based inconspicuous adversarial attack (AdvRain) capable of fooling camera-based perception systems over all objects of the same class. Unlike mask-based FakeWeather attacks that require access to the underlying computing hardware or image memory, our attack is based on emulating the effects of a natural weather condition (i.e., Raindrops) that can be printed on a translucent sticker, which is externally placed over the lens of a camera whenever an adversary plans to trigger an attack. Note, such perturbations are still inconspicuous in real-world deployments and their presence goes unnoticed due to their association with a natural phenomenon. To accomplish this, we develop an iterative process based on performing a random search aiming to identify critical positions to make sure that the performed transformation is adversarial for a target classifier. Our transformation is based on blurring predefined parts of the captured image corresponding to the areas covered by the raindrop. We achieve a drop in average model accuracy of more than 45% and 40% on VGG19 for ImageNet dataset and Resnet34 for Caltech-101 dataset, respectively, using only 20 raindrops.

LiDAR-based estimation of bounding box coordinates using Gaussian process regression and particle swarm optimization

Camera-based object tracking systems in a given closed environment lack privacy and confidentiality. In this study, light detection and ranging (LiDAR) was applied to track objects similar to the camera tracking in a closed environment, guaranteeing privacy and confidentiality. The primary objective was to demonstrate the efficacy of the proposed technique through carefully designed experiments conducted using two scenarios. In Scenario I, the study illustrates the capability of the proposed technique to detect the locations of multiple objects positioned on a flat surface, achieved by analyzing LiDAR data collected from several locations within the closed environment. Scenario II demonstrates the effectiveness of the proposed technique in detecting multiple objects using LiDAR data obtained from a single, fixed location. Real-time experiments are conducted with human subjects navigating predefined paths. Three individuals move within an environment, while LiDAR, fixed at the center, dynamically tracks and identifies their locations at multiple instances. Results demonstrate that a single, strategically positioned LiDAR can adeptly detect objects in motion around it. Furthermore, this study provides a comparison of various regression techniques for predicting bounding box coordinates. Gaussian process regression (GPR), combined with particle swarm optimization (PSO) for prediction, achieves the lowest prediction mean square error of all the regression techniques examined at 0.01. Hyperparameter tuning of GPR using PSO significantly minimizes the regression error. Results of the experiment pave the way for its extension to various real-time applications such as crowd management in malls, surveillance systems, and various Internet of Things scenarios.

Evaluation of functional tests performance using a camera-based and machine learning approach.

The objective of this study is to evaluate the performance of functional tests using a camera-based system and machine learning techniques. Specifically, we investigate whether OpenPose and any standard camera can be used to assess the quality of the Single Leg Squat Test and Step Down Test functional tests. We recorded these exercises performed by forty-six healthy subjects, extract motion data, and classify them to expert assessments by three independent physiotherapists using 15 binary parameters. We calculated ranges of movement in Keypoint-pair orientations, joint angles, and relative distances of the monitored segments and used machine learning algorithms to predict the physiotherapists' assessments. Our results show that the AdaBoost classifier achieved a specificity of 0.8, a sensitivity of 0.68, and an accuracy of 0.7. Our findings suggest that a camera-based system combined with machine learning algorithms can be a simple and inexpensive tool to assess the performance quality of functional tests.

Development of a compact pulsed time-of-flight LiDAR platform for underwater measurements

The mapping and inspection of underwater topography and infrastructure require precise and robust instrumentation which cannot be completely fulfilled by sonar or camera-based systems. At Fraunhofer IPM we have therefore built a versatile platform that implements all necessary building blocks for various laser scanner configurations targeted at subsea applications. Based on this platform, we present two scanner prototypes: One compact and lightweight multi-wavelength airborne laser scanner for deployment with unmanned aerial vehicles. It is intended for easy and fast shallow water bathymetric and topographic mapping. The second scanner is embedded in a pressure resistant housing and designed for submerged deployment on stationary or moving platforms to facilitate numerous inspection tasks. Initial tests of the scanners have been performed in an excavation pond and on a dedicated underwater measurement range. The resulting point clouds indicate promising performance in terms of resolution, accuracy, and speed. Further work will focus on extensive field tests and software optimization for increased usability.

Driver Safety System

Abstract: Blind spots in vehicles and driver drowsiness are significant safety concerns that contribute to road accidents. To address these issues, we propose a comprehensive driver drowsiness and blind spot detection system using advanced technologies and image processing algorithms. The blind spot detection system employs ultrasonic sensors and an Arduino microcontroller board to gather real-time information about potential collision objects in the blind spot area. When a risk is detected, an LED alarm alerts the driver, enhancing their awareness and reducing the likelihood of accidents. Additionally, we explore the implementation of a camera-based blind spot detection system using deep learning techniques, offering a viable option for autonomous vehicles. For driver drowsiness detection, we present a non-intrusive method using a camera to continuously monitor the driver's facial features, such as eye and head movements. By analyzing these features, the system can accurately identify three driver states: awake, drowsy, and sleeping. When drowsiness is detected, the system activates alerts, such as visual and auditory cues, to awaken the driver and ensure their safety. The literature review greatly impacted our system's design, facilitating informed decisions and integration of cutting-edge technologies for an effective driver drowsiness and blind spot detection solution. The research paper also discusses the integration of these systems into various platforms, including mobile apps, Python programs, and IP cameras, providing flexible and cost-effective solutions for different vehicle types. Results from prototype implementations demonstrate the effectiveness and reliability of the proposed methods in detecting blind spots and drowsiness. Overall, this research enhances road safety via efficient blind spot & drowsiness detection, saving lives by addressing driver fatigue. Utilizing tech & machine learning, proposed systems advance active safety & driver assistance

Camera-Based Indoor Positioning System for the Creation of Digital Shadows of Plant Layouts

In the past, plant layouts were regarded as highly static structures. With increasing internal and external factors causing turbulence in operations, it has become more necessary for companies to adapt to new conditions in order to maintain optimal performance. One possible way for such an adaptation is the adjustment of the plant layout by rearranging the individual facilities within the plant. Since the information about the plant layout is considered as master data and changes have a considerable impact on interconnected processes in production, it is essential that this data remains accurate and up-to-date. This paper presents a novel approach to create a digital shadow of the plant layout, which allows the actual state of the physical layout to be continuously represented in virtual space. To capture the spatial positions and orientations of the individual facilities, a pan-tilt-zoom camera in combination with fiducial markers is used. With the help of a prototypically implemented system, the real plant layout was captured and converted into different data formats for further use in exemplary external software systems. This enabled the automatic updating of the plant layout for simulation, analysis and routing tasks in a case study and showed the benefits of using the proposed system for layout capturing in terms of accuracy and effort reduction.

Study on the influence of Al-Si welding wire on porosity sensitivity in laser welding and process optimization

Al-Si welding wire is commonly used in the aluminum alloy welding due to the excellent heat crack-resistance and corrosion-resistance, but the presence of large-sized pores hinders its widespread application in the field of laser welding. This study focuses on the cause of pores formation in the laser welding using Al-Si wires with different Si contents and investigates the dynamic behavior of laser keyhole by a high-speed camera-based molten pool monitoring system. The results showed that with the increase of wire Si content from 5% to 12% (wt.%), the keyhole stability decreased and the weld porosity increased. According to the thermodynamic calculations, the increased Si content enhanced the Reynolds number of molten pool, resulting in increased turbulence tendency. It caused the molten pool being more susceptible to external disturbances, which decreased the stability of keyhole. Furthermore, the reduced dynamic viscosity of molten pool increased both the depth-to-width ratio of keyhole and its opening and closing frequency. These combined effects ultimately contributed to the increased susceptibility to pores formation. Based on the above results, a circular oscillating laser was introduced to guide the regular flow of molten pool. The beam oscillation improved the resistance to fluctuations of molten pool and reduced the depth-to-width ratio of keyhole. Besides, the high-frequency rotating keyhole could capture and re-melt the formed pores. An optimal laser oscillating welding was determined using response surface methodology to control the porosity within the acceptable range.

Comparison and analysis of hyperspectral temperature data in directed energy deposition

Directed energy deposition is an additive manufacturing process that allows the production of near net shape structures. Moreover, the process can also be applied for the repair of high value components. To obtain structures with consistent good characteristics, the directed energy deposition process requires the implementation of a control system. The currently applied approaches for control that are discussed in the literature have specifically focused on melt-pool temperature control. Pyrometers have been used for such purposes; however, they provide only a single scalar value without any spatial information. In this paper, the implementation of a high-speed hyperspectral camera-based system is discussed with a high spatial resolution unlike the pyrometers. Different calibration and temperature estimation procedures for this camera-based system are evaluated and analyzed. The number of effective wavelengths needed for temperature estimation will be discussed in detail and provide an outlook on the potential of this hyperspectral camera-based system. In addition to the number of wavelengths, another important aspect of the temperature estimation methods is the stability with respect to disturbances. Within this paper, the impact of the nominal laser power will be evaluated on the stability of the temperature signals for a control system.

Key Considerations in Assessing the Safety and Performance of Camera-Based Mirror Systems

Camera-based mirror systems (CBMSs) are a relatively new technology in the automotive industry, and much of the United States’ medium- and heavy-duty commercial fleet has been reluctant to convert from standard glass, or “west coast”, mirrors to CBMSs. CBMSs have the potential to reduce the number of truck and passenger vehicle incidents, improving overall fleet safety. CBMSs also have the potential to improve operational efficiency by improving aerodynamics and reducing drag, resulting in better fuel economy, and improving maneuverability. Improvements in overall safety are also possible; the field of view for the driver is potentially 360° with the addition of trailer cameras, allowing for visibility of the rear of the trailer and the front of the truck. These potential improvements seem promising, but the literature on driver surveys clearly shows that there is reluctance to adopt this technology for many reasons. Additionally, more robust testing in the laboratory and in the field is necessary to determine whether CBMSs are adequate to replace standard mirrors on trucks. This analysis provides an overview of key research questions for CBMS testing based on the current literature on the topic (surveys, standards, and previous testing). The purpose of this analysis is to serve as guidance in developing further testing of CBMSs, especially testing involving human subjects.

Sub-micrometer real-time imaging of trajectory of alpha particles using GAGG plate and CMOS camera

High-resolution and real-time imaging of the trajectories of alpha particles is desired in nuclear medicine and nuclear engineering. Although an imaging method using a scintillator plate combined with a magnifying unit and a cooled electron multiplying charge-coupled device (EM-CCD) camera is a possible method of obtaining high-resolution trajectory images, the spatial resolution of the system is limited to ∼2 μm. To overcome the spatial resolution limitations of this method on trajectory imaging, we used a cooled complementally metal oxide (CMOS) camera in which the sensor had a much larger number of pixels, which were also smaller. Using the CMOS camera based imaging system, we could measure the trajectories of alpha particles in real time with the spatial resolution of 0.34 μm FWHM. With smoothing of the images to reduce image noise, spatial resolution was still kept to less than 0.75 μm. We conclude that this CMOS camera-based alpha-particle trajectory-imaging system is promising for alpha-particle or other particles imaging where ultrahigh spatial resolution is required.

퍼스널 모빌리티 산업 활성화 방안 연구

The field of personal mobility is experiencing rapid growth worldwide, driven by the proliferation of shared services and its appeal as an environmentally friendly and convenient mode of transportation. However, the increasing number of safety incidents involving users and pedestrians, attributed to factors such as user inexperience and inadequacies in regulatory frameworks, raises concerns. In this study, we employ text mining techniques to analyze the public perception of safety in personal mobility and propose policy alternatives to promote the development of the personal mobility industry. Specifically, our recommendations include: 1) Advancing technological sophistication through the development of camera-based Vision Positioning System (VPS) technology, augmented reality (AR) technology, and support for helmet materials and innovation research; 2) Improving business models by introducing distance-based charging mechanisms; 3) Strengthening infrastructure through enhanced battery safety measures, expansion and enhancement of bicycle lanes and roads; and 4) Reinforcing safety verification tests by developing and supporting a safety speed monitoring system and introducing specialized licenses for personal mobility operation.

Design and validation of depth camera-based static posture assessment system

Postural abnormalities have become a prevalent issue affecting individuals of all ages, resulting in a diminished quality of life. Easy-use and reliable posture assessment tools can aid in screening for and correcting posture deviation at an early stage. In this study, we present a depth camera-based static posture assessment system to screen for common postural anomalies such as uneven shoulders, pelvic tilt, bowlegs and knock-knees, forward head, scoliosis, and shoulder blade inclination. The system consists of an Azure Kinect camera, a laptop, and evaluation software. Our system accurately measures skeleton and posture indexes and shows favorable agreement with a golden standard optical infrared motion capture system. The findings indicate that the system is a low-cost posture assessment tool with high precision and accuracy, suitable for initial screening of postural abnormalities in individuals of all ages.

Exploratory Development of Algorithms for Determining Driver Attention Status.

Varying driver distraction algorithms were developed using vehicle kinematics and driver gaze data obtained from a camera-based driver monitoring system (DMS). Distracted driving characteristics can be difficult to accurately detect due to wide variation in driver behavior across driving environments. The growing availability of information about drivers and their involvement in the driving task increases the opportunity for accurately recognizing attention state. A baseline for driver distraction levels was developed using a video feed of 24 separate drivers in varying naturalistic driving conditions. This initial assessment was used to develop four buffer-based algorithms that aimed to determine a driver's real-time attentiveness, via a variety of metrics and combinations thereof. Of those tested, the optimal algorithm included ungrouped glance locations and speed. Notably, as an algorithm's performance of detecting very distracted drivers improved, its accuracy for correctly identifying attentive drivers decreased. At a minimum, drivers' gaze position and vehicle speed should be included when designing driver distraction algorithms to delineate between glance patterns observed at high and low speeds. Distraction algorithms should be designed with an understanding of their limitations, including instances in which they may fail to detect distracted drivers, or falsely notify attentive drivers. This research adds to the body of knowledge related to driver distraction and contributes to available methods to potentially address and reduce occurrences. Machine learning algorithms can build on the data elements discussed to increase distraction detection accuracy using robust artificial intelligence.

SpeckleCam: high-resolution computational speckle contrast tomography for deep blood flow imaging.

Laser speckle contrast imaging is widely used in clinical studies to monitor blood flow distribution. Speckle contrast tomography, similar to diffuse optical tomography, extends speckle contrast imaging to provide deep tissue blood flow information. However, the current speckle contrast tomography techniques suffer from poor spatial resolution and involve both computation and memory intensive reconstruction algorithms. In this work, we present SpeckleCam, a camera-based system to reconstruct high resolution 3D blood flow distribution deep inside the skin. Our approach replaces the traditional forward model using diffuse approximations with Monte-Carlo simulations-based convolutional forward model, which enables us to develop an improved deep tissue blood flow reconstruction algorithm. We show that our proposed approach can recover complex structures up to 6 mm deep inside a tissue-like scattering medium in the reflection geometry. We also conduct human experiments to demonstrate that our approach can detect reduced flow in major blood vessels during vascular occlusion.

Digital Motor Biomarkers of Cerebellar Ataxia Using an RGB-Depth Camera-Based Motion Analysis System.

This study aimed to identify quantitative biomarkers of motor function for cerebellar ataxia by evaluating gait and postural control using an RGB-depth camera-based motion analysis system. In 28 patients with degenerative cerebellar ataxia and 33 age- and sex-matched healthy controls, motor tasks (short-distance walk, closed feet stance, and stepping in place) were selected from a previously reported protocol, and scanned using Kinect V2 and customized software. The Clinical Assessment Scale for the Assessment and Rating of Ataxia (SARA) was also evaluated. Compared with the normal control group, the cerebellar ataxia group had slower gait speed and shorter step lengths, increased step width, and mediolateral trunk sway in the walk test (all P < 0.001). Lateral sway increased in the stance test in the ataxia group (P < 0.001). When stepping in place, the ataxia group showed higher arrhythmicity of stepping and increased stance time (P < 0.001). In the correlation analyses, the ataxia group showed a positive correlation between the total SARA score and arrhythmicity of stepping in place (r = 0.587, P = 0.001). SARA total score (r = 0.561, P = 0.002) and gait subscore (ρ = 0.556, P = 0.002) correlated with mediolateral truncal sway during walking. These results suggest that the RGB-depth camera-based motion analyses on mediolateral truncal sway during walking and arrhythmicity of stepping in place are useful digital motor biomarkers for the assessment of cerebellar ataxia, and could be utilized in future clinical trials.

One camera-based laser keyhole welding monitoring system using deep learning

The laser-beam absorptance changes dynamically during laser keyhole welding due to unstable keyhole movements, and monitoring the absorptance can provide a deep understanding of the process. Recently, Kim et al. [1,2] developed a deep-learning-based method to monitor the absorptance by detecting the top and bottom keyhole apertures and estimating the absorptance from the reconstructed keyhole shape based on the detected apertures. However, this method was limited in that it required simultaneously observing the top and bottom keyhole apertures using two cameras. In this study, we proposed a novel deep-learning-based method to monitor the laser-beam absorptance in a keyhole using only one camera during laser keyhole welding of Al 5052-H32 alloy. In this method, both the top and bottom keyhole apertures were simultaneously detected from the images coaxially obtained from the top side. Although part of the bottom apertures may be sometimes obscured when viewed from above, this study demonstrated that the predicted absorptance was accurate enough and sufficient for monitoring laser welding processes of aluminum alloys. Using the developed method, changes in welding mode and generation of welding defects were successfully detected.