Built-in Camera Research Articles

Adapting smartphone-based photoplethysmograpy to suboptimal scenarios

Photoplethysmography (PPG) is an optical technique used to measure the heart rate (HR) and other cardiovascular variables by analyzing volume changes in the microvascular bed of tissue. At the moment, smartphone users can already measure their HR using PPG applications that use the smartphone’s built-in camera. However, available applications are unreliable when artifacts are present, such as those caused by movement, finger pressure, or ambient light changes. This contribution aims to analyze the limitations of a smartphone-based PPG algorithm capable of measuring N–N intervals when such artifacts are present by comparing it to a 2-lead electrocardiography (ECG). By using a Bandpass filter and a zero-crossing detection algorithm on a PPG signal captured at 800 × 600 pixels and 30 Hz, we have designed an approach capable of assessing N–N intervals when movement artifacts are present. An evaluation performed on n = 31 users shows our algorithm is capable of measuring N–N intervals with an average relative error of 9.23 ms, when compared to a 2-lead ECG. Our approach proves the reliability of smartphone-based photoplethysmography to measure N–N intervals, even under the presence of movement artifacts, and opens the door for its future use in remote diagnosis scenarios.

Staring at the Screen: Interview with Photographer Robbie Cooper

From the beginnings of the photography, portrait photography has had a special aura – reading one’s own facial expressions and those of others is after all a very human trait. In his project Immersion, British artist Robbie Cooper presents a specific type of portraits – portraits of people as media consumers. We are all aware of the frightening statistics of the average number of hours spent behind the screen, yet Cooper’s intention was not to moralise. A diverse spectrum of people’s expressions captured during watching various media content tells only one part of our human story. In the Immersion, the screen becomes some kind of mirror, recording intense expressions of the portrayed persons, captured with an in-built camera. Because of the accompanying sound, we can guess what the people are watching – the content includes everything, from video games, pornography to snuff movies. Stills from the movies have less documentary value. With the help of the high quality of the photos, the frozen grimaces become peculiarly similar to the classical portraits from the history of art. Almost eccentric grimaces confuse us and at the same time remind us how realistic virtual reality feels. Cooper had already explored our relationship towards virtual reality in his project Alter Ego, in which he sets the gamers of virtual games next to their avatars. He was interested in the human element of virtual worlds by questioning what imaginary personas can tell us about their creators. Throughout our conversation, questions of human consciousness arose. Keywords: expressive face, grimace, human character and facial expression, media content, Robbie Cooper, role-playing

The Emergence of Visual Crowdsensing: Challenges and Opportunities

Visual crowdsensing (VCS), which leverages built-in cameras of smart devices to attain informative and comprehensive sensing of interesting targets, has become a predominant sensing paradigm of mobile crowdsensing (MCS). Compared to MCS tasks using other sensing modalities, VCS faces numerous unique issues, such as multi-dimensional coverage needs, data redundancy identification and elimination, low-cost transmission, as well as high data processing cost. This paper characterizes the concepts, unique features, and novel application areas of VCS, and investigates its challenges and key techniques. A generic framework for VCS systems is then presented, followed by discussions about the future directions of crowdsourced picture transmission and the experimental setup in VCS system evaluation.

A Generic Framework for Constraint-Driven Data Selection in Mobile Crowd Photographing

Mobile crowd photographing (MCP) is an emerging area of interest for researchers as the built-in cameras of mobile devices are becoming one of the commonly used visual logging approaches in our daily lives. In order to meet diverse MCP application requirements and constraints of sensing targets, a multifacet task model should be defined for a generic MCP data collection framework. Furthermore, MCP collects pictures in a distributed way in which a large number of contributors upload pictures whenever and wherever it is suitable. This inevitably leads to evolving picture streams. This paper investigates the multiconstraint-driven data selection problem in MCP picture aggregation and proposes a pyramid-tree (PTree) model which can efficiently select an optimal subset from the evolving picture streams based on varied coverage needs of MCP tasks. By utilizing the PTree model in a generic MCP data collection framework, which is called CrowdPic, we test and evaluate the effectiveness, efficiency, and flexibility of the proposed framework through crowdsourcing-based and simulation-based experiments. Both the theoretical analysis and simulation results indicate that the PTree-based framework can effectively select a subset with high utility coverage and low redundancy ratio from the streaming data. The overall framework is also proved flexible and applicable to a wide range of MCP task scenarios.

Accuracy of Heart Rate Measurement Using Smartphones During Treadmill Exercise in Male Patients With Ischemic Heart Disease.

ObjectiveTo evaluate the accuracy of a smartphone application measuring heart rates (HRs), during an exercise and discussed clinical potential of the smartphone application for cardiac rehabilitation exercise programs.MethodsPatients with heart disease (14 with myocardial infarction, 2 with angina pectoris) were recruited. Exercise protocol was comprised of a resting stage, Bruce stage II, Bruce stage III, and a recovery stage. To measure HR, subjects held smartphone in their hands and put the tip of their index finger on the built-in camera for 1 minute at each exercise stage such as resting stage, Bruce stage II, Bruce stage III, and recovery stage. The smartphones recorded photoplethysmography signal and HR was calculated every heart beat. HR data obtained from the smartphone during the exercise protocol was compared with the HR data obtained from a Holter electrocardiography monitor (control).ResultsIn each exercise protocol stage (resting stage, Bruce stage II, Bruce stage III, and the recovery stage), the HR averages obtained from a Holter monitor were 76.40±12.73, 113.09±14.52, 115.64±15.15, and 81.53±13.08 bpm, respectively. The simultaneously measured HR averages obtained from a smartphone were 76.41±12.82, 112.38±15.06, 115.83±15.36, and 81.53±13 bpm, respectively. The intraclass correlation coefficient (95% confidence interval) was 1.00 (1.00–1.00), 0.99 (0.98–0.99), 0.94 (0.83–0.98), and 1.00 (0.99–1.00) in resting stage, Bruce stage II, Bruce stage III, and recovery stage, respectively. There was no statistically significant difference between the HRs measured by either device at each stage (p>0.05).ConclusionThe accuracy of measured HR from a smartphone was almost overlapped with the measurement from the Holter monitor in resting stage and recovery stage. However, we observed that the measurement error increased as the exercise intensity increased.

A New Approach Based on Image Processing for Detection of Wear of Guide-Rail Surface in Elevator Systems

Elevators ensure transportation of people inside buildings and increase their life quality. High-rise buildings whose number is increasingly going up today has one or more elevator cabs to provide vertical transportation. A great number of people use elevators in many buildings such as business centres, hotels, hospitals and shopping centres daily. It is highly essential for the elevators used by many people daily to operate constantly. In the event of sudden failure of elevators during operation, people inside them face with a tough situation. Also, people have difficulty during the maintenance-repair period of elevators. Elevator system has counterweight system in order to balance the weight of elevator cab. A guide-rail system has been developed to limit the movements of elevator cab and counterweights on horizontal axis. When an elevator system is operational, cab and counterweight system move reversely. The common failures in elevators are usually seen in the components such as elevator guide-rail system, ropes and motors. In this study, a system based on image processing has been developed in order to prevent wear on guide-rail surface in elevators. In the proposed method, real-time condition monitoring is performed by cameras using built-in system. The images of elevator guide-rail surface are captured via four digital cameras fixed onto elevator cab. The image-processing methods are applied on the images captured by cameras and hence the wears on the surface of guide-rails are detected. The surface of guide-rail is firstly detected in the proposed method. Then, image segmentation and mathematical morphology are applied on the image of guide-rail surface and the wears on the surface of rail are detected. The failure extent of the wear failures detected are calculated. By processing the images captured by four cameras during movement of elevator, the results for surface of guide-rails are obtained. Using these results, reporting is performed. An elevator prototype has been created in order to carry out tests for development of the proposed method. The tests have been conducted by fixing the built-in system and cameras onto this elevator prototype. It is considerably advantageous to detect the failures on elevator guide-rails through image-processing methods. Following a literature review, it is seen that the proposed method is a new approach.

Calibration Method for IATS and Application in Multi-Target Monitoring Using Coded Targets

Abstract The technique of Image Assisted Total Stations (IATS) has been studied for over ten years and is composed of two major parts: one is the calibration procedure which combines the relationship between the camera system and the theodolite system; the other is the automatic target detection on the image by various methods of photogrammetry or computer vision. Several calibration methods have been developed, mostly using prototypes with an add-on camera rigidly mounted on the total station. However, these prototypes are not commercially available. This paper proposes a calibration method based on Leica MS50 which has two built-in cameras each with a resolution of 2560 × 1920 px: an overview camera and a telescope (on-axis) camera. Our work in this paper is based on the on-axis camera which uses the 30-times magnification of the telescope. The calibration consists of 7 parameters to estimate. We use coded targets, which are common tools in photogrammetry for orientation, to detect different targets in IATS images instead of prisms and traditional ATR functions. We test and verify the efficiency and stability of this monitoring method with multi-target.

Poe in Cyberspace: The “Mobile-Friendly” Sites

Poe in Cyberspace: The “Mobile-Friendly” Sites

A highly efficient and reliable heart rate monitoring system using smartphone cameras

In the field of healthcare, a fundamental research topic includes the development of a vital signs monitoring system that allows people to monitor their health state so that they can control the same. Given that the heart is at the center of the human system, it is important to constantly monitor the heart beat rate. Hence, recently, heart rate monitoring systems or methods have started to use smartphones (and their built-in cameras) as computing and sensing platforms. These are ubiquitous devices. However, as most smartphones still have low computational power and as camera features differ across smartphones, smartphone-based heart rate computations can suffer from time-consuming processes. Hence, the results can be inaccurate or inconsistent. In this paper, we propose a highly efficient and reliable method to measure the heart rate from the smartphone camera images of fingertips. The method is composed of region of interest-based signal extraction, signal noise/bias reduction using an adaptive threshold scheme, and cycle miss/duplication handling using an iterative outlier elimination scheme. Existing methods require the real-time operation of high-speed processors and work properly on only specific smartphones. In contrast, the proposed method works consistently in real time with high accuracy on smartphones of any level. This is a very important factor because health-care facilities must be universally accessible, including to individuals who cannot afford buying excessively expensive high-performance smartphones.

Rainfall simulation and Structure-from-Motion photogrammetry for the analysis of soil water erosion in Mediterranean vineyards

Soil water erosion is a serious problem, especially in agricultural lands. Among these, vineyards deserve attention, because they constitute for the Mediterranean areas a type of land use affected by high soil losses. A significant problem related to the study of soil water erosion in these areas consists in the lack of a standardized procedure of collecting data and reporting results, mainly due to a variability among the measurement methods applied. Given this issue and the seriousness of soil water erosion in Mediterranean vineyards, this works aims to quantify the soil losses caused by simulated rainstorms, and compare them with each other depending on two different methodologies: (i) rainfall simulation and (ii) surface elevation change-based, relying on high-resolution Digital Elevation Models (DEMs) derived from a photogrammetric technique (Structure-from-Motion or SfM). The experiments were carried out in a typical Mediterranean vineyard, located in eastern Spain, at very fine scales. SfM data were obtained from one reflex camera and a smartphone built-in camera. An index of sediment connectivity was also applied to evaluate the potential effect of connectivity within the plots. DEMs derived from the smartphone and the reflex camera were comparable with each other in terms of accuracy and capability of estimating soil loss. Furthermore, soil loss estimated with the surface elevation change-based method resulted to be of the same order of magnitude of that one obtained with rainfall simulation, as long as the sediment connectivity within the plot was considered. High-resolution topography derived from SfM revealed to be essential in the sediment connectivity analysis and, therefore, in the estimation of eroded materials, when comparing them to those derived from the rainfall simulation methodology. The fact that smartphones built-in cameras could produce as much satisfying results as those derived from reflex cameras is a high value added for using SfM.

On the precision of third person perspective augmented reality for target designation tasks

The availability of powerful consumer-level smart devices and off-the-shelf software frameworks has tremendously popularized augmented reality (AR) applications. However, since the built-in cameras typically have rather limited field of view, it is usually preferable to position AR tools built upon these devices at a distance when large objects need to be tracked for augmentation. This arrangement makes it difficult or even impossible to physically interact with the augmented object. One solution is to adopt third person perspective (TPP) with which the smart device shows in real time the object to be interacted with, the AR information and the user herself, all captured by a remote camera. Through mental transformation between the user-centric coordinate space and the coordinate system of the remote camera, the user can directly interact with objects in the real world. To evaluate user performance under this cognitively demanding situation, we developed such an experimental TPP AR system and conducted experiments which required subjects to make markings on a whiteboard according to virtual marks displayed by the AR system. The same markings were also made manually with a ruler. We measured the precision of the markings as well as the time to accomplish the task. Our results show that although the AR approach was on average around half a centimeter less precise than the manual measurement, it was approximately three times as fast as the manual counterpart. Additionally, we also found that subjects could quickly adapt to the mental transformation between the two coordinate systems.

Smartphone-Based Cardiac Rehabilitation Program: Feasibility Study.

We introduce a cardiac rehabilitation program (CRP) that utilizes only a smartphone, with no external devices. As an efficient guide for cardiac rehabilitation exercise, we developed an application to automatically indicate the exercise intensity by comparing the estimated heart rate (HR) with the target heart rate zone (THZ). The HR is estimated using video images of a fingertip taken by the smartphone’s built-in camera. The introduced CRP app includes pre-exercise, exercise with intensity guidance, and post-exercise. In the pre-exercise period, information such as THZ, exercise type, exercise stage order, and duration of each stage are set up. In the exercise with intensity guidance, the app estimates HR from the pulse obtained using the smartphone’s built-in camera and compares the estimated HR with the THZ. Based on this comparison, the app adjusts the exercise intensity to shift the patient’s HR to the THZ during exercise. In the post-exercise period, the app manages the ratio of the estimated HR to the THZ and provides a questionnaire on factors such as chest pain, shortness of breath, and leg pain during exercise, as objective and subjective evaluation indicators. As a key issue, HR estimation upon signal corruption due to motion artifacts is also considered. Through the smartphone-based CRP, we estimated the HR accuracy as mean absolute error and root mean squared error of 6.16 and 4.30bpm, respectively, with signal corruption due to motion artifacts being detected by combining the turning point ratio and kurtosis.

Chest compression rate measurement from smartphone video.

BackgroundOut-of-hospital cardiac arrest is a life threatening situation where the first person performing cardiopulmonary resuscitation (CPR) most often is a bystander without medical training. Some existing smartphone apps can call the emergency number and provide for example global positioning system (GPS) location like Hjelp 113-GPS App by the Norwegian air ambulance. We propose to extend functionality of such apps by using the built in camera in a smartphone to capture video of the CPR performed, primarily to estimate the duration and rate of the chest compression executed, if any.MethodsAll calculations are done in real time, and both the caller and the dispatcher will receive the compression rate feedback when detected. The proposed algorithm is based on finding a dynamic region of interest in the video frames, and thereafter evaluating the power spectral density by computing the fast fourier transform over sliding windows. The power of the dominating frequencies is compared to the power of the frequency area of interest. The system is tested on different persons, male and female, in different scenarios addressing target compression rates, background disturbances, compression with mouth-to-mouth ventilation, various background illuminations and phone placements. All tests were done on a recording Laerdal manikin, providing true compression rates for comparison.ResultsOverall, the algorithm is seen to be promising, and it manages a number of disturbances and light situations. For target rates at 110 cpm, as recommended during CPR, the mean error in compression rate (Standard dev. over tests in parentheses) is 3.6 (0.8) for short hair bystanders, and 8.7 (6.0) including medium and long haired bystanders.ConclusionsThe presented method shows that it is feasible to detect the compression rate of chest compressions performed by a bystander by placing the smartphone close to the patient, and using the built-in camera combined with a video processing algorithm performed real-time on the device.

A wearable device for emotional recognition using facial expression and physiological response.

This paper introduces a glasses-typed wearable system to detect user's emotions using facial expression and physiological responses. The system is designed to acquire facial expression through a built-in camera and physiological responses such as photoplethysmogram (PPG) and electrodermal activity (EDA) in unobtrusive way. We used video clips for induced emotions to test the system suitability in the experiment. The results showed a few meaningful properties that associate emotions with facial expressions and physiological responses captured by the developed wearable device. We expect that this wearable system with a built-in camera and physiological sensors may be a good solution to monitor user's emotional state in daily life.

Biometrics for electronic eyes: system authentication with embedded CMOS image sensor

Identically designed electronic devices such as transistors may have random variations in their physical quantities. Inspired by human biometrics, this paper presents a novel approach that enables home appliances to have unique identities based on the intrinsic features of a CMOS image sensor (CIS) embedded in their electronic eyes (i.e., inbuilt cameras). More specifically, the column component of the fixed pattern noise (FPN) that inevitably exists in most modern CISs is utilized as a unique feature of the CIS for describing a unique ID for the embedded system. To practically capture column FPNs that were originally canceled out by a correlated double sampling (CDS) process, this paper also proposes a novel readout scheme. The feasibility of the proposed CIS-based identification is assessed with real measurements using 12 CIS prototypes. In conclusion, a set of applications requiring unique device identifications are presented for demonstrating the utility of the proposed approach in the domain of consumer electronics. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>

Energy-Efficient Aquatic Environment Monitoring Using Smartphone-Based Robots

Monitoring aquatic environment is of great interest to the ecosystem, marine life, and human health. This article presents the design and implementation of Samba—an aquatic surveillance robot that integrates an off-the-shelf Android smartphone and a robotic fish to monitor harmful aquatic processes such as oil spills and harmful algal blooms. Using the built-in camera of the smartphone, Samba can detect spatially dispersed aquatic processes in dynamic and complex environment. To reduce the excessive false alarms caused by the nonwater area (e.g., trees on the shore), Samba segments the captured images and performs target detection in the identified water area only. However, a major challenge in the design of Samba is the high energy consumption resulted from continuous image segmentation. We propose a novel approach that leverages the power-efficient inertial sensors on smartphones to assist image processing. In particular, based on the learned mapping models between inertial and visual features, Samba uses real-time inertial sensor readings to estimate the visual features that guide image segmentation, significantly reducing the energy consumption and computation overhead. Samba also features a set of lightweight and robust computer vision algorithms, which detect harmful aquatic processes based on their distinctive color features. Last, Samba employs a feedback-based rotation control algorithm to adapt to spatiotemporal development of the target aquatic process. We have implemented a Samba prototype and evaluated it through extensive field experiments, lab experiments, and trace-driven simulations. The results show that Samba can achieve a 94% detection rate, a 5% false alarm rate, and a lifetime up to nearly 2 months.

Automation Highlights from the Literature

Automation Highlights from the Literature

Augmented Reality in Informal Learning Environments

Augmented Reality (AR) allows computer-generated imagery information to be overlaid onto a live real world environment in real-time. Technological advances in mobile computing devices (MCD) such as smartphones and tablets (internet access, built-in cameras and GPS) made a greater number of AR applications available. This paper presents the Augmented Reality Musical Gallery (ARMG) exhibition, enhanced by AR. ARMG focuses the twentieth century music history and it is aimed to students from the 2nd Cycle of basic education in Portuguese public schools. In this paper, we will introduce the AR technology and address topics as constructivism, art education, student motivation, and informal learning environments. We conclude by presenting the first part of the ongoing research conducted among a sample group of students contemplating the experiment in educational context.

PEKEMON: A Mobile Application for Detecting Fake Money

Banko Sentral ng Pilipinas (BSP) released and redesigned/new Philippine peso bills last December 2010 in accordance to stop the proliferation of counterfeit money, but circulation of fake money continues. The project developed a mobile application that will help detect fake new Philippine bills by using an algorithm in a mobile application. The researchers devised a tool in detecting fake money by distinguishing between genuine and counterfeit Philippine peso bills using built-in phone camera. The method used in detecting fake money was image segmentation using Sobel Edge Detection Algorithm. It converted the image to grayscale and detected two kinds of edges in an image: Vertical Direction Edges and Horizontal Direction Edges. After the image was converted, it counts the number of white pixels which had been the basis to determine if it is fake or not. The mobile application is available for android operating system only. The programming language that the researchers used is Java. The application resulted to 50% - 70% accuracy in detecting fake bills because it uses a phone camera for detection instead of Ultra Violet Light. It is recommended that the design be enhanced by adding more functionality such as sending a message to the office of Banko Sentral ng Pilipinas (BSP) if the money is detected as fake.

UNMANNED AERIAL VEHICLE USE FOR WOOD CHIPS PILE VOLUME ESTIMATION

Abstract. The rapid development of unmanned aerial vehicles is a challenge for applied research. Many technologies are developed and then researcher are looking up for their application in different sectors. Therefore, we decided to verify the use of the unmanned aerial vehicle for wood chips pile monitoring. We compared the use of GNSS device and unmanned aerial vehicle for volume estimation of four wood chips piles. We used DJI Phantom 3 Professional with the built-in camera and GNSS device (geoexplorer 6000). We used Agisoft photoscan for processing photos and ArcGIS for processing points. Volumes calculated from pictures were not statistically significantly different from amounts calculated from GNSS data and high correlation between them was found (p = 0.9993). We conclude that the use of unmanned aerial vehicle instead of the GNSS device does not lead to significantly different results. Tthe data collection consumed from almost 12 to 20 times less time with the use of UAV. Additionally, UAV provides documentation trough orthomosaic.