Regular Camera Research Articles

Snapshot hyperspectral imaging using wide dilation networks

Hyperspectral (HS) cameras record the spectrum at multiple wavelengths for each pixel in an image, and are used, e.g., for quality control and agricultural remote sensing. We introduce a fast, cost-efficient and mobile method of taking HS images using a regular digital camera equipped with a passive diffraction grating filter, using machine learning for constructing the HS image. The grating distorts the image by effectively mapping the spectral information into spatial dislocations, which we convert into a HS image by a convolutional neural network utilizing novel wide dilation convolutions that accurately model optical properties of diffraction. We demonstrate high-quality HS reconstruction using a model trained on only 271 pairs of diffraction grating and ground truth HS images.

Food Depth Estimation Using Low-Cost Mobile-Based System for Real-Time Dietary Assessment

Real time estimation of nutrition intake from regular food items using mobile-based applications could be a breakthrough in creating public awareness of threats in overeating or faulty food choices. The bottleneck in implementing such systems is to effectively estimate the depths of the food items which is essential to calculate the volumes of foods. Volumes and density of food items can be used to estimate the weights of food eaten and their corresponding nutrition contents. Without specific depth sensors, it is very difficult to estimate the depth of any object from a single camera. Such sensors are equipped only in very advanced and expensive mobile devices. This work investigates the possibilities of using regular cameras to calculate the same using a specific frame structure. We proposed a controlled camera setup to acquire overlapping images of the food from different positions already calibrated to estimate the depths. The results were compared with the Kinect device’s depth measures to show the efficiency of the proposed method. We further investigated the optimum number of camera positions, their corresponding angles, and distances from the object to propose the best configuration for such a controlled system of image acquisition with regular mobile cameras. Overall the proposed method presents a low-cost solution to the depth estimation problem and opens up the possibilities for mobile-based apps for dietary assessment for various health-related problem-solving.
 GUB JOURNAL OF SCIENCE AND ENGINEERING, Vol 6(1), Dec 2019 P 1-11

Remote sensing to control respiratory viral diseases outbreaks using Internet of Vehicles

AbstractThe respiratory viral diseases, such as those caused by the family of coronaviruses, can be extremely contagious and spread through saliva droplets generated by coughing, sneezing, or breathing. In humans, the most common symptoms of the infection include fever and difficulty in breathing. In order to reduce the diffusion of the current “Coronavirus disease 2019 (COVID‐19)” pandemic, the Internet of Things technologies can play an important role; for instance, they can be effectively used for implementing a real‐time patient tracking and warning system at a city scale. Crucial places to install the tracking IoT devices are the public/private vehicles that, augmented with multiple connectivity solutions, can implement the Internet of Vehicles paradigm. In such ubiquitous network environment, vehicles are equipped with a variety of sensors, including regular cameras that can be replaced with thermal cameras. Therefore, this article proposes a new design for widely detecting respiratory viral diseases that leverages IoV to collect real‐time body temperature and breathing rate measurements of pedestrians. This information can be used to recognize geographic areas affected by possible COVID‐19 cases and to implement proactive preventive strategies that would further limit the spread of the disease.

Rapid and non-destructive seed viability prediction using near-infrared hyperspectral imaging coupled with a deep learning approach

Seeds are the basis of the agricultural food industry, greater insights into seed viability before sowing could improve storage management and field performance. In the present study, we aimed to address this issue by using highly cost-efficient near-infrared hyperspectral imaging (NIR-HSI) and a convolutional neural network (CNN) deep learning approach. An NIR-HSI camera was used because it can recognize both molecular vibration information (i.e. chemical component differences) and its spatial distribution in each seed sample; this camera is much more informative than a regular RGB digital camera. Using this technology, the emphasis of this study was firstly to provide a methodology for enhancing the interpretability of viable and non-viable seeds via principal component analysis (PCA) and support vector machine (SVM) viability classification analysis of NIR-HSI data. A CNN was then constructed to“cognize” the differences in viable and non-inviable seeds and classify them automatically. Experimental results indicate that the methodology produces a ~90% classification accuracy for both a five-fold cross-validation set and a test set of naturally aged Japanese mustard spinach seeds. Therefore, this study provides a new strategy for effective and practical seed viability prediction.

Abstract 1655: Hyperspectral metabolic imaging with a camera

Abstract New methods to quantify metabolism in cells, organoids, and tissues are highly demanded for cancer diagnosis and treatment. We developed a light sheet-based volumetric imaging platform to rapidly quantify metabolism via fluorescence spectral signatures of intrinsically fluorescent biomolecules. The ability to quantify metabolism in live cells and tissues can be very powerful as the metabolic activity in rapidly proliferating cells, such as cancer cells, is fundamentally different from non-proliferating cells. Most cancer cells rely on aerobic glycolysis to ferment glucose into lactate independent of the presence of oxygen whereas healthy, differentiated cells primarily rely on oxidative phosphorylation. The rapid generation of cytosolic adenosine triphosphate (ATP) during proliferation results in elevated ratios of the reduced form of nicotinamide adenine dinucleotide (NADH) to NAD+ in cancerous cells that can be detected with fluorescence microscopy of intrinsically (auto)fluorescent molecules such as NADH and flavin adenine dinucleotide (FAD) and its variants. This opens a window to quantify ongoing redox reactions as well as important enzymatic reactions via changes of their fluorescence spectral signatures reporting on the metabolic activity of cells. We developed a light sheet-based hyperspectral imaging platform leveraging custom designed sine/cosine filters and spectral phasor data analysis to directly obtain spectral information with a regular camera. The approach was verified by spectrally resolving 3D reconstructions of fluorescence images of fragments of mouse intestinal crypts. The acquisition speed of light sheet-based hyperspectral imaging is on the order of seconds for one 3D stack allowing for sequential, repeated imaging of several samples over the course of hours to days. In summary, the overall goal of this project is to develop a modular imaging platform to overcome the limitations of current, laser scanning-based spectral imaging and to make quantification of metabolism viable in live three-dimensional cell, organoid and tissue samples, that can then be used to, for example, to study the effects of cancer chemotherapy reagents on the metabolism of patient-derived organoids. Citation Format: Per Niklas Hedde, Enrico Gratton. Hyperspectral metabolic imaging with a camera [abstract]. In: Proceedings of the Annual Meeting of the American Association for Cancer Research 2020; 2020 Apr 27-28 and Jun 22-24. Philadelphia (PA): AACR; Cancer Res 2020;80(16 Suppl):Abstract nr 1655.

Future-Frame Prediction for Fast-Moving Objects with Motion Blur.

We propose a deep neural network model that recognizes the position and velocity of a fast-moving object in a video sequence and predicts the object’s future motion. When filming a fast-moving subject using a regular camera rather than a super-high-speed camera, there is often severe motion blur, making it difficult to recognize the exact location and speed of the object in the video. Additionally, because the fast moving object usually moves rapidly out of the camera’s field of view, the number of captured frames used as input for future-motion predictions should be minimized. Our model can capture a short video sequence of two frames with a high-speed moving object as input, use motion blur as additional information to recognize the position and velocity of the object, and predict the video frame containing the future motion of the object. Experiments show that our model has significantly better performance than existing future-frame prediction models in determining the future position and velocity of an object in two physical scenarios where a fast-moving two-dimensional object appears.

Practical Measurement and Reconstruction of Spectral Skin Reflectance

AbstractWe present two practical methods for measurement of spectral skin reflectance suited for live subjects, and drive a spectral BSSRDF model with appropriate complexity to match skin appearance in photographs, including human faces. Our primary measurement method employs illuminating a subject with two complementary uniform spectral illumination conditions using a multispectral LED sphere to estimate spatially varying parameters of chromophore concentrations including melanin and hemoglobin concentration, melanin blend‐type fraction, and epidermal hemoglobin fraction. We demonstrate that our proposed complementary measurements enable higher‐quality estimate of chromophores than those obtained using standard broadband illumination, while being suitable for integration with multiview facial capture using regular color cameras. Besides novel optimal measurements under controlled illumination, we also demonstrate how to adapt practical skin patch measurements using a hand‐held dermatological skin measurement device, a Miravex Antera 3D camera, for skin appearance reconstruction and rendering. Furthermore, we introduce a novel approach for parameter estimation given the measurements using neural networks which is significantly faster than a lookup table search and avoids parameter quantization. We demonstrate high quality matches of skin appearance with photographs for a variety of skin types with our proposed practical measurement procedures, including photorealistic spectral reproduction and renderings of facial appearance.

Robust Imaging Photoplethysmography in Long-Distance Motion

Imaging photoplethysmography (IPPG) enables contactless physiological parameters monitoring using a regular video camera, which led to an increasing interest in video health monitoring. However, classical IPPG is unable to accurately measure heart rate over a long-distance motion. In this paper, an IPPG heart rate detection framework is proposed based on an adaptive-zoom system called adaptive-zoom IPPG (AZIPPG). It uses an automatic zoom lens to maintain the size of the sensitive area during long-distance motion. The data gathered from AZIPPG were compared with the output of blood volume pulse (BVP) devices and a classical IPPG system. The experimental results showed that AZIPPG achieved high accuracy and correlation in different environments featuring subjects engaged in static and long-distance walking. Actually, AZIPPG eliminates dramatic changes in the ROI, it also introduces image blurring. In this report, the presented theoretical and experimental results indicate that defocused image blurring dose not significantly affect the IPPG's intensity. The pulse data extracted from AZIPPG were comparable to those obtained from the PPG signal of a conventional BVP device. These findings may potentially advance progress in personal health care and telemedicine.

MobiEye: turning your smartphones into a ubiquitous unobtrusive vital sign monitoring system

Recent advances in mobile and wearable technologies have stimulated significantly growing demands for more affordable, user-friendly, pervasive healthcare solutions that can be adopted by the public to proactively manage their health conditions and alleviate the burdens of hospitalization. This study seeks to propose a personalized ubiquitous health monitoring system that can unobtrusively monitor individuals’ vital signs, anywhere at any time. The proposed MobiEye framework makes use of the regular camera available on any smartphones or tablets to record various most important physiological signals without the need for acquiring extra specialized medical devices or attaching any sensor to the body. Through recording the reflected light intensities corresponding to the subtle blood flow changes with blood volume pulses, the proposed technique accurately extracts blood volume pulses from the facial videos recorded in real-world scenarios with the designed protocol. Experiments show that the proposed system achieved $$96 \, \% \,$$ accuracy on average (with the standard deviations of $$\pm 1.2$$ ) for the heart rate estimation and higher correlations between the pulse transit time and the reference systolic blood pressure $$(mean \, r = 0.89,\, SE= 0.05)$$ .

Quantifying wildlife-livestock interactions and their spatio-temporal patterns: Is regular grid camera trapping a suitable approach?

Abstract Camera trapping use has increased significantly in ecological studies in recent decades due to its ability to register information about cryptic and/or elusive species and, more recently, due to its ability to derive population parameters, such as population abundance or density. For these latter applications, camera traps set in a regular grid pattern (CT-RG) are required to obtain representative information of the study area. The present work aims to assess the usefulness of the information collected through CT-RG to study interspecific interactions between animals, in terms of frequency of interaction and their spatiotemporal pattern. The results from CT-RG were compared with those obtained from GPS collars. For this latter methodology, 31 individuals were monitored with GPS-GSM collars (9 red deer [Cervus elaphus], 7 fallow deer [Dama dama], 6 wild boar [Sus scrofa] and 9 cows). The results showed that all the types of interactions recorded by GPS devices were also recorded by CT-RG. However, the relative frequency of each type of interspecific interaction was not precisely estimated using CT-RG. Nor did we observe an overlap between methodologies in the temporality of the interactions. These results are possibly due to most of the interactions tending to occur at aggregation points, which cannot be sufficiently represented by a regular grid. Finally, the spatial pattern obtained from CT-RG correlated with those obtained with GPS collars. Nowadays, camera trapping is being established as an affordable and effective tool to study different population parameters and, therefore, a huge amount of area is monitored with these devices using regular grids. Our results suggest that the information obtained through CT-RG can also be used to study the patterns of interaction between species.

Efficient Relative Pose Estimation for Cameras and Generalized Cameras in Case of Known Relative Rotation Angle

We propose two minimal solvers to the problem of relative pose estimation for a camera with known relative rotation angle. In practice, such angle can be derived from the readings of a 3D gyroscope. Different from other relative pose formulations fusing a camera and a gyroscope, the use of relative rotation angle does not require extrinsic calibration between the two sensors. The first proposed solver is formulated for a calibrated regular camera and requires four-point correspondences from two views. The second solver extends the problem to a generalized camera and requires five-point correspondences. We represent the rotation part of the motion in terms of unit quaternions in order to construct polynomial equations encoding the epipolar constraints. The Grobner basis technique is then used to efficiently derive the polynomial solutions. Our first solver for regular cameras significantly improves the existing state-of-the-art solution. The second solver for generalized cameras is novel. The presented minimal solvers can be used in a hypothesize-and-test architecture such as RANSAC for reliable pose estimation. Experiments on synthetic and real datasets confirm that our algorithms are numerically stable, fast, and robust.

Tiger reappearance in Bhutan’s Bumdeling Wildlife Sanctuary: a case for maintaining effective corridors and metapopulations

The tiger Panthera tigris is globally ‘Endangered’ (Goodrich et al., 2015). The Himalayan Kingdom of Bhutan is an important stronghold (Tempa et al., 2019) for this ecologically and economically important apex predator (Estes et al., 2011; Thinley et al., 2018). Here, we report the reappearance of the tiger in Bumdeling Wildlife Sanctuary (BWS) in north-eastern Bhutan (Fig. 1a), after a likely absence of 12 years. Bumdeling Wildlife Sanctuary is bordered by China in the north and India in the east, and covers 1520 km2 across the districts of Trashiyangtse, Mongar and Lhuentse (Fig. 1a). Altitudes range from 1500 to 6450 m above sea level and the sanctuary contains 745, 42 and 334 species of plants, mammals and birds, respectively (BWS, 2013). Other large predators present in the sanctuary include the snow leopard P. uncia, common leopard P. pardus and dhole Cuon alpinus. They subsist on a diverse ungulate community comprising sambar Rusa unicolor, muntjac Muntiacus muntjac, blue sheep Pseudois nayaur, musk deer Moschus chrysogaster, serow Capricornis thar and goral Naemorhedus goral. There are approximately 800 people who are resident in BWS, predominantly living as subsistence agro-pastoralists (BWS, 2013). Evidence of tiger presence in BWS was initially based on an anecdotal sighting at Thiling village in Sherimuhung Geog in 2001 (subdistrict; Fig. 1b); a claim of 12 yaks predated by tiger between 2003 and 2004 at Rongmateng village in Khoma Geog (Fig. 1b); and sanctuary personnel documenting tiger pugmarks in 2004 at Dongla within Yangtse Geog (Fig. 1b). However, camera trapping in BWS after 2004, including a comprehensive National Tiger Survey (NTS) from 2014 to 2015, failed to capture any tiger images (Thinley et al., 2015). This absence of sightings puzzled local tiger biologists, given the historic presence of tigers in the area, adequate prey availability, minimal hunting and no tiger poaching. Dissatisfied by the poor camera trapping results, sanctuary staff intensified field surveys in 2016 and a tiger pugmark was detected above Rigsumgeonpa monastery in Bumdeling Geog (Fig. 1b). Concurrently, three horses and eight sheep were depredated at Tsango village in Khoma Geog, allegedly by a tiger. Subsequent camera trapping at these livestock predation sites captured 22 tiger images (Fig. 2a,c), confirming at least one individual’s presence within BWS. Opportunistic camera trapping exercises in 2017 yielded nine tiger images (Fig. 2d) near Pangliwog village in Bumdeling Geog, and five images in Sherimuhung Geog. A 2018–2019 study of wildlife in water catchments in BWS and its environs yielded further three tiger images (Fig. 2e) above Gangkardung village in Bumdeling Geog within BWS, and 11 images (Fig. 2b) outside BWS and above Bayling and Bimkhar villages in Yangtse Geog. Analysis of all 50 images revealed a single male tiger, which did not match any of the 68 individual tigers identified in the NTS. This animal could have been an irregular transient not detected in pre-2016 camera trapping, but now resident at BWS. It is unlikely to have ranged from China or India given the functional extinction of the tiger in China (Qin et al., 2015), and no existing tiger records in India adjacent to BWS, due to rampant poaching (Passang Dorji, Forest Ranger, Trashiyangtse Range, Pers. Comm.). The male tiger in BWS is more likely to have originated from populations (Tempa et al., 2019) in either Wangchuck Centennial National Park via Corridor 8, or from Phrumsengla National Park via Corridor 7 (Fig. 1a). The reappearance of the tiger in BWS showcases the effective functionality of Bhutan’s biological corridors. Ensuring this and other tigers’ future survival in BWS entails regular anti-poaching patrols and camera trap monitoring; compensation or insurance programs to offset tiger-related livestock depredation; and designation of a corridor connecting BWS to Sakteng Wildlife Sanctuary (SWS; Fig. 1a). SWS is also linked to Jhomotshangkha Wildlife Sanctuary (JWS; Fig. 1a); both sanctuaries recorded tigers during the NTS, potentially harboring important metapopulations. Radio-collaring of tigers in BWS, SWS and JWS will substantiate ecological research on these metapopulations and confirm the functionality of associated corridors.

Obtaining high-resolution pathology images using a mobile device

In dermatology, pathology images are often required for case review and education. However, without a specialized microscopic camera or expensive slide scanner, taking a clear histologic image can be challenging. Often, placing a regular camera or mobile phone directly against the microscope eyepiece yields suboptimal image quality and a limited field of view. Here, we present a method to obtain a high-resolution microscopic image with an inexpensive mobile phone adaptor and Microsoft PowerPoint (Microsoft, Redmond, WA), allowing viewers to zoom in and out at high resolution and move within the image.

Food tray sealing fault detection using hyperspectral imaging and PCANet

Food trays are very common in shops and supermarkets. Fresh food packaged in trays must be correctly sealed to protect the internal atmosphere and avoid contamination or deterioration. Due to the speed of production, it is not possible to have human quality inspection. Thus, automatic fault detection is a must to reach high production volume. This work describes a deep neural network based on Principal Component Analysis Network (PCANet) for food tray sealing fault detection. The input data come from hyperspectral cameras, showing more characteristics than regular industrial cameras or the human eye as they capture the spectral properties for each pixel. The proposed classification algorithm is divided into three main parts. In the first part, a single image is extracted from the hypercube by using pixel-level fusion method: the cube hyperspectral images are transformed into two-dimensional images to use as the input to the PCANet. Second, a PCANet structure is applied to the fused image. The PCANet has two filter bank layers and one binarization layer (three stages), obtaining a feature vector. Finally, a classification algorithm is used, having the feature vector as input data. The SVM and KNN classifiers were used. The database used in this work is provided by food industry professionals, containing eleven types of contamination in the seal area of the food tray and using metallic opaque cover film. Obtained results show that the design of our framework proposed achieves accuracy of 90% (87% F-measure) and 89% (89% F-measure) for SVM and KNN, respectively. Computation time for classification shows that a food tray speed of 65 trays per second could be reached. As a final result, the influence of the dataset size is analyzed, having PCANet a similar behavior for an extended and a reduced dataset.

Fusionnet: Multispectral Fusion of RGB and NIR Images Using Two Stage Convolutional Neural Networks

In low light condition, color (RGB) images captured by imaging systems suffer from severe noise causing loss of colors and textures. Near infrared (NIR) images, which tend to ignore interference from external lights, have advantage of capturing invisible information that can not be obtained by regular RGB cameras. In this paper, we propose multispectral fusion of RGB and NIR images using two stage convolutional neural networks (CNNs), called FusionNet. Lack of training data is a huge obstacle to the learning-based fusion. We synthesize noisy RGB images for training by adding multiscale Gaussian noise. We adopt two stage CNNs for RGB-NIR fusion that consists of denoising and fusion. First, we use a compact denoising subnetwork to remove severe noise from the input RGB image. Then, we utilize a fusion subnetwork to recover textures of the denoised RGB image with the help of its corresponding NIR image. We provide a perceptually motivated loss function to ensure color/texture consistency between the input RGB image and the output fusion result. Experimental results show that the proposed method produces natural looking fusion results by successfully recovering colors and textures. Moreover, the proposed method outperforms state-of-the-art fusion methods in terms of visual quality and quantitative measurements.

Single-Camera Trilateration

This paper presents a single-camera trilateration scheme which estimates the instantaneous 3D pose of a regular forward-looking camera from a single image of landmarks at known positions. Derived on the basis of the classical pinhole camera model and principles of perspective geometry, the proposed algorithm estimates the camera position and orientation successively. It provides a convenient self-localization tool for mobile robots and vehicles equipped with onboard cameras. Performance analysis has been conducted through extensive simulations with representative examples, which provides an insight into how the input errors and the geometric arrangement of the camera and landmarks affect the performance of the proposed algorithm. The effectiveness of the proposed algorithm has been further verified through an experiment.

Designing Red Eye Reduction Software with Intensity Color Checking Technique

Photographs either with a regular camera or a digital camera where taken from high lighting on people, often producing red spots on the pupils of the eye with the term red eye. This can cause the photo results are not good. With certain software that functions as an image processor, it can easily remove the red eye using only a tool called red eye reduction. So with the existence of the program the results of photos with digital cameras can be edited to eliminate the red eye effect before printing. Red eye reduction used the intensity color checking algorithm in the process of replacing red image pixels and then replacing them with grayish black color according to the process of intensity produced. In this paper program implemented can reduce red eye with the intensity color checking algorithm by processing certain selected regions. The image results can then be printed or saved again in a bitmap format.

Open Vision System for Low-Cost Robotics Education

Vision devices are currently one of the most widely used sensory elements in robots: commercial autonomous cars and vacuum cleaners, for example, have cameras. These vision devices can provide a great amount of information about robot surroundings. However, platforms for robotics education usually lack such devices, mainly because of the computing limitations of low cost processors. New educational platforms using Raspberry Pi are able to overcome this limitation while keeping costs low, but extracting information from the raw images is complex for children. This paper presents an open source vision system that simplifies the use of cameras in robotics education. It includes functions for the visual detection of complex objects and a visual memory that computes obstacle distances beyond the small field of view of regular cameras. The system was experimentally validated using the PiCam camera mounted on a pan unit on a Raspberry Pi-based robot. The performance and accuracy of the proposed vision system was studied and then used to solve two visual educational exercises: safe visual navigation with obstacle avoidance and person-following behavior.

A Fluorescence Sensing Method with Reduced DNA Typing and Low-Cost Instrumentation for Detection of Sample Tampering Cases in Urinalysis.

This work presents a method to unequivocally detect urine sample tampering in cases where integrity of the sample needs to be verified prior to urinalysis. The technique involves the detection of distinct patterns of a triplex short tandem repeats system in DNA extracted from human urine. The analysis is realized with single-dye fluorescence detection and using a regular smartphone camera. The experimental results had demonstrated the efficacy of the analytical approach to obtaining distinct profiles of amplicons in urine from different sample providers. Reproducibility tests with fresh and stored urine have revealed a maximum variation in the profiles within an interval of 5 to 9%. Cases of urine sample tampering via mixture were simulated in the study, and the experiments have identified patterns of mixed genotypes from dual mixtures of urine samples. Moreover, sample adulteration by mixing a non-human fluid with urine in a volume ratio over 25% can be detected. The low cost of the approach is accompanied by the compatibility of the technique to use with different DNA sample preparation protocols and PCR instrumentation. Furthermore, the possibility of realizing the method in an integrated microchip system open great perspectives to conducting sample integrity tests at the site of urine sample reception and/or at resource-limited settings.

A Simple Calibration Procedure for a 2D LiDAR With Respect to a Camera

This paper proposes a simple calibration procedure for a planar scanning laser rangefinder or a 2D LiDAR with respect to a regular monocular camera, in which a point-like target is utilized to obtain point-to-point correspondences, which can make the calibration problem simple. In the proposed method, a small spherical object is attached to a rod, which is hand-held, and the correspondences are simply and effectively acquired by setting the target in front of the sensors and repeatedly moving the target downward by hand, until the LiDAR captures the target. We also describe possible noise present in the proposed data acquisition process, followed by simulation and real experiment. The simulation study examines the performance of the proposed calibration in the presence of noise, in which the LiDAR parameter estimation is successful unless the noise is excessive. In the real experiment, the proposed calibration is conducted by constructing a real calibration system, where we evaluate the reprojection error of the proposed calibration. In both simulation and experiment, a conventional calibration procedure with a v-shaped board target is also conducted for evaluating the proposed calibration procedure. It is confirmed that the proposed method is more effective in point cloud acquisition and calibration performance.