Camera RGB Research Articles

Development of a smartphone-based colorimetric detection for quantification of vitamin C in dietary supplements and skin care products

This research project presents a development of an assay tool for quantification of vitamin C in dietary supplements and skin care products. The proposed method was developed based on decolorization of acidic potassium permanganate (KMnO4) by vitamin C through a redox reaction, with subsequent colorimetric detection by using digital imaging. The imaging was carried out using smartphone camera and free-downloaded RGB color detector application. To obtain optimal conditions, influential parameters such as light source position installed inside a 19 cm × 46 cm × 10 cm lightbox, reagent concentrations, and reaction time were studied. To the best conditions, good linearity (r2 > 0.99) in the range 1–70 mg vitamin C/L was obtained. Relative standard deviation was less than 3.9% indicating high precision. The proposed method was validated by quantification of vitamin C content in 15 sample solutions using the proposed method in comparison with the reference iodometric titration method. The results obtained from those two methods were not significantly different using paired t-test at confidence level of 95% (texp=0.41, tcrit=2.14). The results indicated that utilizing smartphone camera with free-downloaded color detector application is possible as an alternative analysis tool for determination of vitamin C.

Enhancing RAW-to-sRGB with Decoupled Style Structure in Fourier Domain

RAW to sRGB mapping, which aims to convert RAW images from smartphones into RGB form equivalent to that of Digital Single-Lens Reflex (DSLR) cameras, has become an important area of research. However, current methods often ignore the difference between cell phone RAW images and DSLR camera RGB images, a difference that goes beyond the color matrix and extends to spatial structure due to resolution variations. Recent methods directly rebuild color mapping and spatial structure via shared deep representation, limiting optimal performance. Inspired by Image Signal Processing (ISP) pipeline, which distinguishes image restoration and enhancement, we present a novel Neural ISP framework, named FourierISP. This approach breaks the image down into style and structure within the frequency domain, allowing for independent optimization. FourierISP is comprised of three subnetworks: Phase Enhance Subnet for structural refinement, Amplitude Refine Subnet for color learning, and Color Adaptation Subnet for blending them in a smooth manner. This approach sharpens both color and structure, and extensive evaluations across varied datasets confirm that our approach realizes state-of-the-art results. Code will be available at https://github.com/alexhe101/FourierISP.

EV-LFV: Synthesizing Light Field Event Streams From an Event Camera and Multiple RGB Cameras.

Light field videos captured in RGB frames (RGB-LFV) can provide users with a 6 degree-of-freedom immersive video experience by capturing dense multi-subview video. Despite its potential benefits, the processing of dense multi-subview video is extremely resource-intensive, which currently limits the frame rate of RGB-LFV (i.e., lower than 30 fps) and results in blurred frames when capturing fast motion. To address this issue, we propose leveraging event cameras, which provide high temporal resolution for capturing fast motion. However, the cost of current event camera models makes it prohibitive to use multiple event cameras for RGB-LFV platforms. Therefore, we propose EV-LFV, an event synthesis framework that generates full multi-subview event-based RGB-LFV with only one event camera and multiple traditional RGB cameras. EV-LFV utilizes spatial-angular convolution, ConvLSTM, and Transformer to model RGB-LFV's angular features, temporal features, and long-range dependency, respectively, to effectively synthesize event streams for RGB-LFV. To train EV-LFV, we construct the first event-to-LFV dataset consisting of 200 RGB-LFV sequences with ground-truth event streams. Experimental results demonstrate that EV-LFV outperforms state-of-the-art event synthesis methods for generating event-based RGB-LFV, effectively alleviating motion blur in the reconstructed RGB-LFV.

Ambient and wearable system for workers’ stress evaluation

The paradigm of Industry 4.0 involves fully automated and interconnected industrial production processes demanding a great deal of human-machine interaction. This implies the emergence of new problems related to the stress assessment of workers operating in new and more complex work contexts. To address this need, it may be important to implement automated stress detection platform designed to be effective in a real-world work setting. Many works in the literature deal with the stress evaluation topic, they use above all wearable systems that are often intrusive and subject to noise and artifacts that degrade performance. Moreover, most of them integrate supervised machine learning algorithms, which achieve high levels of detection accuracy, but require a complicated training phase, which might not be suitable in a real-world context. To reduce these limitations, a stress detection platform combining data from a wearable and an environmental system is presented in this paper. It analyses heart rate, galvanic skin response and camera RGB signals. The wearable device was designed to be minimally invasive with good signal stability and low noise, while a commercial camera was added to improve the performance of the whole hardware architecture. From the software perspective, the presented solution was first tested and validated using a supervised approach. Subsequently, attention was focused on the analysis and development of an unsupervised solution, implementing three unsupervised algorithms. The best performance was obtained with the Gaussian Mixture Model having an accuracy of 77.4% considering one level of stress and 75.1% with two levels of stress.

Auxiliary Reference Samples for Extrapolating Spectral Reflectance from Camera RGB Signals.

Surface spectral reflectance is useful for color reproduction. In this study, the reconstruction of spectral reflectance using a conventional camera was investigated. The spectrum reconstruction error could be reduced by interpolating camera RGB signals, in contrast to methods based on basis spectra, such as principal component analysis (PCA). The disadvantage of the interpolation method is that it cannot interpolate samples outside the convex hull of reference samples in the RGB signal space. An interpolation method utilizing auxiliary reference samples (ARSs) to extrapolate the outside samples is proposed in this paper. The ARSs were created using reference samples and color filters. The convex hull of the reference samples and ARSs was expanded to enclose outside samples for extrapolation. A commercially available camera was taken as an example. The results show that with the proposed method, the extrapolation error was smaller than that of the computationally time-consuming weighted PCA method. A low cost and fast detection speed for spectral reflectance recovery can be achieved using a conventional camera.

Survey on deep learning-based 3D object detection in autonomous driving

Autonomous driving technology has entered into the fast lane of development in recent years. An essential component of autonomous driving technology is scene perception, especially 3D object detection. This work gives a comprehensive survey on the up-to-date deep learning-based approaches for 3D object detection in autonomous driving, and categorizes the existing detection models into three classes in terms of their input data format, including LiDAR point cloud-based, Camera RGB image-based, and LiDAR point cloud-camera image fusion-based 3D object detection methods. This work also discusses and analyzes these models according to their characteristics, basic frameworks, advantages and disadvantages, and exhibits the benchmark datasets which are commonly used in the research community. At last, this work summarizes the review work and provides a discussion on the practical challenges and future trend of the research domain.

AUTOMATIC ENRICHMENT OF INDOOR 3D MODELS USING A DEEP LEARNING APPROACH BASED ON SINGLE IMAGES WITH UNKNOWN CAMERA POSES

Abstract. 3D building modeling is a diverse field of research with a multitude of challenges, where data integration is an inherent component. The intensively growing market of BIM-related consumer applications requires methods and algorithms that enable efficient updates of existing 3D models without the need for cost-intensive data capturing and repetitive reconstruction processes. We propose a novel approach for semantic enrichment of existing indoor models by window objects, based on amateur camera RGB images with unknown exterior orientation parameters. The core idea of the approach is the parallel estimation of image camera poses with semantic recognition of target objects and their automatic mapping onto a 3D vector model. The presented solution goes beyond pure texture matching and links deep learning detection techniques with camera pose estimation and 3D reconstruction. To evaluate the performance of our procedure, we compare the estimated camera parameters with reference data, obtaining median values of 13.8 cm for the camera position and 1.1° for its orientation. Furthermore, a quality of 3D mapping is assessed based on the comparison to the reference 3D point cloud. All the windows presented in the data source were detected successfully, with a mean distance between both point sets equal to 3.6 cm. The experimental results prove that the presented approach achieves accurate integration of objects extracted from single images with an input 3D model, allowing for an effective increase of its semantic coverage.

Novel AI driven approach to classify infant motor functions

The past decade has evinced a boom of computer-based approaches to aid movement assessment in early infancy. Increasing interests have been dedicated to develop AI driven approaches to complement the classic Prechtl general movements assessment (GMA). This study proposes a novel machine learning algorithm to detect an age-specific movement pattern, the fidgety movements (FMs), in a prospectively collected sample of typically developing infants. Participants were recorded using a passive, single camera RGB video stream. The dataset of 2800 five-second snippets was annotated by two well-trained and experienced GMA assessors, with excellent inter- and intra-rater reliabilities. Using OpenPose, the infant full pose was recovered from the video stream in the form of a 25-points skeleton. This skeleton was used as input vector for a shallow multilayer neural network (SMNN). An ablation study was performed to justify the network’s architecture and hyperparameters. We show for the first time that the SMNN is sufficient to discriminate fidgety from non-fidgety movements in a sample of age-specific typical movements with a classification accuracy of 88%. The computer-based solutions will complement original GMA to consistently perform accurate and efficient screening and diagnosis that may become universally accessible in daily clinical practice in the future.

Continuous Hue-Based Self-Calibration of a Smartphone Spectrometer Applied to Optical Fiber Fabry-Perot Sensor Interrogation.

Smartphone-based optical spectrometers allow the development of a new generation of portable and cost-effective optical sensing solutions that can be easily integrated into sensor networks. However, most commonly the spectral calibration relies on the external reference light sources which have known narrow spectral lines. Such calibration must be repeated each time the fiber and diffraction grating holders are removed from the smartphone and reattached. Moreover, the spectrometer wavelength scale can drift during the measurement because of the smartphone temperature fluctuations. The present work reports on a novel spectral self-calibration approach, based on the correspondence between the light wavelength and the hue features of the spectrum measured using a color RGB camera. These features are caused by the nonuniformity of camera RGB filters’ responses and their finite overlap, which is a typical situation for RGB cameras. Thus, the wavelength scale should be externally calibrated only once for each smartphone spectrometer and can further be continuously verified and corrected using the proposed self-calibration approach. An ability of the plug-and play operation and the temperature drift elimination of the smartphone spectrometer was experimentally demonstrated. Conducted experiments involved interrogation of optical fiber Fabry-Perot interferometric sensor and demonstrated a nanometer-level optical path difference resolution.

Developing a multi-spectral imaging system using a RGB camera under two illuminations

This paper proposes a multi-spectral imaging system, developed using a commercial-grade camera, under two commonly used illumination. Rather than using conventional direct or diffuse light, the novelty of our method is to use a cross-polarized imaging system to eliminate glare and specular highlights. Two RGB images are captured under two different color temperature lighting conditions. An improved reflectance estimation method is developed to transform camera RGB under two illumination to spectral reflectance using a regulated model, combining the polynomial expansion of the camera signals with optimally selected feature. The method was tested using both a semi-gloss ColorChecker SG (140) and matte ColorChecker DC (240) chart. The results indicate that the proposed method significantly outperforms the traditional methods both in terms of spectra and colorimetric accuracy. This new multi-spectral imaging system is sufficiently precise to predict spectra properties and its performance within an acceptable range.

A Flying Gray Ball Multi-illuminant Image Dataset for Color Research

For research in the field of illumination estimation and color constancy, there is a need for ground-truth measurement of the illumination color at many locations within multi-illuminant scenes. A practical approach to obtaining such ground-truth illumination data is presented here. The proposed method involves using a drone to carry a gray ball of known percent surface spectral reflectance throughout a scene while photographing it frequently during the flight using a calibrated camera. The captured images are then post-processed. In the post-processing step, machine vision techniques are used to detect the gray ball within each frame. The camera RGB of light reflected from the gray ball provides a measure of the illumination color at that location. In total, the dataset contains 30 scenes with 100 illumination measurements on average per scene. The dataset is available for download free of charge.

Skin Chromophore Estimation from Mobile Selfie Images using Constrained Independent Component Analysis

Human skin is made up of two primary chromophores: melanin, the pigment in the epidermis giving skin its color; and hemoglobin, the pigment in the red blood cells of the vascular network within the dermis. The relative concentrations of these chromophores provide a vital indicator for skin health and appearance. We present a technique to automatically estimate chromophore maps from RGB images of human faces captured with mobile devices such as smartphones. The ultimate goal is to provide a diagnostic aid for individuals to monitor and improve the quality of their facial skin. A previous method approaches the problem as one of blind source separation, and applies Independent Component Analysis (ICA) in camera RGB space to estimate the chromophores. We extend this technique in two important ways. First we observe that models for light transport in skin call for source separation to be performed in log spectral reflectance coordinates rather than in RGB. Thus we transform camera RGB to a spectral reflectance space prior to applying ICA. This process involves the use of a linear camera model and Principal Component Analysis to represent skin spectral reflectance as a lowdimensional manifold. The camera model requires knowledge of the incident illuminant, which we obtain via a novel technique that uses the human lip as a calibration object. Second, we address an inherent limitation with ICA that the ordering of the separated signals is random and ambiguous. We incorporate a domain-specific prior model for human chromophore spectra as a constraint in solving ICA. Results on a dataset of mobile camera images show high quality and unambiguous recovery of chromophores.

Spectra estimation from raw camera responses based on adaptive local-weighted linear regression

An improved spectral reflectance estimation method is developed to transform raw camera RGB responses to spectral reflectance. The novelty of our method is to apply a local weighted linear regression model for spectral reflectance estimation and construct the weighting matrix using a Gaussian function in CIELAB uniform color space. The proposed method was tested using both a standard color chart and a set of textile samples, with a digital RGB camera and by ten times ten-fold cross-validation. The results demonstrate that our method gives the best accuracy in estimating both the spectral reflectance and the colorimetric values in comparison with existing methods.

High dynamic range image reconstruction in device-independent color space based on camera colorimetric characterization

High dynamic range (HDR) image acquisition techniques can record both bright and dark areas of the real scene. In order to acquire accurate colorimetric information as well as the luminance of the HDR scene, a reconstruction method in CIEXYZ device-independent color space based on camera colorimetric method was proposed in this paper. The camera RGB values of individual pictures are firstly transformed into the corresponding CIEXYZ values by colorimetric characterization with unrestricted capture settings. Then the XYZ values are fused by the weighting function determined according to the characterization accuracy. The prediction accuracy of the proposed method was verified by two phases of experiments to outperform the traditional one, with the color difference being less than 2 CIELAB units under the constant relative spectral power distribution (SPD) of lighting and 3.5 CIELAB units for the illuminations of different correlated color temperatures (CCTs), respectively.

Smartphone snapshot mapping of skin chromophores under triple-wavelength laser illumination

Chromophore distribution maps are useful tools for skin malformation severity assessment and for monitoring of skin recovery after burns, surgeries, and other interactions. The chromophore maps can be obtained by processing several spectral images of skin, e.g., captured by hyperspectral or multispectral cameras during seconds or even minutes. To avoid motion artifacts and simplify the procedure, a single-snapshot technique for mapping melanin, oxyhemoglobin, and deoxyhemoglobin of in-vivo skin by a smartphone under simultaneous three-wavelength (448–532–659 nm) laser illumination is proposed and examined. Three monochromatic spectral images related to the illumination wavelengths were extracted from the smartphone camera RGB image data set with respect to crosstalk between the RGB detection bands. Spectral images were further processed accordingly to Beer’s law in a three chromophore approximation. Photon absorption path lengths in skin at the exploited wavelengths were estimated by means of Monte Carlo simulations. The technique was validated clinically on three kinds of skin lesions: nevi, hemangiomas, and seborrheic keratosis. Design of the developed add-on laser illumination system, image-processing details, and the results of clinical measurements are presented and discussed.

The use of RGB cameras in defining crop development in legumes

The human population is expected to reach 9 billion by 2050 and thus high yield crop varieties need to be developed. Remote sensing can estimate crop parameters non-destructively and quickly. The aim of this study was to compare and evaluate the use of a commercial RGB camera with an expensive canopy sensor in the crop development of two legumes. The RGB camera based vegetation index (NGRDI) was compared with the canopy sensor derived vegetation indices (NDVI and NDRE) for estimating legume crop growth parameters. The results indicated that the use of a simple digital camera RGB can in some cases replace spectral canopy sensors.

Method for hue plane preserving color correction.

Hue plane preserving color correction (HPPCC), introduced by Andersen and Hardeberg [Proceedings of the 13th Color and Imaging Conference (CIC) (2005), pp. 141-146], maps device-dependent color values (RGB) to colorimetric color values (XYZ) using a set of linear transforms, realized by white point preserving 3×3 matrices, where each transform is learned and applied in a subregion of color space, defined by two adjacent hue planes. The hue plane delimited subregions of camera RGB values are mapped to corresponding hue plane delimited subregions of estimated colorimetric XYZ values. Hue planes are geometrical half-planes, where each is defined by the neutral axis and a chromatic color in a linear color space. The key advantage of the HPPCC method is that, while offering an estimation accuracy of higher order methods, it maintains the linear colorimetric relations of colors in hue planes. As a significant result, it therefore also renders the colorimetric estimates invariant to exposure and shading of object reflection. In this paper, we present a new flexible and robust version of HPPCC using constrained least squares in the optimization, where the subregions can be chosen freely in number and position in order to optimize the results while constraining transform continuity at the subregion boundaries. The method is compared to a selection of other state-of-the-art characterization methods, and the results show that it outperforms the original HPPCC method.

Improved method for skin reflectance reconstruction from camera images.

A improved spectral reflectance reconstruction method is developed to transform camera RGB to spectral reflectance for skin images. Rather than using conventional direct or two-step processes, we transform camera RGB to skin reflectance directly using a principal component analysis (PCA) approach. The novelty in our direct method (RGB to spectra) is the use of a skin-specific colour characterisation chart with spectra closer to human skin spectra, and a new database of skin reflectances to derive the PCA bases. The experimental results using the facial images of 17 subjects demonstrate that our new direct method gives a significantly better performance than conventional, two-step methods and direct methods with traditional characterization charts. This new spectral reconstruction algorithm is sufficiently precise to reconstruct spectral properites relating to chromophores and its performance is within the acceptable range for maxillofacial soft tissue prostheses (error < 3 ΔE*ab units).

Rank-based camera spectral sensitivity estimation

In order to accurately predict a digital camera response to spectral stimuli, the spectral sensitivity functions of its sensor need to be known. These functions can be determined by direct measurement in the lab-a difficult and lengthy procedure-or through simple statistical inference. Statistical inference methods are based on the observation that when a camera responds linearly to spectral stimuli, the device spectral sensitivities are linearly related to the camera rgb response values, and so can be found through regression. However, for rendered images, such as the JPEG images taken by a mobile phone, this assumption of linearity is violated. Even small departures from linearity can negatively impact the accuracy of the recovered spectral sensitivities, when a regression method is used. In our work, we develop a novel camera spectral sensitivity estimation technique that can recover the linear device spectral sensitivities from linear images and the effective linear sensitivities from rendered images. According to our method, the rank order of a pair of responses imposes a constraint on the shape of the underlying spectral sensitivity curve (of the sensor). Technically, each rank-pair splits the space where the underlying sensor might lie in two parts (a feasible region and an infeasible region). By intersecting the feasible regions from all the ranked-pairs, we can find a feasible region of sensor space. Experiments demonstrate that using rank orders delivers equal estimation to the prior art. However, the Rank-based method delivers a step-change in estimation performance when the data is not linear and, for the first time, allows for the estimation of the effective sensitivities of devices that may not even have "raw mode." Experiments validate our method.

Critical spectra in the color reproduction process of digital motion picture cameras

Most cameras resolve a color stimulus by three spectral sensitivity responses. Commonly, these do not satisfy the Luther-Ives condition. Hence, there is no linear mapping between the camera's RGB values and the CIE XYZ tristimulus values. Although more complex methods for camera characterization have been proposed, none can solve the issues caused by camera metamerism: While being different to the human eyes, certain stimuli will induce the same RGB response in the camera. To investigate color reproduction errors and the effects of device metamerism, metameric spectra must be found. Previously proposed methods generate metameric pairs artificially. We present a method for finding metameric pairs within a database of real-world measured spectra, which we believe has more practical utility than methods employing synthesized metameric pairs. By using measured camera sensitivities and reflectances from the SOCS database we can identify metamers with a real-world occurrence possibility. We characterize critical spectra by type of object and by hue, considering both CIE illuminants and real-world LED fixtures.