Specular Highlights Research Articles

Flexible Colon Polyp Detection: A Dual Mode Approach for Detection and Segmentation of Colon Polyps with Optional Inpainting for Specular Highlight Mitigation

Flexible Colon Polyp Detection: A Dual Mode Approach for Detection and Segmentation of Colon Polyps with Optional Inpainting for Specular Highlight Mitigation

PDE-based Specular Highlight Elimination

Dealing with reflections in images captured through glass would be real headache, as they can obscure the important stuff behind the glass and make the whole image look messy. This is a major problem in many computer vision tasks.Early studies reported that a popular way to tackle the challenge of removing reflections from [1] single images in deep learning. In this article, we take a deep dive into the research on this topic from 2015 to 2021, focusing on how deep learning is being used for [5] single-image reflection removal [4].We searched through a bunch of important online databases and libraries, like IEEE Xplore, Google Scholar, ScienceDirect, SpringerLink, and ACM Digital Library, to find relevant research papers. After carefully going through them, we picked out 25 papers [9] that fit the criteria for our review.We analyzed these papers to answer seven major questions about how deep learning and [3] neural networks are being used for [6] single-image reflection removal. This will hopefully give future researchers a good understanding of what's been done in this area and help them build on that knowledge.The review also highlights the important challenges that data scientists are facing in this area, and also some promising directions for future research. . And importantly, it provides a list of useful datasets that data scientists can use to benchmark their own deep learning techniques against other studies. Whether you're a researcher hungry for the next challenge or just someone who wants to understand how it all works, this review will equip you with the knowledge and inspiration to delve deeper into this fascinating field.

Joint network for specular highlight detection and adversarial generation of specular-free images trained with polarimetric data

Specular highlights in images pose a significant challenge in algorithms for image segmentation, object detection and other image-based decision-making systems. However, most systems ignore this particular scenario and neglect input images with specular highlights instead of mitigating it in the pre-processing stage. In this paper, we leverage deep neural networks and take advantage of the varying illumination information in polarimetric images for synthesizing specular-free images. We propose a multi-domain Specular Highlight Mitigation Generative Adversarial Network (SHMGAN) with self-attention. SHMGAN consists of a single generator–discriminator pair trained simultaneously using polarimetric images. The proposed GAN uses a dynamically generated attention mask based on a specularity segmentation network without requiring additional manual input. The network is able to learn the illumination variation between the four polarimetric images and a pseudo-diffuse image. Once trained, SHMGAN is able to generate specular-free images from a single RGB image as input; without requiring any additional external labels. The proposed network is trained and tested publicly available datasets of real-world images. SHMGAN is able to accurately identify the specularity affected pixels and generates high visual quality images with mitigated specular reflections. The generated images are realistic and have very low noise, distortions and aberrations compared to the existing state-of-the-art methods for specular highlight removal.

Self-supervised monocular depth estimation for high field of view colonoscopy cameras.

Optical colonoscopy is the gold standard procedure to detect colorectal cancer, the fourth most common cancer in the United Kingdom. Up to 22%-28% of polyps can be missed during the procedure that is associated with interval cancer. A vision-based autonomous soft endorobot for colonoscopy can drastically improve the accuracy of the procedure by inspecting the colon more systematically with reduced discomfort. A three-dimensional understanding of the environment is essential for robot navigation and can also improve the adenoma detection rate. Monocular depth estimation with deep learning methods has progressed substantially, but collecting ground-truth depth maps remains a challenge as no 3D camera can be fitted to a standard colonoscope. This work addresses this issue by using a self-supervised monocular depth estimation model that directly learns depth from video sequences with view synthesis. In addition, our model accommodates wide field-of-view cameras typically used in colonoscopy and specific challenges such as deformable surfaces, specular lighting, non-Lambertian surfaces, and high occlusion. We performed qualitative analysis on a synthetic data set, a quantitative examination of the colonoscopy training model, and real colonoscopy videos in near real-time.

Image Specular Highlight Removal using Generative Adversarial Network and Enhanced Grey Wolf Optimization Technique

Image Specular Highlight Removal using Generative Adversarial Network and Enhanced Grey Wolf Optimization Technique

A Fast Specular Highlight Removal Method for Smooth Liquor Bottle Surface Combined with U2-Net and LaMa Model.

Highlight removal is a critical and challenging problem. In view of the complex highlight phenomenon on the surface of smooth liquor bottles in natural scenes, the traditional highlight removal algorithms cannot semantically disambiguate between all-white or near-white materials and highlights, and the recent highlight removal algorithms based on deep learning lack flexibility in network architecture, have network training difficulties and have insufficient object applicability. As a result, they cannot accurately locate and remove highlights in the face of some small sample highlight datasets with strong pertinence, which reduces the performance of some tasks. Therefore, this paper proposes a fast highlight removal method combining U2-Net and LaMa. The method consists of two stages. In the first stage, the U2-Net network is used to detect the specular reflection component in the liquor bottle input image and generate the mask map for the highlight area in batches. In the second stage, the liquor bottle input image and the mask map generated by the U2-Net are input to the LaMa network, and the surface highlights of the smooth liquor bottle are removed by relying on the powerful image inpainting performance of LaMa. Experiments on our self-made liquor bottle surface highlight dataset showed that this method outperformed other advanced methods in highlight detection and removal.

Fast rendering and display of light field images with a controllable lighting mechanism.

A fast light field (LF) image rendering method with controllable lighting mechanism is proposed and demonstrated. It solves the issue that previous image-based methods could not render and edit lighting effects for LF images. In contrast to previous methods, light cones and normal maps are defined and used to expand the RGBD images into RGBDNθ data, which gives more degrees of freedom to render LF images. Conjugate cameras are used to capture the RGBDN data, which simultaneously solve the pseudoscopic imaging problem. Perspective coherence is used to accelerate the RGBDNθ-based LF rendering process, which has been shown to be on average 30 times faster than the traditional per-viewpoint rendering (PVR) method. Vivid three-dimensional (3D) images with Lambertian reflection and non-Lambertian reflection effects including specular lighting and compound lighting have been reconstructed in 3D space using a homemade LF display system. The proposed method injects more flexibility into the rendering of LF images and can also be used in holographic display, augmented reality, virtual reality, and other fields.

A Specular Highlight Removal Algorithm for Quality Inspection of Fresh Fruits

Nondestructive inspection technology based on machine vision can effectively improve the efficiency of fresh fruit quality inspection. However, fruits with smooth skin and less texture are easily affected by specular highlights during the image acquisition, resulting in light spots appearing on the surface of fruits, which severely affects the subsequent quality inspection. Aiming at this issue, we propose a new specular highlight removal algorithm based on multi-band polarization imaging. First of all, we realize real-time image acquisition by designing a new multi-band polarization imager, which can acquire all the spectral and polarization information through single image capture. Then we propose a joint multi-band-polarization characteristic vector constraint to realize the detection of specular highlight, and next we put forward a Max-Min multi-band-polarization differencing scheme combined with an ergodic least-squares separation for specular highlight removal, and finally, the chromaticity consistency regularization is used to compensate the missing details. Experimental results demonstrate that the proposed algorithm can effectively and stably remove the specular highlight and provide more accurate information for subsequent fruit quality inspection. Besides, the comparison of algorithm speed further shows that our proposed algorithm has a good tradeoff between accuracy and complexity.

Generalized resampled importance sampling

As scenes become ever more complex and real-time applications embrace ray tracing, path sampling algorithms that maximize quality at low sample counts become vital. Recent resampling algorithms building on Talbot et al.'s [2005] resampled importance sampling (RIS) reuse paths spatiotemporally to render surprisingly complex light transport with a few samples per pixel. These reservoir-based spatiotemporal importance resamplers (ReSTIR) and their underlying RIS theory make various assumptions, including sample independence. But sample reuse introduces correlation , so ReSTIR-style iterative reuse loses most convergence guarantees that RIS theoretically provides. We introduce generalized resampled importance sampling (GRIS) to extend the theory, allowing RIS on correlated samples, with unknown PDFs and taken from varied domains. This solidifies the theoretical foundation, allowing us to derive variance bounds and convergence conditions in ReSTIR-based samplers. It also guides practical algorithm design and enables advanced path reuse between pixels via complex shift mappings. We show a path-traced resampler (ReSTIR PT) running interactively on complex scenes, capturing many-bounce diffuse and specular lighting while shading just one path per pixel. With our new theoretical foundation, we can also modify the algorithm to guarantee convergence for offline renderers.

Hierarchical Disordered Colloidal Thin Films with Duplex Optical Elements for Advanced Anti‐Counterfeiting Coding

AbstractMultiscale structures are found in nature to impart brilliant structural colors with integrated properties of multiple optical elements. However, it remains challenging to build more than one optical element into artificial photonic materials in practical manufacturing. Here, hierarchical disordered photonic superstructures (HDPSs) with optical properties arising from tailored interplay of thin‐film interference (TFI) and coherent scattering from amorphous colloidal arrays (ACAs) are fabricated through the combined use of infiltration‐driven colloidal assembly and micro‐imprinting. Uniform thin ACA films with TFI effects (ACA‐TFIs) are first coated on premium glossy photo papers, displaying near‐isotropic colors under natural light and mixed colors upon specular lighting due to the duplex optical effects. Such collective effects are easily modulated by the film thicknesses of ACA‐TFIs. Also, micro‐imprinting that can remove a patterned monolayer of particles from ACA‐TFI films causes suppressed TFI in the imprinted regions, leading to HDPS films with encoded optical features. Collectively, HDPS anti‐counterfeiting labels with covert and sophisticated optical coding are achieved and their enhanced complexity results in infinite possible optical features that are virtually impractical to forge, opening new avenues to high‐level security applications.

Assessing gloss under diffuse and specular lighting

AbstractThe visual sensation of gloss is built on cues deduced from the interaction between light, surfaces sunder evaluation, and surrounding conditions. Gloss is a second‐order attribute of the visual appearance, this means that its perception is not directly encoded on biological sensors but constructed from the global scene in the field of view of the observer. It is then a complex quantity to measure. When most studies based on simulated samples stress on the importance of realistic observation conditions, we measure the effect of environment complexity over perception of real samples. We test two different lighting conditions: either diffuse or a combination of diffuse and collimated lighting in order to approach natural complex illumination patterns. Under both lighting conditions, we test two environments: a standard black light booth, designed according to the ASTM D4449, and a realistic office cubicle. Samples consist in a seven‐level gloss scale ranging from full matt to high gloss. These are presented to observers through pair comparison protocol, according to a maximum likelihood difference scaling algorithm. Our results confirm that gloss constancy is maintained even if the convergence of illumination varies. We however measure that the constancy is lost for matt samples perception.

Spec-Net and Spec-CGAN: Deep learning models for specularity removal from faces

The process of splitting an image into specular and diffuse components is a fundamental problem in computer vision, because most computer vision algorithms, such as image segmentation and tracking, assume diffuse surfaces, so existence of specular reflection can mislead algorithms to make incorrect decisions. Existing decomposition methods tend to work well for images with low specularity and high chromaticity, but they fail in cases of high intensity specular light and on images with low chromaticity. In this paper, we address the problem of removing high intensity specularity from low chromaticity images (faces). We introduce a new dataset, Spec-Face, comprising face images corrupted with specular lighting and corresponding ground truth diffuse images. We also introduce two deep learning models for specularity removal, Spec-Net and Spec-CGAN. Spec-Net takes an intensity channel as input and produces an output image that is very close to ground truth, while Spec-CGAN takes an RGB image as input and produces a diffuse image very similar to the ground truth RGB image. On Spec-Face, with Spec-Net, we obtain a peak signal-to-noise ratio (PSNR) of 3.979, a local mean squared error (LMSE) of 0.000071, a structural similarity index (SSIM) of 0.899, and a Fréchet Inception Distance (FID) of 20.932. With Spec-CGAN, we obtain a PSNR of 3.360, a LMSE of 0.000098, a SSIM of 0.707, and a FID of 31.699. With Spec-Net and Spec-CGAN, it is now feasible to perform specularity removal automatically prior to other critical complex vision processes for real world images, i.e., faces. This will potentially improve the performance of algorithms later in the processing stream, such as face recognition and skin cancer detection.

A Global Optimization Method for Specular Highlight Removal From a Single Image

The presence of specular highlight is a critical issue for both natural and medical images such as those produced by laparoscopes, which can lead to erroneous visual tracking, stereo reconstruction, and image segmentation. Specular highlight removal from a single image is necessary for image analysis and applications. Due to the differences between natural and medical image scenes, existing literature to address this issue has only been effective on natural images or medical images with textureless regions. To overcome this limitation, we propose a global optimization method for specular highlight removal from a single image based on a dichromatic reflection model. In addition to introducing modified illumination chromaticity, the proposed method consists of two novel steps: one for estimating diffuse chromaticity by correcting hue and saturation on highlighted regions, and the other for estimating diffuse and specular reflection coefficients using convex optimization with double regularization. The estimated diffuse chromaticity is proven to approximate the true diffuse chromaticity and the proposed optimization algorithm is guaranteed to find the optimal diffuse coefficients. Experimental results show that the proposed method can effectively remove specular highlights from both natural images and endoscopic images with texture detail preservation. To further demonstrate the efficacy of our proposed method, an application of stereo reconstruction using a public dataset illustrates that our highlight removal method can enhance surface reconstruction accuracy from 1.10mm RMSD to 0.69mm RMSD.

Specular Highlight Removal for High Reflection Surface with Linear Diffuser

In structured light 3D measurement field, when the object has smooth surface, it can form a highlight area due to the specular reflection, and the distortion of the object will make a large measurement error. In order to solve this problem, this paper use seven steps sine-phase shift combined with linear diffuser to remove highlight. Firstly, the principle of removing specular with diffuser is analyzed, then the overall design of the system is introduced, which includes 3D reconstruction and system calibration method. Finally the reconstructed experiments are carried out with ceramic plate. Experimental results show that the proposed method can significantly reduce the highlights area of reconstructed image compared with the highlights area without diffuser. The diffuser can obviously inhibited by highlights, although it can not completely remove the highlights, it plays a very important role in reconstructing specular object with more accurate and better quality.

Light Field Imaging Based Accurate Image Specular Highlight Removal

Specular reflection removal is indispensable to many computer vision tasks. However, most existing methods fail or degrade in complex real scenarios for their individual drawbacks. Benefiting from the light field imaging technology, this paper proposes a novel and accurate approach to remove specularity and improve image quality. We first capture images with specularity by the light field camera (Lytro ILLUM). After accurately estimating the image depth, a simple and concise threshold strategy is adopted to cluster the specular pixels into “unsaturated” and “saturated” category. Finally, a color variance analysis of multiple views and a local color refinement are individually conducted on the two categories to recover diffuse color information. Experimental evaluation by comparison with existed methods based on our light field dataset together with Stanford light field archive verifies the effectiveness of our proposed algorithm.

Perceiving Object Shape from Specular Highlight Deformation, Boundary Contour Deformation, and Active Haptic Manipulation.

It is well known that motion facilitates the visual perception of solid object shape, particularly when surface texture or other identifiable features (e.g., corners) are present. Conventional models of structure-from-motion require the presence of texture or identifiable object features in order to recover 3-D structure. Is the facilitation in 3-D shape perception similar in magnitude when surface texture is absent? On any given trial in the current experiments, participants were presented with a single randomly-selected solid object (bell pepper or randomly-shaped “glaven”) for 12 seconds and were required to indicate which of 12 (for bell peppers) or 8 (for glavens) simultaneously visible objects possessed the same shape. The initial single object’s shape was defined either by boundary contours alone (i.e., presented as a silhouette), specular highlights alone, specular highlights combined with boundary contours, or texture. In addition, there was a haptic condition: in this condition, the participants haptically explored with both hands (but could not see) the initial single object for 12 seconds; they then performed the same shape-matching task used in the visual conditions. For both the visual and haptic conditions, motion (rotation in depth or active object manipulation) was present in half of the trials and was not present for the remaining trials. The effect of motion was quantitatively similar for all of the visual and haptic conditions–e.g., the participants’ performance in Experiment 1 was 93.5 percent higher in the motion or active haptic manipulation conditions (when compared to the static conditions). The current results demonstrate that deforming specular highlights or boundary contours facilitate 3-D shape perception as much as the motion of objects that possess texture. The current results also indicate that the improvement with motion that occurs for haptics is similar in magnitude to that which occurs for vision.

A Specular Highlight Removal Method with Shape Constrained Inpainting

The existence of specular highlight is a great obstacle of shape recovery. For a single gray-scale image with only intensity information, the existed highlight detection methods based on chroma or polarization analysis cannot directly be applied on it. So, a method using surface shape is provided, it makes full use of the imaging process. Through the surface normal estimation, compute the maximum a posteriori probability of each reflection composition under simulated annealing and detect the highlight areas. Finally, remove the highlight based on the assumption of curvature continuity. Experiments reveal that this method is effective on synthetic and real-world images, improves the accuracy of specular highlight removed for image combined specular highlight.

Qualitative shape from shading, specular highlights, and mirror reflections

Qualitative shape from shading, specular highlights, and mirror reflections

Specular Highlight Removal For Image-Based Dietary Assessment.

Traditional dietary assessment methods, consisting of written and orally reported methods, are not widely acceptable or feasible for everyday monitoring. The development of builtin cameras for mobile devices provides a new way of collecting dietary information by acquiring images of foods and beverages. The ability of image analysis techniques to automatically segment and identify food items from food images becomes imperative. Food images, usually consisting of plates, bowls and glasses, are often affected by lighting and specular highlights which present difficulties for image analysis. In this paper, we propose a novel single-image specular highlight removal method to detect and remove specular highlights in food images. We use independent components analysis (ICA) to separate the specular and diffuse components from the original image using only one image. This paper describes the details of the proposed model and also presents experimental results on food images to demonstrate the effectiveness of our approach.

Fast high-resolution appearance editing using superimposed projections

We present a system that superimposes multiple projections onto an object of arbitrary shape and color to produce high-resolution appearance changes. Our system produces appearances at an improved resolution compared to prior works and can change appearances at near interactive rates. Three main components are central to our system. First, the problem of computing compensation images is formulated as a constrained optimization which yields high-resolution appearances. Second, decomposition of the target appearance into base and scale images enables fast swapping of appearances on the object by requiring the constrained optimization to be computed only once per object. Finally, to make high-quality appearance edits practical, an elliptical Gaussian is used to model projector pixels and their interaction between projectors. To the best of our knowledge, we build the first system that achieves high-resolution and high-quality appearance edits using multiple superimposed projectors on complex nonplanar colored objects. We demonstrate several appearance edits including specular lighting, subsurface scattering, inter-reflections, and color, texture, and geometry changes on objects with different shapes and colors.