Bokeh Effect Research Articles

Rendering automatic bokeh recommendation engine for photos using deep learning algorithm

Abstract Automatic bokeh is one of the smartphone’s essential photography effects. This effect enhances the quality of the image where the subject background gets out of focus by providing a soft (i.e., diverse) background. Most smartphones have a single rear camera that is lacking to provide which effects need to be applied to which kind of images. To do the same, smartphones depend on different software to generate the bokeh effect on images. Blur, Color-point, Zoom, Spin, Big Bokeh, Color Picker, Low-key, High-Key, and Silhouette are the popular bokeh effects. With this wide range of bokeh types available, it is difficult for the user to choose a suitable effect for their images. Deep Learning (DL) models (i.e., MobileNetV2, InceptionV3, and VGG16) are used in this work to recommend high-quality bokeh effects for images. Four thousand five hundred images are collected from online resources such as Google images, Unsplash, and Kaggle to examine the model performance. 85% accuracy has been achieved for recommending different bokeh effects using the proposed model MobileNetV2, which exceeds many of the existing models.

Типологія конотативних особливостей гумористичного наративу

Specific markers of the humorous narrative of the literary text are articulated on the basis of the theory of humor, in particular, on such concepts as “incongruity” by A. Schopenhauer, the theory of superiority by T. Hobbes, and “liberation” by Z. Freud. The importance of a receptive reaction to the national criteria of a witticism, which produces humorous pathos, is emphasized. Besides, each literary century has its own specifics of creating the comic. The 20th century, against the background of epochal changes, primarily appeals to irony. The texts of J. Hašek (1883–1923), G. von Rezzori (1914–1998) and F. Iskander (1929–2016) are analyzed. Typologically, they are brought together by the “bokeh effect” (blurring of the background), here it is the historical events that are the background to the main problem that appear “blurred”. In J. Hašek’s unfinished novel “The Fateful Adventures of the Good Soldier Švejk” the illogic of the characters actions appears against the background of the absurdity of war. The fragmentary nature of all depicted events at first deforms the image of Švejk, emphasizing that he is an “absolute idiot”, the pretended idiocy of the hero ironically distorts reality. In a similar way, a humorous narrative is created in the work “Last Year’s Snow” by another quasi-Austrian − G. von Rezzori. The rhizomatic structure of the story together with the narrative, assimilated to a child’s consciousness, creates a comic effect of reception (as well as the quasi-childhood of Švejk). For a children’s narrative, one’s own experiences, rather than historical events, become central. A similar somewhat infantile leveling of reality creates situational comedy in F. Iskander’s stories about the Chik’s childhood. The narrator is also a child, but the author preserves the criteria of a children’s narrative (life-affirming humorous pathos, naive point of view, idealization, unchanged chronotope of childhood). Features of humor inherent in the works of quite different writers connotatively coincide in narrative points. They present the typological incongruence of comic means in creating a humorous narrative.

Self-supervised multi-scale pyramid fusion networks for realistic bokeh effect rendering

Images with visual pleasing bokeh effect are often unattainable for mobile cameras with compact optics and tiny sensors. To balance the aesthetic requirements on photo quality and expensive high-end SLR cameras, synthetic bokeh effect rendering has emerged as an attractive machine learning topic for engineering applications on imaging systems. However, most of bokeh rendering models either heavily relied on prior knowledge such as scene depth or were topic-irrelevant data-driven networks without task-specific knowledge, which restricted models’ training efficiency and testing accuracy. Since bokeh is closely related to a phenomenon called ”circle of confusion”, therefore, in this paper, following the principle of bokeh generation, a novel self-supervised multi-scale pyramid fusion network has been proposed for bokeh rendering. During the pyramid fusion process, structure consistencies are employed to emphasize the importance of respective bokeh components. Task-specific knowledge which mimics the ”circle of confusion” phenomenon through disk blur convolutions is utilized as self-supervised information for network training. The proposed network has been evaluated and compared with several state-of-the-art methods on a public large-scale bokeh dataset- the ”EBB!” Dataset. The experiment performance demonstrates that the proposed network has much better processing efficiency and can achieve better realistic bokeh effect with much less parameters size and running time. Related source codes and pre-trained models of the proposed model will be available soon on https://github.com/zfw-cv/MPFNet.

Rendering Natural Bokeh Effects Based on Depth Estimation to Improve the Aesthetic Ability of Machine Vision

Machine vision is the key to realizing computer-vision tasks such as human–computer interaction and autonomous driving. However, human perception of an image’s beauty is innate. If a machine can increase aesthetic awareness, it will greatly improve the comfort of human perception in human–computer interaction. The bokeh effect is one of the most important ways to improve the artistic beauty of photographic images and the image aesthetic quality. Bokeh rendering of an image can highlight the main object of the image and blur unnecessary or unattractive background details. The existing methods usually have unrealistic rendering effects with obvious artifacts around the foreground boundary. Therefore, we propose a natural bokeh-rendering method based on depth estimation that satisfies the following characteristics: objects in the focal plane are clear and out-of-focus objects are blurred; and the further away from the focal plane, the more blurred the objects are. Our method consists of three modules: depth estimation, background subdivision, and bokeh rendering. The background-subdivision module can select different focal planes to obtain different blur radii, making the bokeh-rendering effect more diverse, so that it does not oversegment objects. The bokeh-rendering module adjusts the degree of bokeh by adjusting the blur-radius factor. In the experimental section, we analyze the model results and present the visualization results.

Depth-aware blending of smoothed images for Bokeh effect generation

Bokeh effect is used in photography to capture images where the closer objects look sharp and everything else stays out-of-focus. Bokeh photos are generally captured using Single Lens Reflex cameras using shallow depth-of-field. Most of the modern smartphones can take bokeh images by leveraging dual rear cameras or a good auto-focus hardware. However, for smartphones with single-rear camera without a good auto-focus hardware, we have to rely on software to generate bokeh images. This kind of system is also useful to generate bokeh effect in already captured images. In this paper, an end-to-end deep learning framework is proposed to generate high-quality bokeh effect from images. The original image and different versions of smoothed images are blended to generate Bokeh effect with the help of a monocular depth estimation network. The model is trained through three phases to generate visually pleasing bokeh effect. The proposed approach is compared against a saliency detection based baseline and a number of approaches proposed in AIM 2019 Challenge on Bokeh Effect Synthesis. Extensive experiments are shown in order to understand different parts of the proposed algorithm. The network is lightweight and can process an HD image in 0.03 s. This approach ranked second in AIM 2019 Bokeh effect challenge-Perceptual Track.

Self-Supervised Multi-Scale Pyramid Fusion Networks for Realistic Bokeh Effect Rendering

Self-Supervised Multi-Scale Pyramid Fusion Networks for Realistic Bokeh Effect Rendering

Multitask Deep Neural Networks for Tele-Wide Stereo Matching

In this paper, we introduce the problem of estimating the real world depth of elements in a scene captured by two cameras with different field of views, where the first field of view (FOV) is a Wide FOV (WFOV) captured by a wide angle lens, and the second FOV is contained in the first FOV and is captured by a tele zoom lens. We refer to the problem of estimating the inverse depth for the union of FOVs, while leveraging the stereo information in the overlapping FOV, as Tele-Wide Stereo Matching (TW-SM). We propose different deep learning solutions to the TW-SM problem. Since the disparity is proportional to the inverse depth, we train stereo matching disparity estimation (SMDE) networks to estimate the disparity for the union WFOV. We further propose an end-to-end deep multitask tele-wide stereo matching neural network (MT-TW-SMNet), which simultaneously learns the SMDE task for the overlapped Tele FOV and the single image inverse depth estimation (SIDE) task for the WFOV. Moreover, we design multiple methods for the fusion of the SMDE and SIDE networks. We evaluate the performance of TW-SM on the popular KITTI and SceneFlow stereo datasets, and demonstrate its practicality by synthesizing the Bokeh effect on the WFOV from a tele-wide stereo image pair.

3D Visible Light Indoor Positioning by Bokeh Based Optical Intensity Measurement in Smartphone Camera

This paper proposes a three dimensional (3D) high-resolution indoor positioning system that uses a single light-emitting diode (LED) and a complementary metal-oxide semiconductor image sensor (CIS). The proposed visible light positioning system is based on the bokeh effect for improving the positioning accuracy. The 3D position is determined by received signal strength (RSS) and angle of arrival (AOA). Optical saturation in pixels causes severe performance degradation in the RSS-based positioning based on the CIS. The bokeh-based image reception technique resolves this problem by spreading the optical power to adjacent pixels. Additionally, via the bokeh effect, the size of the LED image on the CIS is maintained regardless of the distances. The accuracy of the LED extraction for the AOA can be improved by fixed LED size. These characteristics of the proposed positioning scheme are analyzed via simulations, and the feasibility of the method is verified based on the experimental results. The experimental results indicate an estimated positioning average error rate within 4%.

A Fixed-Scale Pixelated MIMO Visible Light Communication System

A pixelated MIMO wireless optical communication system is introduced, which transmits a series of time-varying coded images that can be received and decoded by commercial digital cameras. The system exploits the bokeh effect to obtain fixed-scale images at all link distances by placing a convex lens in front of the transmitter array at its focal length and focusing the receiver at infinity. This spatial-angular mapping simplifies the receiver structure requiring no re-focusing as the receiver moves. As an additional benefit, this mapping can also be exploited to provide location information to the receiver. The channel model is measured and modeled and rateless codes are applied to track the truncation of receive images for various link ranges and angular offsets. A proof-of-concept optical communication system is implemented with an LCD display and a high-speed CMOS camera.

Three-dimensional flame measurements with large field angle.

A system for three-dimensional computed tomography of chemiluminescence was developed to measure flames over a large field angle. Nine gradient-index rods, with a 9 × 1 endoscope and only one camera are used. Its large field of view, simplicity, and low cost make it attractive for inner flow field diagnostics. To study the bokeh effect caused by the imaging system on reconstruction solutions, fluorescent beads were used to determine the blurring function. Experiments using a steady diffusion flame were conducted to validate the system. Three models, namely the clear-imaging, out-of-focus imaging, and deconvolution models, were utilized. Taking the bokeh effect into account, the results suggest that based on run-times the deconvolution model provides the best reconstruction accuracy without increasing computational time.

Efficient Ray Tracing Through Aspheric Lenses and Imperfect Bokeh Synthesis

We present an efficient ray-tracing technique to render bokeh effects produced by parametric aspheric lenses. Contrary to conventional spherical lenses, aspheric lenses do generally not permit a simple closed-form solution of ray-surface intersections. We propose a numerical root-finding approach, which uses tight proxy surfaces to ensure a good initialization and convergence behavior. Additionally, we simulate mechanical imperfections resulting from the lens fabrication via a texture-based approach. Fractional Fourier transform and spectral dispersion add additional realism to the synthesized bokeh effect. Our approach is well-suited for execution on graphics processing units (GPUs) and we demonstrate complex defocus-blur and lens-flare effects.

Stereo-based bokeh effects for photography

Bokeh, a sought-after photo rendering style of out-of-focus blur, typically aims at an esthetic quality which is not available to low-end consumer-grade cameras due to the lens design. We present a bokeh simulation method using stereo-vision techniques. We refine a depth map obtained by stereo matching, also using some minor user interaction. Overexposed regions are recovered according to depth information. A depth-aware bokeh effect is then applied with user-adjustable apertures sizes or shapes. We also simulate swirly bokeh, also known as cat-eye effect. Our method mainly aims at the visual quality of the bokeh effect rather than (so far) at time efficiency. Experiments show that our results are natural looking and that they can be comparable to bokeh effects achieved with expensive real-world bokeh-capable camera systems.

Rendering realistic spectral bokeh due to lens stops and aberrations

Creating bokeh effect in synthesized images can improve photorealism and emphasize interesting subjects. Therefore, we present a novel method for rendering realistic bokeh effects, especially chromatic effects, which are absent for existing methods. This new method refers to two key techniques: an accurate dispersive lens model and an efficient spectral rendering scheme. This lens model is implemented based on optical data of real lenses and considers wavelength dependency of physical lenses by introducing a sequential dispersive ray tracing algorithm inside this model. This spectral rendering scheme is proposed to support rendering of lens dispersion and integration between this new model and bidirectional ray tracing. The rendering experiments demonstrate that our method is able to simulate realistic spectral bokeh effects caused by lens stops and aberrations, especially chromatic aberration, and feature high rendering efficiency.

Realistic rendering of bokeh effect based on optical aberrations

Bokeh effect is an important characteristic for realistic image synthesis. However, existing bokeh rendering methods are incapable of simulating realistic bokeh effects due to not taking into account optical characteristics of real lenses, especially optical aberrations. In this paper, a novel realistic bokeh rendering method, based on an accurate camera lens model and distributed ray tracing, is presented. An optical analysis of the relationship between bokeh and optical aberrations, including spherical aberration, coma, astigmatism and field curvature, is firstly introduced. Based on this analysis, a physically-based camera lens model, which takes detailed lens prescription as input, is then introduced for accurately modeling the aberrations. The position and diameter of the entrance and exit pupils are calculated by tracing rays inside the lens for achieving efficient ray sampling, and a general sequential ray tracing algorithm is proposed to better combine with bidirectional ray tracing. Furthermore, correct integration between the lens model and bidirectional ray tracing is also analyzed. The rendering results demonstrate a variety of realistic bokeh effects caused by the aberrations.

Websites of note

Websites of note

Bokode

We show a new camera based interaction solution where an ordinary camera can detect small optical tags from a relatively large distance. Current optical tags, such as barcodes, must be read within a short range and the codes occupy valuable physical space on products. We present a new low-cost optical design so that the tags can be shrunk to 3mm visible diameter, and unmodified ordinary cameras several meters away can be set up to decode the identity plus the relative distance and angle. The design exploits the bokeh effect of ordinary cameras lenses, which maps rays exiting from an out of focus scene point into a disk like blur on the camera sensor. This bokeh-code or Bokode is a barcode design with a simple lenslet over the pattern. We show that a code with 15 μm features can be read using an off-the-shelf camera from distances of up to 2 meters. We use intelligent binary coding to estimate the relative distance and angle to the camera, and show potential for applications in augmented reality and motion capture. We analyze the constraints and performance of the optical system, and discuss several plausible application scenarios.