Synthetic Graphics Research Articles

Generative Portrait Shadow Removal

We introduce a high-fidelity portrait shadow removal model that can effectively enhance the image of a portrait by predicting its appearance under disturbing shadows and highlights. Portrait shadow removal is a highly ill-posed problem where multiple plausible solutions can be found based on a single image. For example, disentangling complex environmental lighting from original skin color is a non-trivial problem. While existing works have solved this problem by predicting the appearance residuals that can propagate local shadow distribution, such methods are often incomplete and lead to unnatural predictions, especially for portraits with hard shadows. We overcome the limitations of existing local propagation methods by formulating the removal problem as a generation task where a diffusion model learns to globally rebuild the human appearance from scratch as a condition of an input portrait image. For robust and natural shadow removal, we propose to train the diffusion model with a compositional repurposing framework: a pre-trained text-guided image generation model is first fine-tuned to harmonize the lighting and color of the foreground with a background scene by using a background harmonization dataset; and then the model is further fine-tuned to generate a shadow-free portrait image via a shadow-paired dataset. To overcome the limitation of losing fine details in the latent diffusion model, we propose a guided-upsampling network to restore the original high-frequency details (e.g. , wrinkles and dots) from the input image. To enable our compositional training framework, we construct a high-fidelity and large-scale dataset using a lightstage capturing system and synthetic graphics simulation. Our generative framework effectively removes shadows caused by both self and external occlusions while maintaining original lighting distribution and high-frequency details. Our method also demonstrates robustness to diverse subjects captured in real environments.

Post-Earthquake Building Evaluation Using UAVs: A BIM-Based Digital Twin Framework.

Computer vision has shown potential for assisting post-earthquake inspection of buildings through automatic damage detection in images. However, assessing the safety of an earthquake-damaged building requires considering this damage in the context of its global impact on the structural system. Thus, an inspection must consider the expected damage progression of the associated component and the component’s contribution to structural system performance. To address this issue, a digital twin framework is proposed for post-earthquake building evaluation that integrates unmanned aerial vehicle (UAV) imagery, component identification, and damage evaluation using a Building Information Model (BIM) as a reference platform. The BIM guides selection of optimal sets of images for each building component. Then, if damage is identified, each image pixel is assigned to a specific BIM component, using a GrabCut-based segmentation method. In addition, 3D point cloud change detection is employed to identify nonstructural damage and associate that damage with specific BIM components. Two example applications are presented. The first develops a digital twin for an existing reinforced concrete moment frame building and demonstrates BIM-guided image selection and component identification. The second uses a synthetic graphics environment to demonstrate 3D point cloud change detection for identifying damaged nonstructural masonry walls. In both examples, observed damage is tied to BIM components, enabling damage to be considered in the context of each component’s known design and expected earthquake performance. The goal of this framework is to combine component-wise damage estimates with a pre-earthquake structural analysis of the building to predict a building’s post-earthquake safety based on an external UAV survey.

A Graphic/Picture Classification Method Based on Spatial Characteristics of Colors

In order to get optimum color reproduction results on different color devices in color management, a proper rendering intent should be selected according to color characteristics of the image. In terms of differences on color characteristics, images could be classified into synthetic graphics and natural pictures. Different rendering intent should be applied on graphics and pictures. So graphic/picture automatic classification becomes a fundamental task of color management intellectualization. Characteristics on color distribution of a large number of images have been researched in our experiments. Then it is confirmed that the essential difference between graphic and picture is the characteristics on color distribution in the neighborhood of images rather than the number of colors or the volume of image gamut. Thus, the features which have distinct ability to show the differences could be used to build classification rules. In this paper, several mathematical features of image are extracted and selected by their classification performance. Based on these features, the discriminant analysis is adopted to build up discriminated functions. Finally, the accuracy of the functions has been tested and the precision is 96.75%.

Generation of synthetic documents for performance evaluation of symbol recognition & spotting systems

This paper deals with the topic of performance evaluation of symbol recognition & spotting systems. We propose here a new approach to the generation of synthetic graphics documents containing non-isolated symbols in a real context. This approach is based on the definition of a set of constraints that permit us to place the symbols on a pre-defined background according to the properties of a particular domain (architecture, electronics, engineering, etc.). In this way, we can obtain a large amount of images resembling real documents by simply defining the set of constraints and providing a few pre-defined backgrounds. As documents are synthetically generated, the groundtruth (the location and the label of every symbol) becomes automatically available. We have applied this approach to the generation of a large database of architectural drawings and electronic diagrams, which shows the flexibility of the system. Performance evaluation experiments of a symbol localization system show that our approach permits to generate documents with different features that are reflected in variation of localization results.

Boundary macroblock padding in MPEG-4 video decoding using a graphics coprocessor

MPEG-4 is the latest multimedia coding standard that supports object-based coding and manipulation of natural video and synthetic graphics objects. Due to its various features and high coding efficiency, MPEG-4 is becoming popular in video streaming applications. Many graphics coprocessors provide the acceleration of inverse discrete cosine transform (IDCT) and motion compensation for real-time video decoding. Therefore, it is desired to use the graphics coprocessors to accelerate MPEG-4 video decoding as well. Since MPEG-4 video decoding for rectangular video objects is similar to other video coding standards, e.g., MPEG-2, the IDCT and motion compensation can still be executed on the graphics coprocessors. However, we have found that boundary macroblock padding, which is an essential processing step in decoding arbitrarily shaped video objects, could not be efficiently accelerated on the graphics coprocessors due to its complexity. Although we can implement the boundary macroblock padding on the host processor, the frame data processed on the graphics coprocessor need to be transferred to the host processor for padding. In addition, the padded data on the host processor need to be sent back to the graphics coprocessor to be used as a reference for subsequent frames. To avoid this overhead, we present two approaches of boundary macroblock padding. In the first approach, the boundary macroblock padding is partitioned into two tasks, one of which the host processor can perform without the overhead of data transfers. In the second approach, we propose two new instructions and an algorithm that can be easily adopted in the next-generation graphics coprocessors or mediaprocessors, which gives a performance improvement of up to a factor of nine compared to that with the Pentium III.

Stereo Matching with Transparency and Matting

This paper formulates and solves a new variant of the stereo correspondence problem: simultaneously recovering the disparities, true colors, and opacities of visible surface elements. This problem arises in newer applications of stereo reconstruction, such as view interpolation and the layering of real imagery with synthetic graphics for special effects and virtual studio applications. While this problem is intrinsically more difficult than traditional stereo correspondence, where only the disparities are being recovered, it provides a principled way of dealing with commonly occurring problems such as occlusions and the handling of mixed (foreground/background) pixels near depth discontinuities. It also provides a novel means for separating foreground and background objects (matting), without the use of a special blue screen. We formulate the problem as the recovery of colors and opacities in a generalized 3D (x, y, d) disparity space, and solve the problem using a combination of initial evidence aggregation followed by iterative energy minimization.

Premo: An Emerging Standard for Multimedia Presentation Part I: Overview and Framework

Developers needing to realize high-level multimedia applications are essentially left on their own. Only a few programming tools allow the creation of multimedia effects based on a more general model than multimedia documents. No currently available ISO standard encompasses these needs. A standard in this area should focus more on the presentation aspects of multimedia and less on the coding, transfer, or hypermedia document aspects, which are covered other standards. It should also concentrate on programming tools rather than multimedia document format. These are exactly the main concerns of the Premo (Presentation Environment for Multimedia Objects) standard, the subject of the article. Premo's major features can be briefly summarized as follows: Premo is a presentation environment that aims to provide a standard programming environment in a very general sense, one that helps promote portable multimedia applications; Premo targets multimedia presentation, whereas earlier SC24 standards concentrated either on synthetic graphics or image-processing systems; Premo is object oriented. This means that, through standard object-oriented techniques, a Premo implementation becomes extensible and configurable. Object-oriented technology also provides a framework to describe distribution in a consistent manner.

Premo: an emerging standard for multimedia presentation. I. Overview and framework

Developers needing to realize high-level multimedia applications are essentially left on their own. Only a few programming tools allow the creation of multimedia effects based on a more general model than multimedia documents. No currently available ISO standard encompasses these needs. A standard in this area should focus more on the presentation aspects of multimedia and less on the coding, transfer, or hypermedia document aspects, which are covered other standards. It should also concentrate on programming tools rather than multimedia document format. These are exactly the main concerns of the Premo (Presentation Environment for Multimedia Objects) standard, the subject of the article. Premo's major features can be briefly summarized as follows: Premo is a presentation environment that aims to provide a standard programming environment in a very general sense, one that helps promote portable multimedia applications; Premo targets multimedia presentation, whereas earlier SC24 standards concentrated either on synthetic graphics or image-processing systems; Premo is object oriented. This means that, through standard object-oriented techniques, a Premo implementation becomes extensible and configurable. Object-oriented technology also provides a framework to describe distribution in a consistent manner.