Texture Fidelity Research Articles

Frequency Integration and Spatial Compensation Network for infrared and visible image fusion

Infrared and visible image fusion aims to synthesize a fused image that emphasizes the salient objects while retaining the intricate texture and visual quality from both infrared and visible images. In opposite to the majority of existing deep learning-based fusion approaches, which predominantly focus on spatial information and neglect the valuable frequency information, we propose a novel method that delves into both domains simultaneously to tackle the infrared and visible image fusion task. Specifically, we first analyze the frequency characteristics of the two modality images via Fourier transform, and observe that fusion results with complementary attributes from source images can be effectively attained by directly incorporating their phase components. To this end, we propose a Frequency Integration and Spatial Compensation Network (FISCNet), consisting of two core designs: a frequency integration component and a spatial compensation component. The former integrates prominent objects from the source images while maintaining the visual perception from the visible image in the frequency domain, and the latter improves the detailed texture and emphasizes the salient objects through a meticulous compensation mechanism in the spatial domain. Extensive experiments on various benchmarks demonstrate the superiority of our method over state-of-the-art alternatives in terms of both salience preservation and texture fidelity. Code is available at https://github.com/zheng980629/FISCNet.

Accurate 3D models in both geometry and texture: An archaeological application

There are two main ways of building 3D models of archaeological objects. The first way is to use photogrammetry by means of image-based modelling systems. The second way is the use of 3D scanners. Both methods have advantages and problems. In this work, we propose a simple way to merge both methods to preserve the best of each method and obtain 3D models with high accuracy and excellent resolution and texture fidelity. Building on the work of other proposals offering an alternative solution, we demonstrate a model with absolute units, where the colour of the texture is calibrated to assure a very good photo finish. Fidelity results produced by RMS estimation of the deviations of the colour patches offer better results for the photo camera than for the cameras in the scanners used. The final model maintains the metric properties of the scanned model, which was our metric reference.

Face photo-sketch synthesis based on multi-scale feature fusion

The synthesis of face sketches from real face photos and its inverse process have a wide range of applications, such as digital entertainment and assisting in the investigation of criminal cases. However, due to the significant discrepancies in texture between photos and sketches, the transformation between them is still a challenging problem. Recently, the methods based on generative adversarial nets (GANs) have shown encouraging results in the transformation between images, especially the transformation from photos to sketches. However, most of them produce different deformations or blur in the key components of the face, which affects the realism of the synthetic image. To address this challenge, we propose a novel face photo-sketch synthesis algorithm based on multi-scale feature fusion to improve the structural integrity and texture fidelity of the synthetic image. Firstly, an encoder is utilized to extract the multi-scale coding features of the input image. Then the bottom coding features are passed by a dilated convolution module and transmitted to the decoder for decoding. In the decoding process, the decoding features of different scales and the coding features of corresponding scales are concatenated in the channel dimension to obtain the multi-scale coding and decoding fusion features. Finally, the coding and decoding fusion features of different scales are further fused at the output of the decoder and the final synthesis result is generated through a convolution layer. By combining the features of different scales in the encoding and decoding processes in the channel dimension at the same time, we can maintain good image structure and texture details and generate realistic face sketch/photo images. We verify the effectiveness of the proposed method in several challenging datasets. The quantitative and qualitative evaluation shows that the proposed method is superior to other state-of-the-art methods in generating face sketches (or photos) with high visual quality.

A VERSATILE MULTI-CAMERA SYSTEM FOR 3D ACQUISITION AND MODELING

Abstract. Image-based 3D models generation typically involves three stages, namely: 2D image acquisition, data processing, and 3D surface generation and editing. The availability of different easy-to-use and low-cost image acquisition solutions, combined with open-source or commercial processing tools, has democratized the 3D reconstruction and digital twin generation. But high geometric and texture fidelity on small- to medium-scale objects as well as integrated commercial system for mass 3D digitization are not available. The paper presents our effort to build such a system, i.e. a market-ready multi-camera solution and a customized reconstruction process for mass 3D digitization of small to medium objects. The system is realized as a joint work between industrial and academic partners, in order to employ the latest technologies for the needs of the market. The proposed versatile image acquisition and processing system pushes to the limits the 3D digitization pipeline combining a rigid capturing system with photogrammetric reconstruction methods.

COMPREHENSIVE DIGITAL DOCUMENTATION AND PRELIMINARY STRUCTURAL ASSESSMENT OF SATUNSAT: A UNIQUE MAYA ARCHITECTURAL LABYRINTH AT OXKINTOK, YUCATAN, MEXICO

Abstract. Located in the Puuc region of Yucatan, Mexico, the Labyrinth of Oxkintok, also known as Satunsat, is one of the most enigmatic ancient structures in the northern Maya lowlands. Inside this otherwise unremarkable terraced building platform is a series of interconnected vaulted passageways and horizontal ducts that spans three levels. In 2016, we conducted a comprehensive digital documentation of the exterior and interior surfaces and spaces of Satunsat via terrestrial laser scanning (TLS) and structure-from-motion (SfM) photogrammetry. The result is a fused, geometrically accurate, full coverage model with exceptional color and texture fidelity. These data enabled a preliminary structural health assessment and evaluation of the varied masonry techniques employed in the construction of Satunsat and can inform future studies related to the structure’s function and meaning.

Producing High-Quality 3-D Models for Specimen Research with the Creaform 3-D Scanner at the Sternberg Museum of Natural History

Digitization of natural history collections for specimen accessibility, research, and posterity has been a major push in museums over the past decade. One increasingly common form of specimen digitization is three-dimensional (3-D) surface scanning. There are numerous 3-D scanners on the market, including visible light and laser surface scanners. However, mobility of the scanning device, mesh resolution, and texture fidelity are among the most prominent issues facing regular implementation of 3-D scanning procedures for research purposes. Morphometrics research involving the digitization of Xiphactinus at the Sternberg Museum of Natural History (FHSM) using a Creaform Go!SCAN 50 visible light 3-D surface scanner has necessitated the development of a workflow using this 3-D scanner and the associated VXelements software. This workflow maximizes model quality with minimal loss of speed or efficiency when scanning numerous specimens. Major components of this workflow include proper use of positioning targets, appropriate merging of multiple specimen scans, and how to perform post-processing on models without sacrificing model resolution or texture. The primary benefit of the Creaform 3-D scanner is hardware mobility, allowing for use on large exhibited specimens or digitizing specimens during research trips. Shortcomings include relatively low mesh resolution, (bottoming out at 0.5 mm), difficulty scanning highly complex structures, and low texture fidelity of scanned models. However, texture fidelity can be improved through use of the free software, Meshlab, by overlaying high-quality stacked photographs onto the 3-D mesh. Incorporating Meshlab texture overlays into the workflow allows for the production of digital models with improved color and textural data – ideal for accurate visualization of specimens in 3-D viewers for educational outreach and online databases. While not perfect for all applications, the Creaform 3-D scanner allows for rapid digitization of specimens in an increasingly digital collections environment, and is sufficient for some research applications. The workflow established at the FHSM can serve as a framework for other institutions to use the Creaform 3-D scanner to best suit their digitization needs.

Pose-invariant face recognition with homography-based normalization

Pose-invariant face recognition (PIFR) refers to the ability that recognizes face images with arbitrary pose variations. Among existing PIFR algorithms, pose normalization has been proved to be an effective approach which preserves texture fidelity, but usually depends on precise 3D face models or at high computational cost. In this paper, we propose an highly efficient PIFR algorithm that effectively handles the main challenges caused by pose variation. First, a dense grid of 3D facial landmarks are projected to each 2D face image, which enables feature extraction in an pose adaptive manner. Second, for the local patch around each landmark, an optimal warp is estimated based on homography to correct texture deformation caused by pose variations. The reconstructed frontal-view patches are then utilized for face recognition with traditional face descriptors. The homography-based normalization is highly efficient and the synthesized frontal face images are of high quality. Finally, we propose an effective approach for occlusion detection, which enables face recognition with visible patches only. Therefore, the proposed algorithm effectively handles the main challenges in PIFR. Experimental results on four popular face databases demonstrate that the propose approach performs well on both constrained and unconstrained environments.

High-quality streamable free-viewpoint video

We present the first end-to-end solution to create high-quality free-viewpoint video encoded as a compact data stream. Our system records performances using a dense set of RGB and IR video cameras, generates dynamic textured surfaces, and compresses these to a streamable 3D video format. Four technical advances contribute to high fidelity and robustness: multimodal multi-view stereo fusing RGB, IR, and silhouette information; adaptive meshing guided by automatic detection of perceptually salient areas; mesh tracking to create temporally coherent subsequences; and encoding of tracked textured meshes as an MPEG video stream. Quantitative experiments demonstrate geometric accuracy, texture fidelity, and encoding efficiency. We release several datasets with calibrated inputs and processed results to foster future research.

Depth similarity enhanced image summarization algorithm for hole-filling in depth image-based rendering

In free viewpoint video (FVV) and 3DTV, the depth image-based rendering method has been put forward for rendering virtual view video based on multi-view video plus depth (MVD) format. However, the projection with slightly different perspective turns the covered background regions into hole regions in the rendered video. This paper presents a depth enhanced image summarization generation model for the hole-filling via exploiting the texture fidelity and the geometry consistency between the hole and the remaining nearby regions. The texture fidelity and the geometry consistency are enhanced by drawing texture details and pixel-wise depth information into the energy cost of similarity measure correspondingly. The proposed approach offers significant improvement in terms of 0.2dB PSNR gain, 0.06 SSIM gain and subjective quality enhancement for the hole-filling images in virtual viewpoint video.

Spin transformations of discrete surfaces

We introduce a new method for computing conformal transformations of triangle meshes in R 3 . Conformal maps are desirable in digital geometry processing because they do not exhibit shear , and therefore preserve texture fidelity as well as the quality of the mesh itself. Traditional discretizations consider maps into the complex plane, which are useful only for problems such as surface parameterization and planar shape deformation where the target surface is flat. We instead consider maps into the quaternions H, which allows us to work directly with surfaces sitting in R 3 . In particular, we introduce a quaternionic Dirac operator and use it to develop a novel integrability condition on conformal deformations. Our discretization of this condition results in a sparse linear system that is simple to build and can be used to efficiently edit surfaces by manipulating curvature and boundary data, as demonstrated via several mesh processing applications.

Recolouring digital textile printing design with high fidelity

An algorithm of recolouring of polychromatic digital textile printing images is proposed based on the investigation of texture and colour distribution in different channels and image segmentation. A polychromatic image is first separated into monochromatic regions based on watershed transformation in CIELAB colour space. The initial markers are selected by hierarchical histogram analysis to eliminate the inherent drawbacks of over‐segmentation in the watershed algorithm. Then the individual monochromatic regions can be mapped with different colours to obtain desirable designs. The artefacts in the boundaries of different regions are reduced by a technique of colour mixing through Gaussian blurring. The experimental simulation results indicated that the performance of the algorithm was quite good in both texture and colour fidelity.