Real Texture Research Articles

Decoding roughness perception in distributed haptic devices

Abstract The ability to render realistic texture perception using haptic devices has been consistently challenging. A key component of texture perception is roughness. When we touch surfaces, mechanoreceptors present under the skin are activated and the information is processed by the nervous system, enabling perception of roughness/smoothness. Several distributed haptic devices capable of producing localized skin stretch have been developed with the aim of rendering realistic roughness perception; however, current state-of-the-art devices rely on device fabrication and psychophysical experimentation to determine whether a device configuration will perform as desired. Predictive models can elucidate physical mechanisms, providing insight and a more effective design iteration process. Since existing models (1, 2) are derived from responses to normal stimuli only, they cannot predict the performance of laterally actuated devices which rely on frictional shear forces to produce localized skin stretch. They are also unable to predict the augmentation of roughness perception when the actuators are spatially dispersed across the contact patch or actuated with a relative phase difference (3). In this study, we have developed a model that can predict the perceived roughness for arbitrary external stimuli and validated it against psychophysical experimental results from different haptic devices reported in literature. The model elucidates two key mechanisms: 1) The variation in the change of strain across the contact patch can predict roughness perception with strong correlation 2) The inclusion of lateral shear forces is essential to correctly predict roughness perception. Using the model can accelerate device optimization by obviating the reliance on trial-and-error approaches.

Development, validation, and application of a generic image-based noise addition method for simulating reduced dose computed tomography images.

Major efforts in computed tomography (CT) have been focused on reducing radiation dose to patients while maintaining adequate diagnostic quality. To that end, research tools have been developed to simulate reduced-dose images via either image-based or projection-based methods. The former is limited to fully capturing realistic texture, streak, and non-stationary characteristics of reduced dose, while the latter is impractical clinically. To develop and validate an image-based noise addition method that accounts for such attributes while being practical in clinical settings. A noise addition method was developed to add realistic noise in the image domain. The method first estimates the noise power spectrum (NPS) of CT images, which are also forward-projected to form synthetic projections. The projection data are supplemented with random white noise proportional to their attenuation values. The noise sinogram is then back-projected onto the image, filtered by the NPS, and scaled according to the desired dose reduction level. The tool was evaluated using both phantom images and patient data. The phantom images were acquired using a multi-sized image quality phantom (Mercury Phantom 3.0, Duke University), and a thorax anthropomorphic phantom (Lungman Phantom, Kyoto Kagaku) at different dose levels and reconstruction settings. The patient images consisted of two dose levels of various CT examinations and reconstruction settings. The simulated and real reduced-dose images were compared in terms of the noise magnitude and texture (i.e., NPS average frequency, NPS-fav). The utility of this methodology was also assessed for routine clinical use for CT protocol review. For the phantom images, the percent errors in the noise magnitude between the simulated images and the actual images of the Mercury Phantom and anthropomorphic phantom images were 3.34% and 3.50%, respectively. The difference in fav was 0.07mm-1 for the Mercury Phantom and 0.06mm-1 for the anthropomorphic phantom between the simulated and actual images. The average noise magnitude percent error between the simulated and actual patient images was 4.61% with noise texture judged to be visually comparable with some kernel dependencies. When implemented clinically, the tool proved practical to simplify the process of estimating radiation dose reduction for CT protocols, resulting in a 50% dose reduction of our multiple myeloma protocol. The method generated simulated CT images with realistic noise properties similar to images acquired at the same radiation exposure without needing access to raw projection data.

Using a morph-based animation to visualise the face of Pharaoh Ramesses II ageing from middle to old age

Ramesses II was one of the most important Pharaohs to have presided over Egypt during the New Kingdom period. In 2023 researchers Wilkinson, Saleem, Liu and Roughley produced two digital 3D facial depictions showing Ramesses II at different ages: one around the age-at-death at 90 years old and the other, an age-regression at approximately 45 years old, based CT scans of his mummified remains, photographs, and historical information. The presence of two 3D facial depictions of one ancient individual at different ages affords an opportunity to show how Ramesses II might have looked during key moments of his lifetime and just prior to death. This paper describes the workflow adopted to add realistic textures to the facial depictions, and to use a morph-based animation to represent Ramesses II ageing from 45 to 90 years old.

JOA‐GAN: An improved single‐image super‐resolution network for remote sensing based on GAN

AbstractImage super‐resolution (SR) has been widely applied in remote sensing to generate high‐resolution (HR) images without increasing hardware costs. However, SR is a severe ill‐posed problem. As deep learning advances, existing methods have solved this problem to a certain extent. However, the complex spatial distribution of remote sensing images still poses a challenge in effectively extracting abundant high‐frequency details from the images. Here, a single‐image super‐resolution (SISR) network based on the generative adversarial network (GAN) for remote sensing is presented, called JOA‐GAN. Firstly, a joint‐attention module (JOA) is proposed to focus the network on high‐frequency regions in remote sensing images to enhance the quality of image reconstruction. In the generator network, a multi‐scale densely connected feature extraction block (ERRDB) is proposed, which acquires features at different scales using MSconv blocks containing multi‐scale convolutions and automatically adjusts the features by JOA. In the discriminator network, the relative discriminator is used to compute the relative probability instead of the absolute probability, which helps the network learn clearer and more realistic texture details. JOA‐GAN is compared with other advanced methods, and the results demonstrate that JOA‐GAN has improved objective evaluation metrics and achieved superior visual effects.

Infrared Image Super-Resolution Network Utilizing the Enhanced Transformer and U-Net.

Infrared images hold significant value in applications such as remote sensing and fire safety. However, infrared detectors often face the problem of high hardware costs, which limits their widespread use. Advancements in deep learning have spurred innovative approaches to image super-resolution (SR), but comparatively few efforts have been dedicated to the exploration of infrared images. To address this, we design the Residual Swin Transformer and Average Pooling Block (RSTAB) and propose the SwinAIR, which can effectively extract and fuse the diverse frequency features in infrared images and achieve superior SR reconstruction performance. By further integrating SwinAIR with U-Net, we propose the SwinAIR-GAN for real infrared image SR reconstruction. SwinAIR-GAN extends the degradation space to better simulate the degradation process of real infrared images. Additionally, it incorporates spectral normalization, dropout, and artifact discrimination loss to reduce the potential image artifacts. Qualitative and quantitative evaluations on various datasets confirm the effectiveness of our proposed method in reconstructing realistic textures and details of infrared images.

TDEGAN: A Texture-Detail-Enhanced Dense Generative Adversarial Network for Remote Sensing Image Super-Resolution

Image super-resolution (SR) technology can improve the resolution of images and provide clearer and more reliable remote sensing images of high quality to better serve the subsequent applications. However, when reconstructing high-frequency feature areas of remote sensing images, existing SR reconstruction methods are prone to artifacts that affect visual effects and make it difficult to generate real texture details. In order to address this issue, a texture-detail-enhanced dense generative adversarial network (TDEGAN) for remote sensing image SR is presented. The generator uses multi-level dense connections, residual connections, and Shuffle attention (SA) to improve the feature extraction ability. A PatchGAN-style discrimination network is designed to effectively perform local discrimination and helps the network generate rich, detailed features. To reduce the impact of artifacts, we introduce an artifact loss function, which is combined with the exponential moving average (EMA) technique to distinguish the artifacts generated from the actual texture details through local statistics, which can help the network reduce artifacts and generate more realistic texture details. Experiments show that TDEGAN can better restore the texture details of remote sensing images and achieves certain advantages in terms of evaluation indicators and visualization.

The Return of the Warrior: Combining Anthropology, Imaging Advances, and Art in Reconstructing the Face of the Early Medieval Skeleton

Reconstructing the face from the skull is important not only for forensic identification but also as a tool that can provide insight into the appearance of individuals from past populations. It requires a multidisciplinary approach that combines anthropological knowledge, advanced imaging methods, and artistic skills. In the present study, we demonstrate this process on the skull of an early medieval warrior from Croatia. The skeletal remains were prepared and scanned using multi-slice computed tomography (MSCT) and examined using standard anthropological and radiological methods. The analysis revealed that the remains belonged to a 35–45-year-old male individual who had suffered severe cranial trauma, probably causing his death. From MSCT images, we reconstructed a three-dimensional (3D) model of the skull, on which we digitally positioned cylinders demarking the soft tissue thickness and created the face with a realistic texture. A 3D model of the face was then optimized, printed, and used to produce a clay model. Sculpturing techniques added skin textures and facial features with scars and trauma manifestations. Finally, after constructing a plaster model, the model was painted and refined by adding fine details like eyes and hair, and it was prepared for presentation in the form of a sculpture.

Digital Camouflage Generation Based on an Improved CycleGAN Network Model

Abstract This paper proposes a digital camouflage generation method based on an improved CycleGAN to produce camouflage patterns with a high degree of fusion with the background and realistic texture details. Firstly, a SE-ResNet network structure is constructed by combining the residual network ResNet with the channel attention mechanism SENet, enabling flexible adjustment of channel weights to effectively extract crucial channel features and enhance the network's perception capability of important information in images. Secondly, a color preservation loss is introduced to improve the adversarial loss function, thereby avoiding training instability and fluctuation in pattern quality. Experimental results demonstrate that the camouflage patterns generated using the proposed method achieve a Structural Similarity Index (SSIM) of 0.77 and a Peak Signal-to-Noise Ratio (PSNR) of 18.9, representing improvements of 0.27 and 3.3, respectively, compared to the original CycleGAN. This method can generate digital camouflage patterns with richer details, textures, and high fusion with the background.

39‐3: Invited Paper: Kirameki Display: Technical Approaches to Represent Real Texture with Light Fields

We have developed a “Kirameki display” that can represent a real texture of materials. (“Kirameki” means a shining/glittering sense in Japanese.) In addition, we have investigated an improvement of “texture representation” by optimization of both an optical design and a software algorithm. Lastly, we discuss the future technical fields for texture representation on displays.

FashionMorph: Contextually Adaptive Clothing Replacement with CLIP, Segmentation, and Stable Diffusion

Abstract: FashionMorph represents a groundbreaking advancement in digital image manipulation, seamlessly blending advanced segmentation techniques with the language comprehension capabilities of CLIP. This innovative approach simplifies the process of precise garment replacements while ensuring the preservation of realistic textures and boundaries. By incorporating stable diffusion, FashionMorph eliminates the need for extensive training on specific clothing styles, offering users a streamlined experience in expressing their preferences through natural language prompts. This empowers users in photo editing, granting them unparalleled creative freedom. The system comprises a comprehensive pipeline designed for the identification and inpainting of clothing elements. Leveraging CLIP and CLIPSeg models for robust image analysis, FashionMorph accurately identifies clothing elements and generates corresponding masks. The inpainting process utilizes a stable diffusion pipeline, resulting in impressive image quality that surpasses the performance of notable models like DCTransformer, StyleGAN, and ADM. Evaluations based on FID and SFID scores, along with Inception Score assessments, further validate FashionMorph's effectiveness in context-aware garment editing. Visual representations of original images, inpainted results, and masks vividly demonstrate the successful outcomes achieved by FashionMorph, establishing it as an efficient and user-friendly solution for professionals and enthusiasts alike. Among the evaluated models, Stable Diffusion emerges as the standout performer, boasting remarkable FID and SFID scores of 0.35 and 0.20, respectively, indicative of its outstanding image fidelity and diversity. Conversely, DCTransformer exhibits the highest FID and SFID scores, while StyleGAN excels in diversity but falls short in FID. ADM (dropout) strikes a balance between fidelity and diversity. These results underscore the exceptional performance of Stable Diffusion, solidifying its position as the leading approach with the lowest FID and SFID values

Realistic Texture Mapping of 3D Medical Models Using RGBD Camera for Mixed Reality Applications

Augmented and mixed reality in the medical field is becoming increasingly important. The creation and visualization of digital models similar to reality could be a great help to increase the user experience during augmented or mixed reality activities like surgical planning and educational, training and testing phases of medical students. This study introduces a technique for enhancing a 3D digital model reconstructed from cone-beam computed tomography images with its real coloured texture using an Intel D435 RGBD camera. This method is based on iteratively projecting the two models onto a 2D plane, identifying their contours and then minimizing the distance between them. Finally, the coloured digital models were displayed in mixed reality through a Microsoft HoloLens 2 and an application to interact with them using hand gestures was developed. The registration error between the two 3D models evaluated using 30,000 random points indicates values of: 1.1 ± 1.3 mm on the x-axis, 0.7 ± 0.8 mm on the y-axis, and 0.9 ± 1.2 mm on the z-axis. This result was achieved in three iterations, starting from an average registration error on the three axes of 1.4 mm to reach 0.9 mm. The heatmap created to visualize the spatial distribution of the error shows how it is uniformly distributed over the surface of the pointcloud obtained with the RGBD camera, except for some areas of the nose and ears where the registration error tends to increase. The obtained results indicate that the proposed methodology seems effective. In addition, since the used RGBD camera is inexpensive, future approaches based on the simultaneous use of multiple cameras could further improve the results. Finally, the augmented reality visualization of the obtained result is innovative and could provide support in all those cases where the visualization of three-dimensional medical models is necessary.

AFDet: alignment and focusing for aerial object detection

ABSTRACT Object detection in aerial images has become a focus in recent years due to the expansion of its application fields. Aerial objects have the characteristics of multi-scale, arbitrary angle and dense arrangement, which brings considerable challenges to the task. Based on the efficient one-stage detectors, most methods design modules to adaptively align the feature receptive field with the anchors to improve the accuracy of the bounding box regression and calculate loss according to parameter bias. Current methods are not adequately data-driven in feature optimization and the angle parameter regression faces boundary problems. To handle these problems, in this letter, we propose an Align-Focus detector (AFDet) for aerial image object detection. We introduce the ideology of deformable multi-head self-attention to feature optimization and design a new Align-Focus Module (AFM) which can sensitively focus feature encoding response to the real texture area of the objects. In addition, we apply KF-IOU Loss to solve the boundary problem. We conduct necessary experiments on DOTA and HRSC2016 datasets, and AFDet achieves the highest mAP performance compared to the existing methods.

PCE-Palm: Palm Crease Energy Based Two-Stage Realistic Pseudo-Palmprint Generation

The lack of large-scale data seriously hinders the development of palmprint recognition. Recent approaches address this issue by generating large-scale realistic pseudo palmprints from Bézier curves. However, the significant difference between Bézier curves and real palmprints limits their effectiveness. In this paper, we divide the Bézier-Real difference into creases and texture differences, thus reducing the generation difficulty. We introduce a new palm crease energy (PCE) domain as a bridge from Bézier curves to real palmprints and propose a two-stage generation model. The first stage generates PCE images (realistic creases) from Bézier curves, and the second stage outputs realistic palmprints (realistic texture) with PCE images as input. In addition, we also design a lightweight plug-and-play line feature enhancement block to facilitate domain transfer and improve recognition performance. Extensive experimental results demonstrate that the proposed method surpasses state-of-the-art methods. Under extremely few data settings like 40 IDs (only 2.5% of the total training set), our model achieves a 29% improvement over RPG-Palm and outperforms ArcFace with 100% training set by more than 6% in terms of TAR@FAR=1e-6.

Doing more with less: Realistic stereoscopic three-dimensional anatomical modeling from smartphone photogrammetry.

Traditional teaching methods struggle to convey three-dimensional concepts effectively. While 3D virtual models and virtual reality platforms offer a promising approach to teaching anatomy, their cost and specialized equipment pose limitations, especially in disadvantaged areas. A simpler alternative is to use virtual 3D models displayed on regular screens, but they lack immersion, realism, and stereoscopic vision. To address these challenges, we developed an affordable method utilizing smartphone-based 360° photogrammetry, virtual camera recording, and stereoscopic display (anaglyph or side-by-side technique). In this study, we assessed the feasibility of this method by subjecting it to various specimen types: osteological, soft organ, neuroanatomical, regional dissection, and a dedicated 3D-printed testing phantom. The results demonstrate that the 3D models obtained feature a complete mesh with a high level of detail and a realistic texture. Mesh and texture resolutions were estimated to be approximately 1 and 0.2 mm, respectively. Additionally, stereoscopic animations were both feasible and effective in enhancing depth perception. The simplicity and affordability of this method position it as a technique of choice for creating easily photorealistic anatomical models combined with stereoscopic depth visualization.

Development and Evaluation of a Learning-based Model for Real-time Haptic Texture Rendering.

Current Virtual Reality (VR) environments lack the haptic signals that humans experience during real-life interactions, such as the sensation of texture during lateral movement on a surface. Adding realistic haptic textures to VR environments requires a model that generalizes to variations of a user's interaction and to the wide variety of existing textures in the world. Current methodologies for haptic texture rendering exist, but they usually develop one model per texture, resulting in low scalability. We present a deep learning-based action-conditional model for haptic texture rendering and evaluate its perceptual performance in rendering realistic texture vibrations through a multi-part human user study. This model is unified over all materials and uses data from a vision-based tactile sensor (GelSight) to render the appropriate surface conditioned on the user's action in real-time. For rendering texture, we use a high-bandwidth vibrotactile transducer attached to a 3D Systems Touch device. The results of our user study shows that our learning-based method creates high-frequency texture renderings with comparable or better quality than state-of-the-art methods without the need to learn a separate model per texture. Furthermore, we show that the method is capable of rendering previously unseen textures using a single GelSight image of their surface.

Multicentric study on the reproducibility and robustness of PET-based radiomics features with a realistic activity painting phantom.

Previously, we developed an "activity painting" tool for PET image simulation; however, it could simulate heterogeneous patterns only in the air. We aimed to improve this phantom technique to simulate arbitrary lesions in a radioactive background to perform relevant multi-center radiomic analysis. We conducted measurements moving a 22Na point source in a 20-liter background volume filled with 5 kBq/mL activity with an adequately controlled robotic system to prevent the surge of the water. Three different lesion patterns were "activity-painted" in five PET/CT cameras, resulting in 8 different reconstructions. We calculated 46 radiomic indeces (RI) for each lesion and imaging setting, applying absolute and relative discretization. Reproducibility and reliability were determined by the inter-setting coefficient of variation (CV) and the intraclass correlation coefficient (ICC). Hypothesis tests were used to compare RI between lesions. By simulating precisely the same lesions, we confirmed that the reconstructed voxel size and the spatial resolution of different PET cameras were critical for higher order RI. Considering conventional RIs, the SUVpeak and SUVmean proved the most reliable (CV<10%). CVs above 25% are more common for higher order RIs, but we also found that low CVs do not necessarily imply robust parameters but often rather insensitive RIs. Based on the hypothesis test, most RIs could clearly distinguish between the various lesions using absolute resampling. ICC analysis also revealed that most RIs were more reproducible with absolute discretization. The activity painting method in a real radioactive environment proved suitable for precisely detecting the radiomic differences derived from the different camera settings and texture characteristics. We also found that inter-setting CV is not an appropriate metric for analyzing RI parameters' reliability and robustness. Although multicentric cohorts are increasingly common in radiomics analysis, realistic texture phantoms can provide indispensable information on the sensitivity of an RI and how an individual RI parameter measures the texture.

Realism of Tactile Texture Playback: A Combination of Stretch and Vibration.

This study investigates the effects of two stimulation modalities (stretch and vibration) on natural touch sensation on the volar forearm. The skin-textile interaction was implemented by scanning three textures across the left forearm. The resulting skin displacements were recorded by the digital image correlation technique to capture the information imparted by the textures. The texture recordings were used to create three playback modes (stretch, vibration, and both), which were reproduced on the right forearm. Two psychophysical experiments compared the texture scans to rendered texture playbacks. The first experiment used a matching task and found that to maximize perceptual realism, i.e., similarity to a physical reference, subjects preferred the rendered texture to have a playback intensity of 1X - 2X higher on DC components (stretch), and 1X - 3.5X higher on AC components (vibration), varying across textures. The second experiment elicited similarity ratings between the texture scans and playbacks and showed that a combination of stretch and vibration was required to create differentiated texture sensations. However, the intensity amplification and use of two stimuli were still insufficient to create fully realistic texture sensations. We conclude that mechanisms beyond single-site uniaxial stimuli are needed to reproduce realistic textural sensations.

Advancing sun glint correction in high-resolution marine UAV RGB imagery for coral reef monitoring

Sun glint presents a significant challenge in marine ecological remote sensing by obscuring valuable features of benthic communities, thus hindering accurate monitoring of these communities. More specifically, sun glint leads to inaccurate coral reef identification when it comes to utilize the high-resolution marine Unmanned Aerial Vehicle (UAV) imagery. Despite the availability of many physical models that address this problem using ocean remote sensing satellite images, there is still a lack of effective sun glint correction solutions for high-resolution RGB imagery obtained by consumer-grade UAVs in marine ecological monitoring, particularly at the centimeter level. In this study, an innovative sun glint correction pipeline combining a Foreground Attention-based Semantic Segmentation Network (FANet) with optical flow-based pixel propagation has been proposed, aiming to achieve accurate sun glint detection and restoration of real texture of coral reefs. To further understand the inner mode of sun glint segmentation FANet model, we conduct explainable analysis to ensure that it learns the informative sun glint features and makes reliable decisions. According to the comprehensive experimental evaluation using three datasets, our FANet model exhibits accurate and effective sun glint detection, achieving an IoU range of 70.86%–83.96%. Moreover, the sun-glint free images processed by our sun glint correction method have been employed for large-scale image stitching and shallow-water coral reef habitat semantic mapping. The results demonstrate that our sun glint correction approach successfully restores the texture of coral reefs and prepares a reliable foundation for subsequent downstream applications. We believe that our method as an open-source sun glint correction solution in high-resolution marine UAV RGB images is able to enhance the accuracy and efficiency of various marine ecological remote sensing tasks at a low-cost. The relevant code can be found in https://github.com/jyqinnn/Sun-glint-correction.

TwinTex: Geometry-Aware Texture Generation for Abstracted 3D Architectural Models

Coarse architectural models are often generated at scales ranging from individual buildings to scenes for downstream applications such as Digital Twin City, Metaverse, LODs, etc. Such piece-wise planar models can be abstracted as twins from 3D dense reconstructions. However, these models typically lack realistic texture relative to the real building or scene, making them unsuitable for vivid display or direct reference. In this paper, we present TwinTex , the first automatic texture mapping framework to generate a photorealistic texture for a piece-wise planar proxy. Our method addresses most challenges occurring in such twin texture generation. Specifically, for each primitive plane, we first select a small set of photos with greedy heuristics considering photometric quality, perspective quality and facade texture completeness. Then, different levels of line features (LoLs) are extracted from the set of selected photos to generate guidance for later steps. With LoLs, we employ optimization algorithms to align texture with geometry from local to global. Finally, we fine-tune a diffusion model with a multi-mask initialization component and a new dataset to inpaint the missing region. Experimental results on many buildings, indoor scenes and man-made objects of varying complexity demonstrate the generalization ability of our algorithm. Our approach surpasses state-of-the-art texture mapping methods in terms of high-fidelity quality and reaches a human-expert production level with much less effort.

TREE DIGITISATION FROM POINT CLOUDS WITH UNREAL ENGINE

Abstract. Trees are fundamental parts of urban areas and green urbanism. Although much effort is being put into the digitisation of urban areas, trees present great complexity and are usually replaced by predefined models. On the one hand, trees are elements composed of trunk, branches, and leaves, each with a completely different structure and geometry. On the other hand, the tree parts are closely related to each species. Therefore, in order to obtain a realistic digital urban environment, in 3D models such as CityGML or Metaverse, it is necessary that the trees correspond faithfully to reality. The aim of this work is to propose a method to digitise trees from Mobile Laser Scanning and Terrestrial Laser Scanning data. The process takes advantage of the differentiation between trunks and leaves for their segmentation by point cloud geometric features. Unreal Engine is then used to digitise each part. Trunk and branches are geometrically preserved. For dense canopy trees, predefined leaves according to the species are imported and the alpha shape of the crown is filled. For non-dense canopy trees, the canopy is imported and modified to fit the branches. The method was tested on four real case studies. The results show realistic trees, with correct trunk and foliage segmentation, but highly dependent on the life/canopy repositories. Unreal Engine was a very complete and useful tool for the digitisation of trees generating realistic textures and lighting options.