Appearance Representation Research Articles

Neural SSS: Lightweight Object Appearance Representation

AbstractWe present a method for capturing the BSSRDF (bidirectional scattering‐surface reflectance distribution function) of arbitrary geometry with a neural network. We demonstrate how a compact neural network can represent the full 8‐dimensional light transport within an object including heterogeneous scattering. We develop an efficient rendering method using importance sampling that is able to render complex translucent objects under arbitrary lighting. Our method can also leverage the common planar half‐space assumption, which allows it to represent one BSSRDF model that can be used across a variety of geometries. Our results demonstrate that we can render heterogeneous translucent objects under arbitrary lighting and obtain results that match the reference rendered using volumetric path tracing.

Santiago Jiménez - Unsupervised Learning of Disentangled and Interpretable Representations of Material Appearance

Unsupervised Learning of Disentangled and Interpretable Representations of Material Appearance

Unsupervised Learning of Disentangled and Interpretable Representations of Material Appearance

As humans, we have learned through experience how to interpret the visual appearance of materials in our environment, enabling us to predict the properties of an object just by looking at it for a few seconds. Although this seems like a straightforward task, the final appearance showed by a material is the result of a non-trivial interaction between confounding factors like geometry, illumination, or viewing angle, which we do not completely understand. Creating an algorithm able to disentangle the perceptual factors of material appearance present in an image, just like humans do, would be a breakthrough in several fields (e.g., in architecture, it could help to create prototypes that accurately reproduce how they will be perceived in the real life; following the inverse path of going from perceptual factors to images, it could be used in product design to intuitively define the perceptual features of a desired material, instead of delving into its mathematical definition). Here, we propose a learning-based algorithm capable of effectively disentangling certain perceptual features of images in an unsupervised way.

Identity-Guided Collaborative Learning for Cloth-Changing Person Reidentification.

Cloth-changing person reidentification (ReID) is a newly emerging research topic aimed at addressing the issues of large feature variations due to cloth-changing and pedestrian view/pose changes. Although significant progress has been achieved by introducing extra information (e.g., human contour sketching information, human body keypoints, and 3D human information), cloth-changing person ReID remains challenging because pedestrian appearance representations can change at any time. Moreover, human semantic information and pedestrian identity information are not fully explored. To solve these issues, we propose a novel identity-guided collaborative learning scheme (IGCL) for cloth-changing person ReID, where the human semantic is effectively utilized and the identity is unchangeable to guide collaborative learning. First, we design a novel clothing attention degradation stream to reasonably reduce the interference caused by clothing information where clothing attention and mid-level collaborative learning are employed. Second, we propose a human semantic attention and body jigsaw stream to highlight the human semantic information and simulate different poses of the same identity. In this way, the extraction features not only focus on human semantic information that is unrelated to the background but are also suitable for pedestrian pose variations. Moreover, a pedestrian identity enhancement stream is proposed to enhance the identity importance and extract more favorable identity robust features. Most importantly, all these streams are jointly explored in an end-to-end unified framework, and the identity is utilized to guide the optimization. Extensive experiments on six public clothing person ReID datasets (LaST, LTCC, PRCC, NKUP, Celeb-reID-light, and VC-Clothes) demonstrate the superiority of the IGCL method. It outperforms existing methods on multiple datasets, and the extracted features have stronger representation and discrimination ability and are weakly correlated with clothing.

Hierarchical mussel farm reconstruction from video with object tracking

ABSTRACT As the mussel farming industry expands, particularly in regions such as New Zealand, there is a growing need for advanced monitoring and management solutions to ensure sustainability and operational efficiency. The current reliance on manual infrequent observation of aquaculture structures limits farmers' ability to monitor them in real time. Addressing these challenges, large-scale 3D reconstruction provides a practical solution by facilitating the creation of replicable depictions of mussel farm scenes and floatation buoys derived from recorded video, supporting off-site evaluation and enabling precise decision making. This paper introduces a novel approach to enhance the visualisation and monitoring of mussel farm floatation through a hierarchical reconstruction process. In contrast to earlier studies, we focus on recovering not only the overall environment (background) but also the finer details of key elements such as buoys to create a comprehensive representation of mussel farm geometry and appearance. We propose to segment the scene into the background and granular object instances and reconstruct them separately in a multi-stage process. The initial 3D scene reconstruction is performed using the Structure-from-Motion (SfM) technique, leveraging video footage captured by a vessel-mounted camera. This coarse reconstruction serves as the foundation for subsequent fine-grained enhancements. To recover finer details, object tracking is applied and the trajectories obtained are then conjunct with geometry triangulation to determine the real-world positions of individual buoys. A multiple-scale denoising method, grounded in dominant direction correlation, is implemented to eliminate non-reliable tracking objects and reduce reconstruction artifacts, ensuring the accuracy of the final results. This hierarchical reconstruction approach contributes to the advancement of mussel farm management by offering a powerful technology for comprehensive visualisation, enabling farmers to make informed decisions based on a detailed understanding of the mussel farm's geometry and dynamics.

RGBT Tracking via Challenge-Based Appearance Disentanglement and Interaction.

RGB and thermal source data suffer from both shared and specific challenges, and how to explore and exploit them plays a critical role in representing the target appearance in RGBT tracking. In this paper, we propose a novel approach, which performs target appearance representation disentanglement and interaction via both modality-shared and modality-specific challenge attributes, for robust RGBT tracking. In particular, we disentangle the target appearance representations via five challenge-based branches with different structures according to their properties, including three parameter-shared branches to model modality-shared challenges and two parameter-independent branches to model modality-specific challenges. Considering the complementary advantages between modality-specific cues, we propose a guidance interaction module to transfer discriminative features from one modality to another one to enhance the discriminative ability of weak modality. Moreover, we design an aggregation interaction module to combine all challenge-based target representations, which could form more discriminative target representations and fit the challenge-agnostic tracking process. These challenge-based branches are able to model the target appearance under certain challenges so that the target representations can be learned by a few parameters even in the situation of insufficient training data. In addition, to relieve labor costs and avoid label ambiguity, we design a generation strategy to generate training data with different challenge attributes. Comprehensive experiments demonstrate the superiority of the proposed tracker against the state-of-the-art methods on four benchmark datasets.

The experience of comparing reconstructions of the external appearance made on skulls with existing lifetime photographs

The conducted research is aimed at correcting the method of graphic reconstruction of the appearance based on the skull. The method is widely used in both anthropology and criminology. The Forensic Center of Ministry of Internal Affairs of Russia was sent the skulls for which there were lifetime photographs. The restoration of the appearance in the form of graphic reconstructions was carried out by students who completed the course of Anthropological reconstruction at the RSUH, under the guidance of Prof., head of the Laboratory E.V. Veselovskaya. Strictly after the restoration of the appearance was completed, lifetime photos were provided, on the basis of which the degree of similarity of the reconstruction with the original was assessed. The purpose of the experiment is to correct the method of graphic reconstruction of the appearance based on the skull and supplement it with new details. Anthropologists carried out gender and age determination of individuals for a more accurate representation of appearance. Based on the results of comparing lifetime photographic images with the performed reconstructions, it can bed that their overall satisfactory level of compliance is sufficient for correct identification. However, a number of inaccuracies have been identified. As a result, areas of the face have been identified that require further refinement of the method of anthropological reconstruction. So, the thickness, and especially the shape, of the lips turned out to be an unresolved problem. The size of the iris is an important physiognomic feature. It is recommended to clarify the definition of the skull of the lifetime physiognomic height of the face, the width of the nose and the distance between the nasolabial folds.

FM-3DFR: Facial Manipulation-Based 3-D Face Reconstruction.

3-D Morphable model (3DMM) has widely benefited 3-D face-involved challenges given its parametric facial geometry and appearance representation. However, previous 3-D face reconstruction methods suffer from limited power in facial expression representation due to the unbalanced training data distribution and insufficient ground-truth 3-D shapes. In this article, we propose a novel framework to learn personalized shapes so that the reconstructed model well fits the corresponding face images. Specifically, we augment the dataset following several principles to balance the facial shape and expression distribution. A mesh editing method is presented as the expression synthesizer to generate more face images with various expressions. Besides, we improve the pose estimation accuracy by transferring the projection parameter into the Euler angles. Finally, a weighted sampling method is proposed to improve the robustness of the training process, where we define the offset between the base face model and the ground-truth face model as the sampling probability of each vertex. The experiments on several challenging benchmarks have demonstrated that our method achieves state-of-the-art performance.

고상안의 이순신 인물품평에 관한 고증연구

Yi Sun-sin is a national hero who defeated Japanese invasions (Imjin War) and a globally renowned great commander. Recently, there have been various issues raised in our society regarding the standard portrait of Yi Sun-sin. In our academic community, the issue of Yi Sun-sin's appearance has been discussed in connection with the portrait of Admiral Yi Sun-sin. However, there are records from the Joseon Dynasty that describe the appearance of Admiral Yi Sun-sin, but they contain conflicting information. Therefore, the question of how to represent the appearance of Admiral Yi Sun-sin in the future is a significant challenge. In this study, in this very context, we have compiled and compared records that mention Admiral Yi Sun-sin's appearance and have examined them in comparison with existing portraits, attempting to estimate Admiral Yi Sun-sin's appearance. Including Ryu Seong-ryong (柳成龍), figures like Kim Eung-nam, Yun Hyu, Hong Woo-won, Choi Yoo-hae, Kwak Yeol, and others recorded Admiral Yi Sun-sin as a courageous commander who possessed both literary and martial virtues. In contrast, Go Sang-an pointed out that Admiral Yi Sun-sin had shrunken lips and an appearance lacking plumpness, giving an unfortunate impression. Unlike other records, Go Sang-an specifically pointed out the shortcomings in Admiral Yi Sun-sin's lips and appearance, which also led some to consider it as a possible representation of Admiral Yi Sun-sin's actual appearance. In this context, this study focused on Go Sang-an's records regarding Admiral Yi Sun-sin's appearance and compared them with other records, analyzing the physiognomic significance contained in his mentions. To do so, a comparison was made with current portraits of Admiral Yi Sun-sin and his descendants. As a result, it became evident that the mention of Go Sang-an did not reveal the actual appearance of Admiral Yi Sun-sin but rather connected it to the sudden death of Admiral Yi Sun-sin in later years, expressing it through physiognomy. Furthermore, based on records regarding Admiral Yi Sun-sin's health status at that time, it was confirmed that the meaning behind Go Sang-an's character evaluation was not about flaws in Admiral Yi Sun-sin's appearance but rather that Go Sang-an made his observations and recordings during a period when Admiral Yi Sun-sin was extremely exhausted from warfare and suffered from typhoid fever, indicating very poor health. Therefore, Go Sang-an's mention should be correctly understood as an assessment made after observing Admiral Yi Sun-sin in a unique state of severe illness, later using physiognomy to describe his shrunken lips, emaciated appearance, and assessed him as having a short life and unfortunate appearance. In conclusion, the author believes that the mentions of Go Sang-an should be seen as providing evidence of Admiral Yi Sun-sin's dedication to defending the country and his deep concerns and anxiety, even as he neglected his health, and considers this interpretation to be valid.

Disentangling Structure and Appearance in ViT Feature Space

We present a method for semantically transferring the visual appearance of one natural image to another. Specifically, our goal is to generate an image in which objects in a source structure image are “painted” with the visual appearance of their semantically related objects in a target appearance image. To integrate semantic information into our framework, our key idea is to leverage a pre-trained and fixed Vision Transformer (ViT) model. Specifically, we derive novel disentangled representations of structure and appearance extracted from deep ViT features. We then establish an objective function that splices the desired structure and appearance representations, interweaving them together in the space of ViT features. Based on our objective function, we propose two frameworks of semantic appearance transfer – “Splice”, which works by training a generator on a single and arbitrary pair of structure-appearance images, and “SpliceNet”, a feed-forward real-time appearance transfer model trained on a dataset of images from a specific domain . Our frameworks do not involve adversarial training, nor do they require any additional input information such as semantic segmentation or correspondences. We demonstrate high-resolution results on a variety of in-the-wild image pairs, under significant variations in the number of objects, pose, and appearance. Code and supplementary material are available in our project page: splice-vit.github.io.

Consistent representation via contrastive learning for skin lesion diagnosis

BackgroundSkin lesions are a prevalent ailment, with melanoma emerging as a particularly perilous variant. Encouragingly, artificial intelligence displays promising potential in early detection, yet its integration within clinical contexts, particularly involving multi-modal data, presents challenges. While multi-modal approaches enhance diagnostic efficacy, the influence of modal bias is often disregarded. MethodsIn this investigation, a multi-modal feature learning technique termed “Contrast-based Consistent Representation Disentanglement” for dermatological diagnosis is introduced. This approach employs adversarial domain adaptation to disentangle features from distinct modalities, fostering a shared representation. Furthermore, a contrastive learning strategy is devised to incentivize the model to preserve uniformity in common lesion attributes across modalities. Emphasizing the learning of a uniform representation among models, this approach circumvents reliance on supplementary data. ResultsAssessment of the proposed technique on a 7-point criteria evaluation dataset yields an average accuracy of 76.1% for multi-classification tasks, surpassing researched state-of-the-art methods. The approach tackles modal bias, enabling the acquisition of a consistent representation of common lesion appearances across diverse modalities, which transcends modality boundaries. This study underscores the latent potential of multi-modal feature learning in dermatological diagnosis. ConclusionIn summation, a multi-modal feature learning strategy is posited for dermatological diagnosis. This approach outperforms other state-of-the-art methods, underscoring its capacity to enhance diagnostic precision for skin lesions.

Stochastic video normality network for abnormal event detection in surveillance videos

Video Anomaly Detection (VAD) aims to automatically identify unexpected spatial–temporal patterns to detect abnormal events in surveillance videos. Existing unsupervised VAD methods either use a single proxy task to represent the prototypical normal patterns, or split the video into static and dynamic parts and learn spatial and temporal normality separately, which cannot model the inherent uncertainty of motion. In this regard, we propose for the first time to learn video normality via stochastic motion representation. Specifically, the proposed Stochastic Video Normality (SVN) network learns the prototypical local appearance patterns via deterministic multi-task learning and global motion patterns in a non-deterministic manner. The stochastic representation module uses recurrent networks to model historical motion as a conditional Gaussian distribution and infers motion explicitly with the learned prior. Besides, a mask auto-encoder is introduced to explore the relationship between appearance and motion, boosting the model to learn spatial–temporal normality. Experimental results show that our proposed SVN network performs comparably to the state-of-the-art methods. Extensive analysis demonstrates the effectiveness of multi-task learning-based appearance representation and stochastic motion representation for unsupervised video anomaly detection.

Sensory tuning in neuronal movement commands

Movement control is critical for successful interaction with our environment. However, movement does not occur in complete isolation of sensation, and this is particularly true of eye movements. Here, we show that the neuronal eye movement commands emitted by the superior colliculus (SC), a structure classically associated with oculomotor control, encompass a robust visual sensory representation of eye movement targets. Thus, similar saccades toward different images are associated with different saccade-related "motor" bursts. Such sensory tuning in SC saccade motor commands appeared for all image manipulations that we tested, from simple visual features to real-life object images, and it was also strongest in the most motor neurons in the deeper collicular layers. Visual-feature discrimination performance in the motor commands was also stronger than in visual responses. Comparing SC motor command feature discrimination performance to that in the primary visual cortex during steady-state gaze fixation revealed that collicular motor bursts possess a reliable perisaccadic sensory representation of the peripheral saccade target's visual appearance, exactly when retinal input is expected to be most uncertain. Our results demonstrate that SC neuronal movement commands likely serve a fundamentally sensory function.

CCT-Unet: A U-Shaped Network Based on Convolution Coupled Transformer for Segmentation of Peripheral and Transition Zones in Prostate MRI.

The accurate segmentation of prostate region in magnetic resonance imaging (MRI) can provide reliable basis for artificially intelligent diagnosis of prostate cancer. Transformer-based models have been increasingly used in image analysis due to their ability to acquire long-term global contextual features. Although Transformer can provide feature representations of the overall appearance and contour representations at long distance, it does not perform well on small-scale datasets of prostate MRI due to its insensitivity to local variation such as the heterogeneity of the grayscale intensities in the peripheral zone and transition zone across patients; meanwhile, the convolutional neural network (CNN) could retain these local features well. Therefore, a robust prostate segmentation model that can aggregate the characteristics of CNN and Transformer is desired. In this work, a U-shaped network based on the convolution coupled Transformer is proposed for segmentation of peripheral and transition zones in prostate MRI, named the convolution coupled Transformer U-Net (CCT-Unet). The convolutional embedding block is first designed for encoding high-resolution input to retain the edge detail of the image. Then the convolution coupled Transformer block is proposed to enhance the ability of local feature extraction and capture long-term correlation that encompass anatomical information. The feature conversion module is also proposed to alleviate the semantic gap in the process of jumping connection. Extensive experiments have been conducted to compare our CCT-Unet with several state-of-the-art methods on both the ProstateX open dataset and the self-bulit Huashan dataset, and the results have consistently shown the accuracy and robustness of our CCT-Unet in MRI prostate segmentation.

TextStyleBrush: Transfer of Text Aesthetics From a Single Example.

We present a novel approach for disentangling the content of a text image from all aspects of its appearance. The appearance representation we derive can then be applied to new content, for one-shot transfer of the source style to new content. We learn this disentanglement in a self-supervised manner. Our method processes entire word boxes, without requiring segmentation of text from background, per-character processing, or making assumptions on string lengths. We show results in different text domains which were previously handled by specialized methods, e.g., scene text, handwritten text. To these ends, we make a number of technical contributions: (1) We disentangle the style and content of a textual image into a non-parametric, fixed-dimensional vector. (2) We propose a novel approach inspired by StyleGAN but conditioned over the example style at different resolution and content. (3) We present novel self-supervised training criteria which preserve both source style and target content using a pre-trained font classifier and text recognizer. Finally, (4) we also introduce Imgur5K, a new challenging dataset for handwritten word images. We offer numerous qualitative photo-realistic results of our method. We further show that our method surpasses previous work in quantitative tests on scene text and handwriting datasets, as well as in a user study.

NeRFFaceLighting: Implicit and Disentangled Face Lighting Representation Leveraging Generative Prior in Neural Radiance Fields

3D-aware portrait lighting control is an emerging and promising domain, thanks to the recent advance of generative adversarial networks and neural radiance fields. Existing solutions typically try to decouple the lighting from the geometry and appearance for disentangled control with an explicit lighting representation (e.g., Lambertian or Phong). However, they either are limited to a constrained lighting condition (e.g., directional light) or demand a tricky-to-fetch dataset as supervision for the intrinsic compositions (e.g., the albedo). We propose NeRFFaceLighting to explore an implicit representation for portrait lighting based on the pretrained tri-plane representation to address the above limitations. We approach this disentangled lighting-control problem by distilling the shading from the original fused representation of both appearance and lighting (i.e., one tri-plane) to their disentangled representations (i.e., two tri-planes) with the conditional discriminator to supervise the lighting effects. We further carefully design the regularization to reduce the ambiguity of such decomposition and enhance the ability of generalization to unseen lighting conditions. Moreover, our method can be extended to enable 3D-aware real portrait relighting. Through extensive quantitative and qualitative evaluations, we demonstrate the superior 3D-aware lighting control ability of our model compared to alternative and existing solutions.

Re-cognizing the new self: The neurocognitive plasticity of self-processing following facial transplantation

The face is a defining feature of our individuality, crucial for our social interactions. But what happens when the face connected to the self is radically altered or replaced? We address the plasticity of self-face recognition in the context of facial transplantation. While the acquisition of a new face following facial transplantation is a medical fact, the experience of a new identity is an unexplored psychological outcome. We traced the changes in self-face recognition before and after facial transplantation to understand if and how the transplanted face gradually comes to be perceived and recognized as the recipient's own new face. Neurobehavioral evidence documents a strong representation of the pre-injury appearance pre-operatively, while following the transplantation, the recipient incorporates the new face into his self-identity. The acquisition of this new facial identity is supported by neural activity in medial frontal regions that are considered to integrate psychological and perceptual aspects of the self.

Features of the perceived age among representatives of ethno-cultural groups with different appearance

The authors focus on the phenomenon of perceived age and the ethno-cultural factor of its construction. The subject of the study is: the perceived age of the objects of perception, attitude to them, self-assessment of age and attitude to their appearance of the subjects of perception (women, representatives of Slavic and Caucasian peoples). The study was carried out within the framework of a socio-psychological approach to the appearance and model of ethnolucism by V.A. Labunskaya. The main method of research was the procedure of "Photo-visual representations of the external appearance" by T.A. Vorontsova. The scientific novelty of the study was the identification of the influence of the ethno-cultural factor on the construction of the age of an unfamiliar person and the assessment of their age. The main conclusions of the study: 1) the belonging of the subjects of perception to an ethnocultural group associated with the type of appearance affects the self-esteem of the perceived age, the planned life expectancy, the attitude towards unfamiliar others; 2) the age attributed to the perceived stranger is associated with the attitude towards him regardless of the ethno-cultural group; 3) the structure of the perception of the age of an unfamiliar other is identical from the point of view of the central and peripheral categories and does not depend on the ethno-cultural factor.

Online multiple object tracker with enhanced features

The online multiobject tracking (MOT), which integrates object detection and tracking into a single network, has made breakthroughs in recent years. However, most online trackers have a more monotonous prediction of the tracking offset in two consecutive frames, which may not be reliable when facing extreme situations such as occlusion and object deformation. Once the tracking offset of an object is biased, its corresponding tracklet will no longer maintain temporal consistency, which will seriously affect the tracking performance. In this paper, we propose a new online multiple object tracker with feature enhancement mechanism, namely En-Tracker. In En-Tracker, a multibranch kinematic analysis network (MKANet) is designed to address the above problems. MKANet estimates the pixel offset and instance offset of the object in parallel based on imitating the human thinking to joint position and appearance representations. Note that these two types of offsets compensate and facilitate each other to effectively deal with some extreme scenarios. In addition, we propose a kinematic-assisted feature synthesis enhancement (KFSE) module, which has a more comprehensive enhancement mechanism. Specifically, KFSE propagates previous tracking information to the current frame based on kinematic trend analysis while enhancing the characterization of detection features and appearance embeddings, which not only assists in object detection but also ensures the uniqueness of the appearance embeddings. Extensive experiments on MOT16 and MOT17 verify the effectiveness and advantage of our model.

Dual Stream Conditional Generative Adversarial Network Fusion for Video Abnormal Behavior Detection

Deep learning has been successfully applied to video anomaly detection. However, the way that deep network learn spatio-temporal features autonomously will ignore the specificity of different pattern features. Therefore, this paper focuses on how to efficiently learn deep appearance feature, introduces the idea of learning appearance information by predicting future frame, and proposes dual stream conditional generative adversarial network fusion for video abnormal behavior detection. The video frame and its corresponding optical flow image are transferred to the conditional generative adversarial network to learn the motion feature representation. In addition, inputting the video frame and its corresponding future frame to the network to generate the appearance representation complementary to motion feature. The model is only trained with normal events, therefore it is not able to generate abnormal events accurately. During the test, for the foreground moving targets, the images generated by the model are compared with the ground truth to obtain a two-stream anomaly probability distribution model based on the mean square error used to achieve the purpose of region anomaly detection. Experiments on the public datasets show that the proposed method can effectively detect and locate abnormal behaviors in the video.