Scene Content Research Articles

High-resolution fully-polarimetric synthetic aperture radar dataset

Abstract Objectives Fully-polarimetric synthetic aperture radar (PolSAR) data contain a rich body of elementary scattering physics information that is critically valuable for a broad range of applications and scientific purposes. However, there is a lack of available high-resolution (< 0.3048-m) data available for PolSAR phenomenology research. This article introduces a high-resolution PolSAR data set collected and provided by Sandia National Laboratories (SNL). The data sets were collected to support studying high-resolution scattering physics from different types of clutter and applications such as polarimetric-based terrain classification. Data description The data set presented consists of 39 200 m × 200 m complex-valued PolSAR image sets with 0.12-m (~ 4.7-inch) resolution and is the result of two different collection campaigns, one campaign in 2013 and the other in 2015. The data sets were collected with SNL’s airborne fully-polarimetric X-band (9.6 GHz center frequency) radar system and contains a variety of natural and cultural clutter environments with scene content ranging from agricultural fields to urbanized areas collected throughout the greater Albuquerque, New Mexico, U.S.A. region.

𝒢‐Style: Stylized Gaussian Splatting

AbstractWe introduce 𝒢‐Style, a novel algorithm designed to transfer the style of an image onto a 3D scene represented using Gaussian Splatting. Gaussian Splatting is a powerful 3D representation for novel view synthesis, as—compared to other approaches based on Neural Radiance Fields—it provides fast scene renderings and user control over the scene. Recent pre‐prints have demonstrated that the style of Gaussian Splatting scenes can be modified using an image exemplar. However, since the scene geometry remains fixed during the stylization process, current solutions fall short of producing satisfactory results. Our algorithm aims to address these limitations by following a three‐step process: In a pre‐processing step, we remove undesirable Gaussians with large projection areas or highly elongated shapes. Subsequently, we combine several losses carefully designed to preserve different scales of the style in the image, while maintaining as much as possible the integrity of the original scene content. During the stylization process and following the original design of Gaussian Splatting, we split Gaussians where additional detail is necessary within our scene by tracking the gradient of the stylized color. Our experiments demonstrate that 𝒢‐Style generates high‐quality stylizations within just a few minutes, outperforming existing methods both qualitatively and quantitatively.

VPRF: Visual Perceptual Radiance Fields for Foveated Image Synthesis.

Neural radiance fields (NeRF) has achieved revolutionary breakthrough in the novel view synthesis task for complex 3D scenes. However, this new paradigm struggles to meet the requirements for real-time rendering and high perceptual quality in virtual reality. In this paper, we propose VPRF, a novel visual perceptual based radiance fields representation method, which for the first time integrates the visual acuity and contrast sensitivity models of human visual system (HVS) into the radiance field rendering framework. Initially, we encode both the appearance and visual sensitivity information of the scene into our radiance field representation. Then, we propose a visual perceptual sampling strategy, allocating computational resources according to the HVS sensitivity of different regions. Finally, we propose a sampling weight-constrained training scheme to ensure the effectiveness of our sampling strategy and improve the representation of the radiance field based on the scene content. Experimental results demonstrate that our method renders more efficiently, with higher PSNR and SSIM in the foveal and salient regions compared to the state-of-the-art FoV-NeRF. The results of the user study confirm that our rendering results exhibit high-fidelity visual perception.

Spatial Contextual Information Modulates Affordance Processing and Early Electrophysiological Markers of Scene Perception.

Scene perception allows humans to extract information from their environment and plan navigation efficiently. The automatic extraction of potential paths in a scene, also referred to as navigational affordances, is supported by scene-selective regions (SSRs) that enable efficient human navigation. Recent evidence suggests that the activity of these SSRs can be influenced by information from adjacent spatial memory areas. However, it remains unexplored how these contextual information could influence the extraction of bottom-up information, such as navigational affordances, from a scene and the underlying neural dynamics. Therefore, we analyzed ERPs in 26 young adults performing scene and spatial memory tasks in artificially generated rooms with varying numbers and locations of available doorways. We found that increasing the number of navigational affordances only impaired performance in the spatial memory task. ERP results showed a similar pattern of activity for both tasks, but with increased P2 amplitude in the spatial memory task compared with the scene memory. Finally, we reported no modulation of the P2 component by the number of affordances in either task. This modulation of early markers of visual processing suggests that the dynamics of SSR activity are influenced by a priori knowledge, with increased amplitude when participants have more contextual information about the perceived scene. Overall, our results suggest that prior spatial knowledge about the scene, such as the location of a goal, modulates early cortical activity associated with SSRs, and that this information may interact with bottom-up processing of scene content, such as navigational affordances.

Supporting motion-capture acting with collaborative Mixed Reality

Technologies such as chroma-key, LED walls, motion capture (mocap), 3D visual storyboards, and simulcams are revolutionizing how films featuring visual effects are produced. Despite their popularity, these technologies have introduced new challenges for actors. An increased workload is faced when digital characters are animated via mocap, since actors are requested to use their imagination to envision what characters see and do on set. This work investigates how Mixed Reality (MR) technology can support actors during mocap sessions by presenting a collaborative MR system named CoMR-MoCap, which allows actors to rehearse scenes by overlaying digital contents onto the real set. Using a Video See-Through Head Mounted Display (VST-HMD), actors can see digital representations of performers in mocap suits and digital scene contents in real time. The system supports collaboration, enabling multiple actors to wear both mocap suits to animate digital characters and VST-HMDs to visualize the digital contents. A user study involving 24 participants compared CoMR-MoCap to the traditional method using physical props and visual cues. The results showed that CoMR-MoCap significantly improved actors’ ability to position themselves and direct their gaze, and it offered advantages in terms of usability, spatial and social presence, embodiment, and perceived effectiveness over the traditional method.

Bridging Implicit and Explicit Geometric Transformation for Single-Image View Synthesis.

Creating novel views from a single image has achieved tremendous strides with advanced autoregressive models, as unseen regions have to be inferred from the visible scene contents. Although recent methods generate high-quality novel views, synthesizing with only one explicit or implicit 3D geometry has a trade-off between two objectives that we call the "seesaw" problem: 1) preserving reprojected contents and 2) completing realistic out-of-view regions. Also, autoregressive models require a considerable computational cost. In this paper, we propose a single-image view synthesis framework for mitigating the seesaw problem while utilizing an efficient non-autoregressive model. Motivated by the characteristics that explicit methods well preserve reprojected pixels and implicit methods complete realistic out-of-view regions, we introduce a loss function to complement two renderers. Our loss function promotes that explicit features improve the reprojected area of implicit features and implicit features improve the out-of-view area of explicit features. With the proposed architecture and loss function, we can alleviate the seesaw problem, outperforming autoregressive-based state-of-the-art methods and generating an image ≈ 100 times faster. We validate the efficiency and effectiveness of our method with experiments on RealEstate10 K and ACID datasets.

Assessment and Application of Digital Museum Visitors' Emotional Experience Based on Virtual Reality Technology and Emotion Recognition Algorithm

Abstract Each person will have a different emotional experience for the same scene or exhibit, which reduces the accuracy of emotional recognition and leads to the complexity of the evaluation of visitors' emotional experience in digital museums. In order to improve the user experience of digital museums, the evaluation and optimization methods of visitors' emotional experience of digital museums based on virtual reality (VR) technology and emotion recognition algorithm are studied. A spectrogram is generated according to the voice sent by tourists when they visit the digital museum, which is based on the CSWNet_CRNN emotion recognition depth learning model input, which evaluates tourists' emotional experience results and draws their emotional responses to digital museums. The visual and auditory features of the digital museum scene with positive emotional experience are extracted, respectively. Using VR technology, the extracted features are applied to each link of the digital museum scene content design, optimizing the digital museum VR scene and improving the digital museum experience. The experiment shows that the tourist emotion recognition accuracy of this method is high, and the emotion recognition accuracy of 300 random tourists can reach 100%. In terms of generating new scenes, the feature extraction results of this scene are consistent with the feature estimation of positive emotions by ordinary people. The use of extracted features to optimize the digital museum scene has better realism and detail accuracy, which can be favored by most people and promotes the sustainable development of digital museums.

Metacognitive Awareness and the Subjective Experience of Remembering in Aphantasia.

Individuals with aphantasia, a nonclinical condition typically characterized by mental imagery deficits, often report reduced episodic memory. However, findings have hitherto rested largely on subjective self-reports, with few studies experimentally investigating both objective and subjective aspects of episodic memory in aphantasia. In this study, we tested both aspects of remembering in aphantasic individuals using a custom 3-D object and spatial memory task that manipulated visuospatial perspective, which is considered to be a key factor determining the subjective experience of remembering. Objective and subjective measures of memory performance were taken for both object and spatial memory features under different perspective conditions. Surprisingly, aphantasic participants were found to be unimpaired on all objective memory measures, including those for object memory features, despite reporting weaker overall mental imagery experience and lower subjective vividness ratings on the memory task. These results add to newly emerging evidence that aphantasia is a heterogenous condition, where some aphantasic individuals may lack metacognitive awareness of mental imagery rather than mental imagery itself. In addition, we found that both participant groups remembered object memory features with greater precision when encoded and retrieved in the first person versus third person, suggesting a first-person perspective might facilitate subjective memory reliving by enhancing the representational quality of scene contents.

Subjective Affective Responses to Natural Scenes Require Understanding, Not Spatial Frequency Bands.

It is debated whether emotional processing and response depend on semantic identification or are preferentially tied to specific information in natural scenes, such as global features or local details. The present study aimed to further examine the relationship between scene understanding and affective response while manipulating visual content. To this end, we presented affective and neutral natural scenes which were progressively band-filtered to contain global features (low spatial frequencies) or local details (high spatial frequencies) and assessed both affective response and scene understanding. We observed that, if scene content was correctly reported, subjective ratings of arousal and valence were modulated by the affective content of the scene, and this modulation was similar across spatial frequency bands. On the other hand, no affective modulation of subjective ratings was observed if picture content was not correctly reported. The present results indicate that subjective affective response requires content understanding, and it is not tied to a specific spatial frequency range.

Representations of imaginary scenes and their properties in cortical alpha activity

Imagining natural scenes enables us to engage with a myriad of simulated environments. How do our brains generate such complex mental images? Recent research suggests that cortical alpha activity carries information about individual objects during visual imagery. However, it remains unclear if more complex imagined contents such as natural scenes are similarly represented in alpha activity. Here, we answer this question by decoding the contents of imagined scenes from rhythmic cortical activity patterns. In an EEG experiment, participants imagined natural scenes based on detailed written descriptions, which conveyed four complementary scene properties: openness, naturalness, clutter level and brightness. By conducting classification analyses on EEG power patterns across neural frequencies, we were able to decode both individual imagined scenes as well as their properties from the alpha band, showing that also the contents of complex visual images are represented in alpha rhythms. A cross-classification analysis between alpha power patterns during the imagery task and during a perception task, in which participants were presented images of the described scenes, showed that scene representations in the alpha band are partly shared between imagery and late stages of perception. This suggests that alpha activity mediates the top-down re-activation of scene-related visual contents during imagery.

External task switches activate default mode regions without enhanced processing of the surrounding scene

Abstract Default mode network (DMN) activity, measured with fMRI, typically increases during internally directed thought, and decreases during tasks that demand externally focused attention. However, Crittenden et al. (2015) and Smith et al. (2018) reported increased DMN activity during demanding external task switches between different cognitive domains, compared to within-domain switches and task repeats. This finding is hard to reconcile with many dominant views of DMN function. Here, we aimed to replicate this DMN task-switch effect in a similar paradigm and test whether it reflects increased representation of broader context, specifically of a scene presented behind the focal task. In the Core DMN, we found significant activity for all task switches, compared to task repeats, and stronger activity for switches between rest and task. Although the content of the background scene was attended, recalled, and neurally decodable, there was no evidence that this differed by switch type. Therefore, external task switches activated DMN without enhanced processing of the surrounding background. Surprisingly, DMN activity at within-domain switches was no less than at between-domain switches. We suggest that modulation of DMN activity by task switches reflects a shift in the current cognitive model and depends on the overall complexity of that model.

Learning Content-Enhanced Mask Transformer for Domain Generalized Urban-Scene Segmentation

Domain-generalized urban-scene semantic segmentation (USSS) aims to learn generalized semantic predictions across diverse urban-scene styles. Unlike generic domain gap challenges, USSS is unique in that the semantic categories are often similar in different urban scenes, while the styles can vary significantly due to changes in urban landscapes, weather conditions, lighting, and other factors. Existing approaches typically rely on convolutional neural networks (CNNs) to learn the content of urban scenes. In this paper, we propose a Content-enhanced Mask TransFormer (CMFormer) for domain-generalized USSS. The main idea is to enhance the focus of the fundamental component, the mask attention mechanism, in Transformer segmentation models on content information. We have observed through empirical analysis that a mask representation effectively captures pixel segments, albeit with reduced robustness to style variations. Conversely, its lower-resolution counterpart exhibits greater ability to accommodate style variations, while being less proficient in representing pixel segments. To harness the synergistic attributes of these two approaches, we introduce a novel content-enhanced mask attention mechanism. It learns mask queries from both the image feature and its down-sampled counterpart, aiming to simultaneously encapsulate the content and address stylistic variations. These features are fused into a Transformer decoder and integrated into a multi-resolution content-enhanced mask attention learning scheme. Extensive experiments conducted on various domain-generalized urban-scene segmentation datasets demonstrate that the proposed CMFormer significantly outperforms existing CNN-based methods by up to 14.0% mIoU and the contemporary HGFormer by up to 1.7% mIoU. The source code is publicly available at https://github.com/BiQiWHU/CMFormer.

A survey on 3D object detection in real time for autonomous driving.

This survey reviews advances in 3D object detection approaches for autonomous driving. A brief introduction to 2D object detection is first discussed and drawbacks of the existing methodologies are identified for highly dynamic environments. Subsequently, this paper reviews the state-of-the-art 3D object detection techniques that utilizes monocular and stereo vision for reliable detection in urban settings. Based on depth inference basis, learning schemes, and internal representation, this work presents a method taxonomy of three classes: model-based and geometrically constrained approaches, end-to-end learning methodologies, and hybrid methods. There is highlighted segment for current trend of multi-view detectors as end-to-end methods due to their boosted robustness. Detectors from the last two kinds were specially selected to exploit the autonomous driving context in terms of geometry, scene content and instances distribution. To prove the effectiveness of each method, 3D object detection datasets for autonomous vehicles are described with their unique features, e. g., varying weather conditions, multi-modality, multi camera perspective and their respective metrics associated to different difficulty categories. In addition, we included multi-modal visual datasets, i. e., V2X that may tackle the problems of single-view occlusion. Finally, the current research trends in object detection are summarized, followed by a discussion on possible scope for future research in this domain.

SEMANTICS OF PERFECTIVE AND IMPERFECTIVE ASPECT

This article examines the concept of state aspect and holistic endpoint, action and state verbs are incomplete rather than complete, they do not encode any endpoint, and the verbs to do and succeed are complete because they are endpoints. Temporary instruments also serve to preserve the content of individual scenes, events as a whole, and completeness.

Neural Subspaces for Light Fields.

We introduce a framework for compactly representing light field content with the novel concept of neural subspaces. While the recently proposed neural light field representation achieves great compression results by encoding a light field into a single neural network, the unified design is not optimized for the composite structures exhibited in light fields. Moreover, encoding every part of the light field into one network is not ideal for applications that require rapid transmission and decoding. We recognize this problem's connection to subspace learning. We present a method that uses several small neural networks, specializing in learning the neural subspace for a particular light field segment. Moreover, we propose an adaptive weight sharing strategy among those small networks, improving parameter efficiency. In effect, this strategy enables a concerted way to track the similarity among nearby neural subspaces by leveraging the layered structure of neural networks. Furthermore, we develop a soft-classification technique to enhance the color prediction accuracy of neural representations. Our experimental results show that our method better reconstructs the light field than previous methods on various light field scenes. We further demonstrate its successful deployment on encoding light fields with irregular viewpoint layout and dynamic scene content.

Impact of focus cue presentation on perceived realism of 3-D scene structure: Implications for scene perception and for display technology.

Stereoscopic imagery often aims to evoke three-dimensional (3-D) percepts that are accurate and realistic-looking. The "gap" between 3-D imagery and real scenes is small, but focus cues typically remain incorrect because images are displayed on a single focal plane. Research has concentrated on the resulting vergence-accommodation conflicts. Yet, incorrect focus cues may also affect the appearance of 3-D imagery. We investigated whether incorrect focus cues reduce perceived realism of 3-D structure ("depth realism"). Experiment1 used a multiple-focal-planes display to compare depth realism with correct focus cues vs. conventional stereo presentation. The stimuli were random-dot stereograms, which isolated the role of focus cues. Depth realism was consistently lower with incorrect focus cues, providing proof-of-principle evidence that they contribute to perceptual realism. Experiments2 and3 examined whether focus cues play a similar role with realistic objects, presented with an almost complete set of visual cues using a high-resolution, high-dynamic-range multiple-focal-planes display. We also examined the efficacy of approximating correct focus cues via gaze-contingent depth-of-field rendering. Improvements in depth realism with correct focus cues were less clear in more realistic scenes, indicating that the role of focus cues in depth realism depends on scene content. Rendering-based approaches, if anything, reduced depth realism, which we attribute to their inability to present higher-order aspects of blur correctly. Our findings suggest future general 3-D display solutions may need to present focus cues correctly to maximise perceptual realism.

DRLIE: Flexible Low-Light Image Enhancement via Disentangled Representations.

Low-light image enhancement (LIME) aims to convert images with unsatisfied lighting into desired ones. Different from existing methods that manipulate illumination in uncontrollable manners, we propose a flexible framework to take user-specified guide images as references to improve the practicability. To achieve the goal, this article models an image as the combination of two components, that is, content and exposure attribute, from an information decoupling perspective. Specifically, we first adopt a content encoder and an attribute encoder to disentangle the two components. Then, we combine the scene content information of the low-light image with the exposure attribute of the guide image to reconstruct the enhanced image through a generator. Extensive experiments on public datasets demonstrate the superiority of our approach over state-of-the-art alternatives. Particularly, the proposed method allows users to enhance images according to their preferences, by providing specific guide images. Our source code and the pretrained model are available at https://github.com/Linfeng-Tang/DRLIE.

View synthesis with multiplane images from computationally generated RGB-D light fields

Image based view synthesis using deep neural networks provide novel scene views from a set of captured single or multiple images. Multiplane images (MPI) represent scene content as set of RGBα planes within a reference view frustum and render novel views by projecting the content into the target viewpoints. Image based view synthesis with multiple images is very popularly deployed in various areas because it effectively represents geometric uncertainty in ambiguous regions and can convincingly simulate non-Lambertian effects. However, previous image based view synthesis approaches suffer from interpolating and extrapolating information in pixels or ray spaces to generate seamless novel views without occlusion. To effectively improve visual performance for view interpolation and extrapolation, this paper proposes a novel view synthesis with MPI images. From a monocular RGB image, light field images are computationally generated, the proposed depth map guided deep network produces robust MPI using the light field images and their corresponding depth images, and the MPI network embedded with depth attention blocks forces semantic and geometric information to be uniformly distributed and divided among layers. The proposed approach achieves 3.5% and 4.02% improvements in SSIM and PSNR values, comparing to the SOTA approaches. Qualitative analysis on benchmark dataset also verifies the robustness of the proposed approach.

Scene-aware Foveated Neural Radiance Fields.

Foveated rendering provides an idea for improving the image synthesis performance of neural radiance fields (NeRF) methods. In this paper, we propose a scene-aware foveated neural radiance fields method to synthesize high-quality foveated images in complex VR scenes at high frame rates. Firstly, we construct a multi-ellipsoidal neural representation to enhance the neural radiance field's representation capability in salient regions of complex VR scenes based on the scene content. Then, we introduce a uniform sampling based foveated neural radiance field framework to improve the foveated image synthesis performance with one-pass color inference, and improve the synthesis quality by leveraging the foveated scene-aware objective function. Our method synthesizes high-quality binocular foveated images at the average frame rate of 66 frames per second (FPS) in complex scenes with high occlusion, intricate textures, and sophisticated geometries. Compared with the state-of-the-art foveated NeRF method, our method achieves significantly higher synthesis quality in both the foveal and peripheral regions with 1.41-1.46× speedup. We also conduct a user study to prove that the perceived quality of our method has a high visual similarity with the ground truth.

LTM-NeRF: Embedding 3D Local Tone Mapping in HDR Neural Radiance Field.

Recent advances in Neural Radiance Fields (NeRF) have provided a new geometric primitive for novel view synthesis. High Dynamic Range NeRF (HDR NeRF) can render novel views with a higher dynamic range. However, effectively displaying the scene contents of HDR NeRF on diverse devices with limited dynamic range poses a significant challenge. To address this, we present LTM-NeRF, a method designed to recover HDR NeRF and support 3D local tone mapping. LTM-NeRF allows for the synthesis of HDR views, tone-mapped views, and LDR views under different exposure settings, using only the multi-view multi-exposure LDR inputs for supervision. Specifically, we propose a differentiable Camera Response Function (CRF) module for HDR NeRF reconstruction, globally mapping the scene's HDR radiance to LDR pixels. Moreover, we introduce a Neural Exposure Field (NeEF) to represent the spatially varying exposure time of an HDR NeRF to achieve 3D local tone mapping, for compatibility with various displays. Comprehensive experiments demonstrate that our method can not only synthesize HDR views and exposure-varying LDR views accurately but also render locally tone-mapped views naturally. For video results, please view our supplemental materials or visit the project page at: https://xhuangcv.github.io/LTM-NeRF/.