Scene Layout Research Articles

Research on the Creative Principles of Chinese Thriller Films

The Chinese thriller film, as a specific genre, exhibits unique principles and characteristics in its creation, which is also significant in terms of cultural heritage and creative expression. Nevertheless, the current systematic investigation of the research principles and creative connotations of the sample remains inadequate. The objective of this paper is to analyze the research principles of Chinese thriller-horror films and identify the sources and evolution of the narrative materials underlying them. The analysis employs both the literature analysis method and the case study method. The former is used to examine the expression of the combination of thriller imagery and traditional Chinese cultural symbols in the film, while the latter is employed to explore the cultural elements belonging to the Chinese nation's tradition. Additionally, the analysis considers the distinctive characteristics of the film in terms of artistic scene layout, sound, and color, as well as the significance and role of such films. By examining the fundamental principles of Chinese thriller film-making, it is possible to gain a deeper understanding of the distinctive appeal and artistic nuances of Chinese thrillers, offering insights that can inform and inspire the creation of Chinese films.

Robust Shape Fitting for 3D Scene Abstraction.

Humans perceive and construct the world as an arrangement of simple parametric models. In particular, we can often describe man-made environments using volumetric primitives such as cuboids or cylinders. Inferring these primitives is important for attaining high-level, abstract scene descriptions. Previous approaches for primitive-based abstraction estimate shape parameters directly and are only able to reproduce simple objects. In contrast, we propose a robust estimator for primitive fitting, which meaningfully abstracts complex real-world environments using cuboids. A RANSAC estimator guided by a neural network fits these primitives to a depth map. We condition the network on previously detected parts of the scene, parsing it one-by-one. To obtain cuboids from single RGB images, we additionally optimise a depth estimation CNN end-to-end. Naively minimising point-to-primitive distances leads to large or spurious cuboids occluding parts of the scene. We thus propose an improved occlusion-aware distance metric correctly handling opaque scenes. Furthermore, we present a neural network based cuboid solver which provides more parsimonious scene abstractions while also reducing inference time. The proposed algorithm does not require labour-intensive labels, such as cuboid annotations, for training. Results on the NYU Depth v2 dataset demonstrate that the proposed algorithm successfully abstracts cluttered real-world 3D scene layouts.

Inter-object discriminative graph modeling for indoor scene recognition

Variable scene layouts and coexisting objects across scenes make indoor scene recognition still a challenging task. Leveraging object information within scenes to enhance the distinguishability of feature representations has emerged as a key approach in this domain. Currently, most object-assisted methods use a separate branch to process object information, combining object and scene features heuristically. However, few of them pay attention to interpretably handling the hidden discriminative knowledge within object information. In this paper, we propose to leverage discriminative object knowledge to enhance scene feature representations. Initially, we capture the object-scene discriminative relationships from a probabilistic perspective, which are transformed into an Inter-Object Discriminative Prototype (IODP). Given the abundant prior knowledge from IODP, we subsequently construct a Discriminative Graph Network (DGN), in which pixel-level scene features are defined as nodes and the discriminative relationships between node features are encoded as edges. DGN aims to incorporate inter-object discriminative knowledge into the image representation through graph convolution and mapping operations (GCN). With the proposed IODP and DGN, we obtain state-of-the-art results on several widely used scene datasets, demonstrating the effectiveness of the proposed approach.

The visual statistical learning overcomes scene dissimilarity through an independent clustering process.

Contextual cueing is a phenomenon of visual statistical learning observed in visual search tasks. Previous research has found that the degree of deviation of items from its centroid, known as variability, determines the extent of generalization for that repeated scene. Introducing variability increases dissimilarity between multiple occurrences of the same repeated layout significantly. However, current theories do not explain the mechanisms that help to overcome this dissimilarity during contextual cue learning. We propose that the cognitive system initially abstracts specific scenes into scene layouts through an automatic clustering unrelated to specific repeated scenes, and subsequently uses these abstracted scene layouts for contextual cue learning. Experiment 1 indicates that introducing greater variability in search scenes leads to a hindering in the contextual cue learning. Experiment 2 further establishes that conducting extensive visual searches involving spatial variability in entirely novel scenes facilitates subsequent contextual cue learning involving corresponding scene variability, confirming that learning clustering knowledge precedes the contextual cue learning and is independent of specific repeated scenes. Overall, this study demonstrates the existence of multiple levels of learning in visual statistical learning, where item-level learning can serve as material for layout-level learning, and the generalization reflects the constraining role of item-level knowledge on layout-level knowledge.

Learning a Memory-Enhanced Multi-Stage Goal-Driven Network for Egocentric Trajectory Prediction.

We propose a memory-enhanced multi-stage goal-driven network (ME-MGNet) for egocentric trajectory prediction in dynamic scenes. Our key idea is to build a scene layout memory inspired by human perception in order to transfer knowledge from prior experiences to the current scenario in a top-down manner. Specifically, given a test scene, we first perform scene-level matching based on our scene layout memory to retrieve trajectories from visually similar scenes in the training data. This is followed by trajectory-level matching and memory filtering to obtain a set of goal features. In addition, a multi-stage goal generator takes these goal features and uses a backward decoder to produce several stage goals. Finally, we integrate the above steps into a conditional autoencoder and a forward decoder to produce trajectory prediction results. Experiments on three public datasets, JAAD, PIE, and KITTI, and a new egocentric trajectory prediction dataset, Fuzhou DashCam (FZDC), validate the efficacy of the proposed method.

Economic benefit analysis of scene transformation of business model-taking Suning.cn as an example

With the rapid development of the new economy promoted by the new generation of technological revolution, new scenes, new formats and new models have become the direction and path of China's economic transformation and upgrading. For the retail industry, which is significantly affected by the impact of the Internet, scene transformation is the hottest and most powerful straw to reverse the current difficulties. This paper takes Suning.cn as the target case company to study the economic benefits brought by its scene transformation. Firstly, it points out the dilemma faced by the retail industry at present, and analyzes and explains the concept of scene with other experts' theories. Secondly, the scene layout of Suning is sorted out and compared with that of Sichuan Changhong in terms of profitability, operation ability and development ability. Finally, from the financial perspective, it summarizes the economic benefits brought by the scene transformation to Suning.

Interval-valued spherical fuzzy quality function deployment methodology: Metaverse collaborative system design application

The metaverse collaborative system (MCS) aims to assist users in working together, sharing resources, and participating in joint activities in the metaverse environment. As an interactive system, identifying essential design requirements (DRs) based on user requirements (URs) during the conceptual design phase of MCS is challenging as it directly affects the cost of subsequent optimization design and user experimentation, as well as user experience. Therefore, we propose a quality function deployment (QFD) method based on multi-criteria decision-making (MCDM) in an interval-valued spherical fuzzy (IVSF) environment to establish a mapping relationship between URs and DRs and identify human-centered DRs to guide the conceptual design process of MCS. Firstly, we summarized the URs for MCS design and calculated their weights using IVSF based analytical hierarchy process (IVSF-AHP). Afterward, we determined DRs based on URs and established the UR-DR relationship in the house of quality of QFD. Next, we integrate the VlseKriterijuska Optimizacija I Komoromisno Resenje (VIKOR) method in the IVSF environment into QFD to determine the priority of DRs. The results indicate that system functional design, scene layout design, compatibility design, and multi-user collaborative optimization design are important DRs for designing MCS and should be carefully considered. We validated the applicability of this method through an urban planning MCS design example and provided a conceptual design case including functional blueprints. Sensitivity analysis and comparative analysis demonstrate the flexibility and reliability of the proposed method.

A single-stream adaptive scene layout modeling method for scene recognition

A single-stream adaptive scene layout modeling method for scene recognition

Sequential selection and calibration of video frames for 3D outdoor scene reconstruction

Abstract3D scene understanding and reconstruction aims to obtain a concise scene representation from images and reconstruct the complete scene, including the scene layout, objects bounding boxes and shapes. Existing holistic scene understanding methods primarily recover scenes from single images, with a focus on indoor scenes. Due to the complexity of real‐world, the information provided by a single image is limited, resulting in issues such as object occlusion and omission. Furthermore, captured data from outdoor scenes exhibits characteristics of sparsity, strong temporal dependencies and a lack of annotations. Consequently, the task of understanding and reconstructing outdoor scenes is highly challenging. The authors propose a sparse multi‐view images‐based 3D scene reconstruction framework (SMSR). It divides the scene reconstruction task into three stages: initial prediction, refinement, and fusion stage. The first two stages extract 3D scene representations from each viewpoint, while the final stage involves selection, calibration and fusion of object positions and orientations across different viewpoints. SMSR effectively address the issue of object omission by utilizing small‐scale sequential scene information. Experimental results on the general outdoor scene dataset UrbanScene3D‐Art Sci and our proprietary dataset Software College Aerial Time‐series Images, demonstrate that SMSR achieves superior performance in the scene understanding and reconstruction.

Impact of AI in the Animation Industry

Abstract: This manuscript develops into the transformative role of artificial intelligence (AI) in the field of animation. Over the years, AI has revolutionized various industries including healthcare, banking, e-commerce, agriculture, manufacturing, and others., and animation is no exception. This paper provides an overview of the applications of AI in animation, including character creation, motion synthesis, scene generation, generating facial expressions and body language for characters, planning out scene layouts and camera movements, and generating backgrounds and environments and more. It discusses the challenges and ethical considerations associated with AI-driven animation and presents insights into the future directions of AI in the animation industry

Back to Reality: Learning Data-Efficient 3D Object Detector With Shape Guidance.

In this paper, we propose a weakly-supervised approach for 3D object detection, which makes it possible to train a strong 3D detector with position-level annotations (i.e. annotations of object centers and categories). In order to remedy the information loss from box annotations to centers, our method makes use of synthetic 3D shapes to convert the position-level annotations into virtual scenes with box-level annotations, and in turn utilizes the fully-annotated virtual scenes to complement the real labels. Specifically, we first present a shape-guided label-enhancement method, which assembles 3D shapes into physically reasonable virtual scenes according to the coarse scene layout extracted from position-level annotations. Then we transfer the information contained in the virtual scenes back to real ones by applying a virtual-to-real domain adaptation method, which refines the annotated object centers and additionally supervises the training of detector with the virtual scenes. Since the shape-guided label enhancement method generates virtual scenes by human-heuristic physical constraints, the layout of the fixed virtual scenes may be unreasonable with varied object combinations. To address this, we further present differentiable label enhancement to optimize the virtual scenes including object scales, orientations and locations in a data-driven manner. Moreover, we further propose a label-assisted self-training strategy to fully exploit the capability of detector. By reusing the position-level annotations and virtual scenes, we fuse the information from both domains and generate box-level pseudo labels on the real scenes, which enables us to directly train a detector in fully-supervised manner. Extensive experiments on the widely used ScanNet and Matterport3D datasets show that our approach surpasses current weakly-supervised and semi-supervised methods by a large margin, and achieves comparable detection performance with some popular fully-supervised methods with less than 5% of the labeling labor.

Haisor : Human-aware Indoor Scene Optimization via Deep Reinforcement Learning

3D scene synthesis facilitates and benefits many real-world applications. Most scene generators focus on making indoor scenes plausible via learning from training data and leveraging extra constraints such as adjacency and symmetry. Although the generated 3D scenes are mostly plausible with visually realistic layouts, they can be functionally unsuitable for human users to navigate and interact with furniture. Our key observation is that human activity plays a critical role and sufficient free space is essential for human-scene interactions. This is exactly where many existing synthesized scenes fail—the seemingly correct layouts are often not fit for living. To tackle this, we present a human-aware optimization framework Haisor for 3D indoor scene arrangement via reinforcement learning, which aims to find an action sequence to optimize the indoor scene layout automatically. Based on the hierarchical scene graph representation, an optimal action sequence is predicted and performed via Deep Q-Learning with Monte Carlo Tree Search (MCTS), where MCTS is our key feature to search for the optimal solution in long-term sequences and large action space. Multiple human-aware rewards are designed as our core criteria of human-scene interaction, aiming to identify the next smart action by leveraging powerful reinforcement learning. Our framework is optimized end-to-end by giving the indoor scenes with part-level furniture layout including part mobility information. Furthermore, our methodology is extensible and allows utilizing different reward designs to achieve personalized indoor scene synthesis. Extensive experiments demonstrate that our approach optimizes the layout of 3D indoor scenes in a human-aware manner, which is more realistic and plausible than original state-of-the-art generator results, and our approach produces superior smart actions, outperforming alternative baselines.

Spatial Steerability of GANs via Self-Supervision from Discriminator.

Generative models make huge progress to the photorealistic image synthesis in recent years. To enable human to steer the image generation process and customize the output, many works explore the interpretable dimensions of the latent space in GANs. Existing methods edit the attributes of the output image such as orientation or color scheme by varying the latent code along certain directions. However, these methods usually require additional human annotations for each pretrained model, and they mostly focus on editing global attributes. In this work, we propose a self-supervised approach to improve the spatial steerability of GANs without searching for steerable directions in the latent space or requiring extra annotations. Specifically, we design randomly sampled Gaussian heatmaps to be encoded into the intermediate layers of generative models as spatial inductive bias. Along with training the GAN model from scratch, these heatmaps are being aligned with the emerging attention of the GAN's discriminator in a self-supervised learning manner. During inference, users can interact with the spatial heatmaps in an intuitive manner, enabling them to edit the output image by adjusting the scene layout, moving, or removing objects. Moreover, we incorporate DragGAN into our framework, which facilitates fine-grained manipulation within a reasonable time and supports a coarse-to-fine editing process. Extensive experiments show that the proposed method not only enables spatial editing over human faces, animal faces, outdoor scenes, and complicated multi-object indoor scenes but also brings improvement in synthesis quality. Code, models, and demo video are available at https://genforce.github.io/SpatialGAN/.

Divide and Conquer: Improving Multi-Camera 3D Perception With 2D Semantic-Depth Priors and Input-Dependent Queries.

3D perception tasks, such as 3D object detection and Bird's-Eye-View (BEV) segmentation using multi-camera images, have drawn significant attention recently. Despite the fact that accurately estimating both semantic and 3D scene layouts are crucial for this task, existing techniques often neglect the synergistic effects of semantic and depth cues, leading to the occurrence of classification and position estimation errors. Additionally, the input-independent nature of initial queries also limits the learning capacity of Transformer-based models. To tackle these challenges, we propose an input-aware Transformer framework that leverages Semantics and Depth as priors (named SDTR). Our approach involves the use of an S-D Encoder that explicitly models semantic and depth priors, thereby disentangling the learning process of object categorization and position estimation. Moreover, we introduce a Prior-guided Query Builder that incorporates the semantic prior into the initial queries of the Transformer, resulting in more effective input-aware queries. Extensive experiments on the nuScenes and Lyft benchmarks demonstrate the state-of-the-art performance of our method in both 3D object detection and BEV segmentation tasks.

NR-RRT: Neural Risk-Aware Near-Optimal Path Planning in Uncertain Nonconvex Environments

Balancing the trade-off between safety and efficiency is of significant importance for path planning under uncertainty. Many risk-aware path planners have been developed to explicitly limit the probability of collision to an acceptable bound in uncertain environments. However, convex obstacles or Gaussian uncertainties are usually assumed to make the problem tractable in the existing method. These assumptions limit the generalization and application of path planners in real-world implementations. In this article, we propose to apply deep learning methods to the sampling-based planner, developing a novel risk bounded near-optimal path planning algorithm named neural risk-aware RRT (NR-RRT). Specifically, a deterministic risk contours map is maintained by perceiving the probabilistic nonconvex obstacles, and a neural network sampler is proposed to predict the next most-promising safe state. Furthermore, the recursive divide-and-conquer planning and bidirectional search strategies are used to accelerate the convergence to a near-optimal solution with guaranteed bounded risk. Worst-case theoretical guarantees can also be proven owing to a standby safety guaranteed planner utilizing a uniform sampling distribution. Simulation experiments demonstrate that the proposed algorithm outperforms the state-of-the-art remarkably for finding risk bounded low-cost paths in seen and unseen environments with uncertainty and nonconvex constraints.

Real-scene-constrained virtual scene layout synthesis for mixed reality

Real-scene-constrained virtual scene layout synthesis for mixed reality

Generating novel scene compositions from single images and videos

Given a large dataset for training, generative adversarial networks (GANs) can achieve remarkable performance for the image synthesis task. However, training GANs in extremely low data regimes remains a challenge as overfitting often occurs, leading to memorization or training divergence. In this work, we introduce SIV-GAN, an unconditional generative model that can generate new scene compositions from a single training image or a single video clip. We propose a two-branch discriminator architecture with content and layout branches that are designed to judge internal content and scene layout realism separately from each other. This discriminator design enables the synthesis of visually plausible, novel compositions of a scene with varying content and layout while preserving the context of the original sample. Compared to previous single-image GANs, our model generates more diverse images of higher quality while not being restricted to a single image setting. We further introduce a new challenging task of learning from a few frames of a single video. In this training setup the training images are highly similar to each other, which makes it difficult for prior GAN models to achieve a synthesis of both high quality and diversity.

L2 Learners’ Signed Language Processing Relates, in Part, to Perspective‐Taking Skills

AbstractThe comprehension of signed language requires linguistic and visual–spatial processing, such as perspective‐taking for correctly interpreting the layout of a spatial scene. However, little is known about how adult second‐language (L2) learners process visual–spatial constructions in a signed language that they are studying, including which angles of viewing are most challenging to process and whether there are relationships between perspective‐taking and the comprehension of non‐spatial (i.e., non‐scene based) constructions. We examine the performance of 95 intermediate signers of American Sign Language (ASL) on linguistic and non‐linguistic perspective‐taking tests. Half the participants completed a test of narrative comprehension that included visual–spatial scenes, and half took a test of signed phonological and morphophonological discrimination. Performance on linguistic perspective‐taking correlated moderately with performance on the narrative, but not with the discrimination test. These findings support the claim that perspective‐taking skills are yoked to some—but not all—aspects of signed language learning.

Deep Generative Modeling Based on VAE-GAN for 3D Indoor Scene Synthesis

With the advancement of virtual reality and 3D game technology, the demand for high-quality 3D indoor scene generation has surged. Addressing this need, this paper presents a method leveraging a VAE-GAN-based framework to conquer two primary challenges in 3D scene representation and deep generative networks. First, we introduce a matrix representation to encode fine-grained object attributes, alongside a complete graph to implicitly capture object spatial relations—effectively encapsulating both local and global scene structures. Second, we devise a unique generative framework based on VAE-GAN and the Bayesian optimization. This framework learns a Gaussian distribution of encoded object attributes through a VAE-GAN network, allowing for sampling and decoding of the distribution to generate new object attributes. Subsequently, a U-Net is employed to learn spatial relations between objects. Lastly, the Bayesian optimization module amalgamates the generated object attributes, spatial relations, and priors learned from data, conducting global optimization to generate a logical scene layout. Experimental results on a large-scale 3D indoor scene dataset substantiate that our method effectively learns inter-object relations and generates diverse and plausible indoor scenes. Comparative experiments and user studies further validate that our method surpasses the current state-of-the-art techniques in indoor scene generation and is comparable to real training scenes.

User-Aware Rendering: Merging the Strengths of Device- and User-Perspective Rendering in Handheld AR

In handheld AR, users have only a small screen to see the augmented scene, making decisions about scene layout and rendering techniques crucial. Traditional device-perspective rendering (DPR) uses the device camera's full field of view, enabling fast scene exploration, but ignoring what the user sees around the device screen. In contrast, user-perspective rendering (UPR) emulates the feeling of looking through the device like a glass pane, which enhances depth perception, but severely limits the field of view in which virtual objects are displayed, impeding scene exploration and search. We introduce the notion of User-Aware Rendering. By following the principles of UPR, but pretending the device is larger than it actually is, it combines the strengths of UPR and DPR. We present two studies showing that User-Aware AR imitating a 50% larger device successfully achieves both enhanced depth perception and fast scene exploration in typical search and selection tasks.