Layout Of Objects Research Articles

Visual perspective taking and action understanding

Understanding what others are doing is a fundamental aspect of social cognition and a skill that is arguably linked to visuospatial perspective taking (VPT), the ability to apprehend the spatial layout of a scene from another's perspective. Yet, with few and notable exceptions, action understanding and VPT are rarely studied together. Participants (43 females, 37 males) made judgements about the spatial layout of objects in a scene from the perspective of an avatar who was positioned at 0°, 90°, 270° or 180° relative to the participant. In a variant of a traditional VPT task, the avatar either interacted with the objects in the scene, by pointing to or reaching for them, or was present but did not engage with the objects. Although the task was identical across all conditions - to say whether a target object is to the right or left of a control object - we show that the avatar's actions modulates performance. Specifically, participants were more accurate when the avatar engaged with the target object, and correspondingly, less accurate and slower when the avatar interacted with the control objects. As these effects were independent of the angular disparity between participant and avatar perspectives, we conclude that action understanding and VPT are likely linked via the rapid deployment of two separate cognitive mechanisms. All participants provided a measure of self-reported empathy and we show that response times decrease with increasing empathy scores for female but not for male participants. However, within the range of ‘typical’ empathy scores, defined here as the interquartile range where 50 % of the data lie, females were faster than males. These findings lend further insight into the relationship between spatial and social perspective taking.

Online Scene Semantic Understanding Based on Sparsely Correlated Network for AR.

Real-world understanding serves as a medium that bridges the information world and the physical world, enabling the realization of virtual-real mapping and interaction. However, scene understanding based solely on 2D images faces problems such as a lack of geometric information and limited robustness against occlusion. The depth sensor brings new opportunities, but there are still challenges in fusing depth with geometric and semantic priors. To address these concerns, our method considers the repeatability of video stream data and the sparsity of newly generated data. We introduce a sparsely correlated network architecture (SCN) designed explicitly for online RGBD instance segmentation. Additionally, we leverage the power of object-level RGB-D SLAM systems, thereby transcending the limitations of conventional approaches that solely emphasize geometry or semantics. We establish correlation over time and leverage this correlation to develop rules and generate sparse data. We thoroughly evaluate the system's performance on the NYU Depth V2 and ScanNet V2 datasets, demonstrating that incorporating frame-to-frame correlation leads to significantly improved accuracy and consistency in instance segmentation compared to existing state-of-the-art alternatives. Moreover, using sparse data reduces data complexity while ensuring the real-time requirement of 18 fps. Furthermore, by utilizing prior knowledge of object layout understanding, we showcase a promising application of augmented reality, showcasing its potential and practicality.

Revealing the exploitability of heap overflow through PoC analysis

The exploitable heap layouts are used to determine the exploitability of heap vulnerabilities in general-purpose applications. Prior studies have focused on using fuzzing-based methods to generate more exploitable heap layouts. However, the exploitable heap layout cannot fully demonstrate the exploitability of a vulnerability, as it is uncertain whether the attacker can control the data covered by the overflow. In this paper, we propose the Heap Overflow Exploitability Evaluator (Hoee), a new approach to automatically reveal the exploitability of heap buffer overflow vulnerabilities by evaluating proof-of-concepts (PoCs) generated by fuzzers. Hoee leverages several techniques to collect dynamic information at runtime and recover heap object layouts in a fine-grained manner. The overflow context is carefully analyzed to determine whether the sensitive pointer is corrupted, tainted, or critically used. We evaluate Hoee on 34 real-world CVE vulnerabilities from 16 general-purpose programs. The results demonstrate that Hoee accurately identifies the key factors for developing exploits in vulnerable contexts and correctly recognizes the behavior of overflow.

Method for Collaborative Layout Optimization of Ship Equipment and Pipe Based on Improved Multi-Agent Reinforcement Learning and Artificial Fish Swarm Algorithm

The engine room is the core area of a ship, critical to its operation, safety, and efficiency. Currently, many researchers merely address the ship engine room layout design (SERLD) problem using optimization algorithms and independent layout strategies. However, the engine room environment is complex, involving two significantly different challenges: equipment layout and pipe layout. Traditional methods fail to achieve optimal collaborative layout objectives. To address this research gap, this paper proposes a collaborative layout method that combines improved reinforcement learning and heuristic algorithms. For equipment layout, the engine room space is first discretized into a grid, and a Markov decision process (MDP) framework suitable for equipment layout is proposed, including state space, action space, and reward mechanisms suitable for equipment layout. An improved adaptive guided multi-agent Q-learning (AGMAQL) algorithm is employed to train the layout model in a centralized manner, with enhancements made to the agent’s exploration state, exploration action, and learning strategy. For pipe layout, this paper proposes an improved adaptive trajectory artificial fish swarm algorithm (ATAFSA). This algorithm incorporates a hybrid encoding method, adaptive strategy, scouting strategy, and parallel optimization strategy, resulting in enhanced stability, accuracy, and problem adaptability. Subsequently, by comprehensively considering layout objectives and engine room attributes, a collaborative layout method incorporating hierarchical and adaptive weight strategies is proposed. This method optimizes in phases according to the layout objectives and priorities of different stages, achieving multi-level optimal layouts and providing designers with various reference schemes with different focuses. Finally, based on a typical real-world engine room engineering case, various leading algorithms and strategies are tested and compared. The results show that the proposed AGMAQL-ATAFSA (AGMAQL-ATA) exhibits robustness, efficiency, and engineering practicality. Compared to previous research methods and algorithms, the final layout quality improved overall: equipment layout effectiveness increased by over 4.0%, pipe optimization efficiency improved by over 40.4%, and collaborative layout effectiveness enhanced by over 2.2%.

Building Category Graphs Representation with Spatial and Temporal Attention for Visual Navigation

Given an object of interest, visual navigation aims to reach the object’s location based on a sequence of partial observations. To this end, an agent needs to (1) acquire specific knowledge about the relations of object categories in the world during training and (2) locate the target object based on the pre-learned object category relations and its trajectory in the current unseen environment. In this article, we propose a Category Relation Graph (CRG) to learn the knowledge of object category layout relations and a Temporal-Spatial-Region attention (TSR) architecture to perceive the long-term spatial-temporal dependencies of objects, aiding navigation. We establish CRG to learn prior knowledge of object layout and deduce the positions of specific objects. Subsequently, we propose the TSR architecture to capture relationships among objects in temporal, spatial, and regions within observation trajectories. Specifically, we implement a Temporal attention module (T) to model the temporal structure of the observation sequence, implicitly encoding historical moving or trajectory information. Then, a Spatial attention module (S) uncovers the spatial context of the current observation objects based on CRG and past observations. Last, a Region attention module (R) shifts the attention to the target-relevant region. Leveraging the visual representation extracted by our method, the agent accurately perceives the environment and easily learns a superior navigation policy. Experiments on AI2-THOR demonstrate that our CRG-TSR method significantly outperforms existing methods in both effectiveness and efficiency. The supplementary material includes the code and will be publicly available.

Directed Diffusion: Direct Control of Object Placement through Attention Guidance

Text-guided diffusion models such as DALLE-2, Imagen, and Stable Diffusion are able to generate an effectively endless variety of images given only a short text prompt describing the desired image content. In many cases the images are of very high quality. However, these models often struggle to compose scenes containing several key objects such as characters in specified positional relationships. The missing capability to ``direct'' the placement of characters and objects both within and across images is crucial in storytelling, as recognized in the literature on film and animation theory. In this work, we take a particularly straightforward approach to providing the needed direction. Drawing on the observation that the cross-attention maps for prompt words reflect the spatial layout of objects denoted by those words, we introduce an optimization objective that produces ``activation'' at desired positions in these cross-attention maps. The resulting approach is a step toward generalizing the applicability of text-guided diffusion models beyond single images to collections of related images, as in storybooks. Directed Diffusion provides easy high-level positional control over multiple objects, while making use of an existing pre-trained model and maintaining a coherent blend between the positioned objects and the background. Moreover, it requires only a few lines to implement.

Photography Exploration Training using Still Life Techniques for Karmel Tangerang Vocational School Students

Photography techniques that are supported by equipment that makes it easier for users make this medium increasingly popular with people from all walks of life. Photography has become part of various life activities. Not only selfies, photography has been used in all human activities. Millions of images are produced every day and published on social media pages. This phenomenon aligns that photography has become the property of all society, not just photographers. So the quality of the resulting photos is not just a photograph but can have its own value both in terms of aesthetics and function. For this reason, through training at Karmel Vocational School, it is hoped that it can help hone students' skills so that they can produce photos with the correct techniques so as to produce marketable photographic works with attractive quality. Training with a direct learning method using camera equipment, lighting, object layout, both the background and the objects to be photographed. With this direct learning, it is hoped that students will be able to experiment and explore directly the photo objects they will be photographing. Problems discovered during shooting can be discussed directly to produce maximum photos and provide new knowledge about the techniques of photographing objects/still life.

Predictive simulation of construction site noise emissions from heavy equipment

PurposeConstruction activities conducted in urban areas are often a source of significant noise disturbances, which cause psychological and health issues for residents as well as long-term auditory impairments for construction workers. The limited effectiveness of passive noise control measures due to the close proximity of the construction site to surrounding neighborhoods often results in complaints and eventually lawsuits. These can then lead to delays and cost overruns for the construction projects.Design/methodology/approachThe paper proposes a novel approach to integrating construction noise as an additional dimension into scheduling construction works. To achieve this, a building information model, including the three-dimensional construction site layout object geometry, resource allocation and schedule information, is utilized. The developed method explores further project data that are typically available, such as the assigned equipment to a task, its precise location, and the estimated duration of noisy tasks. This results in a noise prediction model by using noise mapping techniques and suggesting less noisy alternative ways of construction. Finally, noise data obtained from sensors in a case study contribute real values for validating the proposed approach, which can be used later to suggest solutions for noise mitigation.FindingsThe results of this study indicate that the proposed approach can accurately predict construction noise given a few available parameters from digital project planning and sensors installed on a construction site. Proactively integrating construction noise control measures into the planning process has benefits for both residents and construction managers, as it reduces construction noise-related disturbances, prevents unexpected legal issues and ensures the health and well-being of the workforce.Originality/valueWhile previous research has concentrated on real-time data collection using sensors, a more effective solution would also involve addressing and mitigating construction noise during the pre-construction work planning phase.

DRCmpVis: Visual Comparison of Physical Targets in Mobile Diminished and Mixed Reality.

Numerous physical objects in our daily lives are grouped or ranked according to a stereotyped presentation style. For example, in a library, books are typically grouped and ranked based on classification numbers. However, for better comparison, we often need to re-group or re-rank the books using additional attributes such as ratings, publishers, comments, publication years, keywords, prices, etc., or a combination of these factors. In this paper, we propose a novel mobile DR/MR-based application framework named DRCmpVis to achieve in-context multi-attribute comparisons of physical objects with text labels or textual information. The physical objects are scanned in the real world using mobile cameras. All scanned objects are then segmented and labeled by a convolutional neural network and replaced (diminished) by their virtual avatars in a DR environment. We formulate three visual comparison strategies, including filtering, re-grouping, and re-ranking, which can be intuitively, flexibly, and seamlessly performed on their avatars. This approach avoids breaking the original layouts of the physical objects. The computation resources in virtual space can be fully utilized to support efficient object searching and multi-attribute visual comparisons. We demonstrate the usability, expressiveness, and efficiency of DRCmpVis through a user study, NASA TLX assessment, quantitative evaluation, and case studies involving different scenarios.

Exploration of the optimization path of interior art design based on computer BIM technology

Abstract When science and technology have been developed faster, many interior art design concepts and computer technology have been updated with the times. In this paper, the geometric data and semantic data in interior design are transformed through BIM modeling, the spatial structure elements are placed in the world coordinate system, and the relative coordinates are calculated with the spatial structure elements of the interior art design building entity as the origin. With the help of BIM technology, the interior art design layout is carried out. Based on the layout constraint function and spatial syntax quantization function, which constitute the layout objective optimization function, the optimization solution is calculated. Finally, Depthmap software is used to analyze the interior design layout for VGA and intelligent body simulation. The results show that after 100 iterations, the mean value of the coefficient of the degree of BIM application is 0.9373, the coefficient of the application environment is 0.951, and the degree of the application of the BIM technology shows a growing trend, indicating that the BIM technology is an inevitable direction for the reform of the interior art design technology.

Semantic-Layout-Guided Image Synthesis for High-Quality Synthetic-Aperature Radar Detection Sample Generation

With the widespread application and functional complexity of deep neural networks (DNNs), the demand for training samples is increasing. This elevated requirement also extends to DNN-based SAR object detection. Most public SAR object detection datasets are oriented to marine targets such as ships, while data sets oriented to land targets are relatively rare, though they are an effective way to improve the land object detection capability of deep models through SAR sample generation. In this paper, a synthesis generation collaborative SAR sample augmentation framework is proposed to achieve flexible and diverse high-quality sample augmentation. First, a semantic-layout-guided image synthesis strategy is proposed to generate diverse detection samples. The issues of object location rationality and object layout diversity are also addressed. Meanwhile, a pix2pixGAN network guided by layout maps is utilized to achieve diverse background augmentation. Second, a progressive training strategy of diffusion models is proposed to achieve semantically controllable SAR sample generation to further improve the diversity of scene clutter. Finally, a sample cleaning method considering distribution migration and network filtering is employed to further improve the quality of detection samples. The experimental results show that this semantic synthesis generation method can outperform existing sample augmentation methods, leading to a comprehensive improvement in the accuracy metrics of classical detection networks.

HRRSODN: high-Resolution remote sensing object detection network fusion of road-adjacent relationships

Application of deep learning technology in remote sensing image target detection(RSITD)has made remarkable progress in recent years. However, existing methods usually fuse spatial context information at the pixel level while ignoring the mining of spatial relationships between ground objects. This paper proposes a high-resolution remote sensing object detection network fusion of road-adjacent relationships (HRRSODN) that integrates road proximity relations. Taking artificial features such as toll stations adjacent to roads as an example, the spatial relationship of road proximity is used to assist visual information and improve the extraction accuracy. The method optimizes detection by: Using the spatial relationship as prior information; fusing coordinate attention that captures the spatial relationship between objects; and adding the attention generated by the spatial relationship to the attention generated in the original X and Y directions. This enables the model to understand the spatial layout of objects in the image. The distance loss function concretizes the spatial relationship into a distance index. The distance loss is incorporated into the training process by measuring the spatial distance between feature targets. Thus, during training, the network focuses not only on the positional information of the targets but also pays more attention to the spatial layout of the features. Experimental results show that, compared with conventional methods, the proposed method has achieved significant improvements in detection performance and mining spatial relationships. This verifies the importance of considering the spatial relationship of ground objects in remote sensing target detection. Further, this paper proposes a new method for target detection in remote sensing images (RSIs), which effectively utilizes the spatial relationship information between targets, and provides a new idea for further improving target detection in RSI.

IBL‐NeRF: Image‐Based Lighting Formulation of Neural Radiance Fields

AbstractWe propose IBL‐NeRF, which decomposes the neural radiance fields (NeRF) of large‐scale indoor scenes into intrinsic components. Recent approaches further decompose the baked radiance of the implicit volume into intrinsic components such that one can partially approximate the rendering equation. However, they are limited to representing isolated objects with a shared environment lighting, and suffer from computational burden to aggregate rays with Monte Carlo integration. In contrast, our prefiltered radiance field extends the original NeRF formulation to capture the spatial variation of lighting within the scene volume, in addition to surface properties. Specifically, the scenes of diverse materials are decomposed into intrinsic components for rendering, namely, albedo, roughness, surface normal, irradiance, and prefiltered radiance. All of the components are inferred as neural images from MLP, which can model large‐scale general scenes. Especially the prefiltered radiance effectively models the volumetric light field, and captures spatial variation beyond a single environment light. The prefiltering aggregates rays in a set of predefined neighborhood sizes such that we can replace the costly Monte Carlo integration of global illumination with a simple query from a neural image. By adopting NeRF, our approach inherits superior visual quality and multi‐view consistency for synthesized images as well as the intrinsic components. We demonstrate the performance on scenes with complex object layouts and light configurations, which could not be processed in any of the previous works.

Underwater Target Detection Based on Parallel High-Resolution Networks.

A parallel high-resolution underwater target detection network is proposed to address the problems of complex underwater scenes and limited target feature extraction capability. First, a high-resolution network (HRNet), a lighter high-resolution human posture estimation network, is used to improve the target feature representation and effectively reduce the semantic information lost in the image during sampling. Then, the attention module (A-CBAM) is improved to capture complex feature distributions by modeling the two-dimensional space in the activation function stage through the introduction of the flexible rectified linear units (FReLU) activation function to achieve pixel-level spatial information modeling capability. Feature enhancement in the spatial and channel dimensions is performed to improve understanding of fuzzy targets and small target objects and to better capture irregular and detailed object layouts. Finally, a receptive field augmentation module (RFAM) is constructed to obtain sufficient semantic information and rich detail information to further enhance the robustness and discrimination of features and improve the detection capability of the model for multi-scale underwater targets. Experimental results show that the method achieves 81.17%, 77.02%, and 82.9% mean average precision (mAP) on three publicly available datasets, specifically underwater robot professional contest (URPC2020, URPC2018) and pattern analysis, statistical modeling, and computational learning visual object classes (PASCAL VOC2007), respectively, demonstrating the effectiveness of the proposed network.

Scene Graph to Image Synthesis via Knowledge Consensus

In this paper, we study graph-to-image generation conditioned exclusively on scene graphs, in which we seek to disentangle the veiled semantics between knowledge graphs and images. While most existing research resorts to laborious auxiliary information such as object layouts or segmentation masks, it is also of interest to unveil the generality of the model with limited supervision, moreover, avoiding extra cross-modal alignments. To tackle this challenge, we delve into the causality of the adversarial generation process, and reason out a new principle to realize a simultaneous semantic disentanglement with an alignment on target and model distributions. This principle is named knowledge consensus, which explicitly describes a triangle causal dependency among observed images, graph semantics and hidden visual representations. The consensus also determines a new graph-to-image generation framework, carried on several adversarial optimization objectives. Extensive experimental results demonstrate that, even conditioned only on scene graphs, our model surprisingly achieves superior performance on semantics-aware image generation, without losing the competence on manipulating the generation through knowledge graphs.

Transformer-based image generation from scene graphs

Graph-structured scene descriptions can be efficiently used in generative models to control the composition of the generated image. Previous approaches are based on the combination of graph convolutional networks and adversarial methods for layout prediction and image generation, respectively. In this work, we show how employing multi-head attention to encode the graph information, as well as using a transformer-based model in the latent space for image generation can improve the quality of the sampled data, without the need to employ adversarial models with the subsequent advantage in terms of training stability.The proposed approach, specifically, is entirely based on transformer architectures both for encoding scene graphs into intermediate object layouts and for decoding these layouts into images, passing through a lower dimensional space learned by a vector-quantized variational autoencoder. Our approach shows an improved image quality with respect to state-of-the-art methods as well as a higher degree of diversity among multiple generations from the same scene graph. We evaluate our approach on three public datasets: Visual Genome, COCO, and CLEVR. We achieve an Inception Score of 13.7 and 12.8, and an FID of 52.3 and 60.3, on COCO and Visual Genome, respectively. We perform ablation studies on our contributions to assess the impact of each component. Code is available at https://github.com/perceivelab/trf-sg2im.

Re:]Entanglements: Colonial Collections in Decolonial Times curated by Paul Basu

Re:]Entanglements: Colonial Collections in Decolonial Times curated by Paul Basu

Indoor Visual Re-Localization for Long-Term Autonomous Robots Based on Object-Level Features and Semantic Relationships

Visual re-localization has become one of the key technologies for long-term autonomous robots. Existing methods, mostly focusing on addressing day-night, weather, and seasonal changes, are not applicable in indoor scenarios. At the same time, the layouts of objects in indoor scenes are highly dynamic over time due to human interactions with the environment, which makes indoor re-localization challenging. This letter presents a novel indoor visual re-localization method for long-term autonomous robots. First, a scene graph model is proposed, incorporating object-level features and semantic relationships, which overcomes the influence of dynamic objects by understanding the interactions among objects. Then, a visual re-localization method is developed based on the proposed scene graph model. It adopts graph matching technologies to incorporate pairwise object interactions as important features for re-localization, and designs a feature reweighting strategy to further reduce the impact of outliers in dynamic scenes. The proposed re-localization method has been verified in both photorealistic simulation environments and real-world scenarios. The results show that our approach exhibits higher robustness to diverse object changes and performs comparably to the state-of-the-art methods when illumination changes occur.

General lighting can overcome accidental viewing.

When seeing an object in a scene, the presumption of seeing that object from a general viewpoint (as opposed to an accidental viewpoint) is a useful heuristic to decide which of many interpretations of this object is correct. Similar heuristic assumptions on illumination quality might also be used for scene interpretation. Here we tested that assumption and asked if illumination information helps determine object properties when seen from an accidental viewpoint. Test objects were placed on a flat surface and illumination was varied while keeping the objects' images constant. Observers judged the shape or rigidity of static or moving simple objects presented in accidental view. They also chose which of two seemingly very similar faces was familiar. We found: (1) Objects might appear flat without shadow information but were perceived to be volumetric objects or non-planar in the presence of cast shadows. (2) Apparently non-rigid objects became rigid with shadow information. (3) Shading and shadows helped to infer which of two face was the familiar one. Previous results had shown that cast shadows help determine spatial layout of objects. Our study shows that other properties of objects like rigidity or 3D-shape can be disambiguated by shadow information.

Spatial Cognition of the Visually Impaired: A Case Study in a Familiar Environment.

This paper aims to explore the factors influencing the spatial cognition of the visually impaired in familiar environments. Massage hospitals are some of the few places that can provide work for the visually impaired in China. Studying the spatial cognition of the visually impaired in a massage hospital could be instructive for the design of working environments for the visually impaired and other workplaces in the future. First, the subjective spatial cognition of the visually impaired was evaluated by object layout tasks for describing the spatial relationships among object parts. Second, physiological monitoring signal data, including the electrodermal activity, heart rate variability, and electroencephalography, were collected while the visually impaired doctors walked along prescribed routes based on the feature analysis of the physical environment in the hospital, and then their physiological monitoring signal data for each route were compared. The visual factors, physical environmental factors, and human-environment interactive factors that significantly impact the spatial cognition of visually impaired people were discussed. (1) visual acuity affects the spatial cognition of the visually impaired in familiar environments; (2) the spatial cognition of the visually impaired can be promoted by a longer staying time and the more regular sequence of a physical environment; (3) the spatial comfort of the visually impaired can be improved by increasing the amount of greenery; and (4) the visual comfort of the visually impaired can be reduced by rich interior colors and contrasting lattice floor tiles.