Photorealistic Objects Research Articles

ProteusNeRF: Fast Lightweight NeRF Editing using 3D-Aware Image Context

Neural Radiance Fields (NeRFs) have recently emerged as a popular option for photo-realistic object capture due to their ability to faithfully capture high-fidelity volumetric content even from handheld video input. Although much research has been devoted to efficient optimization leading to real-time training and rendering, options for interactive editing NeRFs remain limited. We present a very simple but effective neural network architecture that is fast and efficient while maintaining a low memory footprint. This architecture can be incrementally guided through user-friendly image-based edits. Our representation allows straightforward object selection via semantic feature distillation at the training stage. More importantly, we propose a local 3D-aware image context to facilitate view-consistent image editing that can then be distilled into fine-tuned NeRFs, via geometric and appearance adjustments. We evaluate our setup on a variety of examples to demonstrate appearance and geometric edits and report 10-30x speedup over concurrent work focusing on text-guided NeRF editing. Video results and code can be found on our project webpage at https://proteusnerf.github.io.

Negative cues minimize visual search specificity effects

Prior target knowledge (i.e., positive cues) improves visual search performance. However, there is considerable debate about whether distractor knowledge (i.e., negative cues) can guide search. Some studies suggest the active suppression of negatively cued search items, while others suggest the initial capture of attention by negatively cued items. Prior work has used pictorial or specific text cues but has not explicitly compared them. We build on that work by comparing positive and negative cues presented pictorially and as categorical text labels using photorealistic objects and eye movement measures. Search displays contained a target (cued on positive trials), a lure from the target category (cued on negative trials), and four categorically-unrelated distractors. Search performance with positive cues resulted in stronger attentional guidance and faster object recognition for pictorial relative to categorical cues (i.e., a pictorial advantage, suggesting specific visual details afforded by pictorial cues improved search). However, in most search performance metrics, negative cues mitigate the pictorial advantage. Given that the negatively cued items captured attention, generated target guidance but mitigated the pictorial advantage, these results are partly consistent with both existing theories. Specific visual details provided in positive cues produce a large pictorial advantage in all measures, whereas specific visual details in negative cues only produce a small pictorial advantage for object recognition but not for attentional guidance. This asymmetry in the pictorial advantage suggests that the down-weighting of specific negatively cued visual features is less efficient than the up-weighting of specific positively cued visual features.

The relationship between the subjective experience of real-world cognitive failures and objective target-detection performance in visual search

Visual search is a common occurrence in everyday life, such as searching for the location of keys, identifying a friend in a crowd, or scanning an upcoming intersection for hazards while driving. Visual search is also used in professional contexts, such as medical diagnostic imaging and airport baggage screening. These contexts are often characterised by low-prevalence or rare targets. Here we tested whether individual differences in the detection of targets in visual search could be predicted from variables derived from the rich informational source of participants' subjective experience of their cognitive and attentional function in everyday life. We tested this in both low-prevalence (Experiment 1) and high-prevalence (Experiment 2) visual search conditions. In both experiments, participants completed a visual search with arrays containing multiple photorealistic objects, and their task was to detect the presence of a gun. Following this, they completed the Cognitive Failures Questionnaire (CFQ) and the Attentional Control Scale (ACS). In Experiment 1, the target was present on 2% of trials, while in Experiment 2, it was present on 50%. In both experiments, participants' scores on the False Triggering component of the CFQ were negatively associated with accuracy on target-present trials, while participants' scores on the Forgetfulness component of the CFQ were positively associated with target-present accuracy. These results show that objective performance in visual search can be predicted from subjective experiences of cognitive function. They also highlight that the CFQ is not monolithic. Instead, the CFQ subfactors can have qualitatively different relationships with performance. Theoretical and practical implications are discussed.

360°-Based Virtual Field Trips to Waterworks in Higher Education

360° models are a form of virtual reality (VR) that allow the viewer to view and explore a photorealistic object from multiple locations within the model. Hence, 360° models are an option to perform virtual field trips (VFT) independent of time and location. Thanks to recent technical progress, 360° models are creatable with little effort. Due to their characteristics of visualization and explorability, 360° models appear as excellent learning tools, especially when additional didactic features, such as annotations, are used. The subject of this explorative field study is a 360° model of a waterworks that has been annotated for learning purposes. Data are collected from a total of 55 learners in four cohorts from study programs in environmental engineering and urban studies using a questionnaire that included standardized measurement instruments on motivation, emotion, and usability. Furthermore, the eight learners of cohort 1 are surveyed using semi-structured interviews on learning, operation and features of the 360° model. Overall, a very positive view on learning suitability of 360° models in VFTs is revealed. In addition, further potential for development of the 360° model could be identified. The results indicate that VTFs based on 360° models might be valuable learning tools, because of their applicability without great effort on the part of either the lecturers or the students. VFTs based on 360° models might serve as a supplement to conventional learning activities or in self-directed learning activities.

Hybrid foraging search in younger and older age.

In hybrid foraging tasks, observers search visual displays, so called patches, for multiple instances of any of several types of targets with the goal of collecting targets as quickly as possible. Here, targets were photorealistic objects. Younger and older adults collected targets by mouse clicks. They could move to the next patch whenever they decided to do so. The number of targets held in memory varied between 8 and 64 objects, and the number of items (targets and distractors) in the patches varied between 60 and 105 objects. Older adults foraged somewhat less efficiently than younger adults due to a more exploitative search strategy. When target items became depleted in a patch and search slowed down, younger adults acted according to the optimal foraging theory and moved on to the next patch when the instantaneous rate of collection was close to their average rate of collection. Older adults, by contrast, were more likely to stay longer and spend time searching for the last few targets. Within a patch, both younger and older adults tended to collect the same type of target in "runs." This behavior is more efficient than continual switching between target types. Furthermore, after correction for general age-related slowing, RT × set size functions revealed largely preserved attention and memory functions in older age. Hybrid foraging tasks share features with important real-world search tasks. Differences between younger and older observers on this task may therefore help to explain age differences in many complex search tasks of daily life. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Searching for the right word: Hybrid visual and memory search for words.

In "hybrid search" (Wolfe Psychological Science, 23(7), 698-703, 2012), observers search through visual space for any of multiple targets held in memory. With photorealistic objects as the stimuli, response times (RTs) increase linearly with the visual set size and logarithmically with the memory set size, even when over 100 items are committed to memory. It is well-established that pictures of objects are particularly easy to memorize (Brady, Konkle, Alvarez, & Oliva Proceedings of the National Academy of Sciences, 105, 14325-14329, 2008). Would hybrid-search performance be similar if the targets were words or phrases, in which word order can be important, so that the processes of memorization might be different? In Experiment 1, observers memorized 2, 4, 8, or 16 words in four different blocks. After passing a memory test, confirming their memorization of the list, the observers searched for these words in visual displays containing two to 16 words. Replicating Wolfe (Psychological Science, 23(7), 698-703, 2012), the RTs increased linearly with the visual set size and logarithmically with the length of the word list. The word lists of Experiment 1 were random. In Experiment 2, words were drawn from phrases that observers reported knowing by heart (e.g., "London Bridge is falling down"). Observers were asked to provide four phrases, ranging in length from two words to no less than 20 words (range 21-86). All words longer than two characters from the phrase, constituted the target list. Distractor words were matched for length and frequency. Even with these strongly ordered lists, the results again replicated the curvilinear function of memory set size seen in hybrid search. One might expect to find serial position effects, perhaps reducing the RTs for the first (primacy) and/or the last (recency) members of a list (Atkinson & Shiffrin, 1968; Murdock Journal of Experimental Psychology, 64, 482-488, 1962). Surprisingly, we showed no reliable effects of word order. Thus, in "London Bridge is falling down," "London" and "down" were found no faster than "falling."

3DNN: 3D Nearest Neighbor

In this paper, we describe a data-driven approach to leverage repositories of 3D models for scene understanding. Our ability to relate what we see in an image to a large collection of 3D models allows us to transfer information from these models, creating a rich understanding of the scene. We develop a framework for auto-calibrating a camera, rendering 3D models from the viewpoint an image was taken, and computing a similarity measure between each 3D model and an input image. We demonstrate this data-driven approach in the context of geometry estimation and show the ability to find the identities, poses and styles of objects in a scene. The true benefit of 3DNN compared to a traditional 2D nearest-neighbor approach is that by generalizing across viewpoints, we free ourselves from the need to have training examples captured from all possible viewpoints. Thus, we are able to achieve comparable results using orders of magnitude less data, and recognize objects from never-before-seen viewpoints. In this work, we describe the 3DNN algorithm and rigorously evaluate its performance for the tasks of geometry estimation and object detection/segmentation, as well as two novel applications: affordance estimation and photorealistic object insertion.

More target features in visual working memory leads to poorer search guidance: Evidence from contralateral delay activity

The visual-search literature has assumed that the top-down target representation used to guide search resides in visual working memory (VWM). We directly tested this assumption using contralateral delay activity (CDA) to estimate the VWM load imposed by the target representation. In Experiment 1, observers previewed four photorealistic objects and were cued to remember the two objects appearing to the left or right of central fixation; Experiment 2 was identical except that observers previewed two photorealistic objects and were cued to remember one. CDA was measured during a delay following preview offset but before onset of a four-object search array. One of the targets was always present, and observers were asked to make an eye movement to it and press a button. We found lower magnitude CDA on trials when the initial search saccade was directed to the target (strong guidance) compared to when it was not (weak guidance). This difference also tended to be larger shortly before search-display onset and was largely unaffected by VWM item-capacity limits or number of previews. Moreover, the difference between mean strong- and weak-guidance CDA was proportional to the increase in search time between mean strong-and weak-guidance trials (as measured by time-to-target and reaction-time difference scores). Contrary to most search models, our data suggest that trials resulting in the maintenance of more target features results in poor search guidance to a target. We interpret these counterintuitive findings as evidence for strong search guidance using a small set of highly discriminative target features that remain after pruning from a larger set of features, with the load imposed on VWM varying with this feature-consolidation process.

Optimal and Preferred Eye Landing Positions in Objects and Scenes

Viewing position effects are commonly observed in reading, but they have only rarely been investigated in object perception or in the realistic context of a natural scene. In two experiments, we explored where people fixate within photorealistic objects and the effects of this landing position on recognition and subsequent eye movements. The results demonstrate an optimal viewing position-objects are processed more quickly when fixation is in the centre of the object. Viewers also prefer to saccade to the centre of objects within a natural scene, even when making a large saccade. A central landing position is associated with an increased likelihood of making a refixation, a result that differs from previous reports and suggests that multiple fixations within objects, within scenes, occur for a range of reasons. These results suggest that eye movements within scenes are systematic and are made with reference to an early parsing of the scene into constituent objects.

인터넷 항공영상의 왜곡보정과 실감적 3차원 건물 모델링

Many internet portals provide maps with spatial information services. Recently, various images including aerial, satellite, street view, and photo-realistic 3D city models are provided as well as maps. This study suggested a method for geometric correction of the panoramic aerial images in the internet portal and 3D building modeling using information which is available in the internet. The key of this study is to obtain all necessary data easily from internet without restrictions. Practically, the ground control coordinates could be available from geo-referenced internet maps, and stereo pairs of the aerial images and close-range photographs for photo-realistic object modeling are provided by the internet service. However, the ground control points are not suitable for accurate mapping. RMSE of the plotting was about 9 meters and reduced upto 4 meters after coordinate transformation. The proposed methods would be applicable to various applications of photo-realistic object modeling which do not require high accuracy.

A 3D Painterly Rendering Framework for Photon Mapping

This paper presents a novel framework for 3D painterly rendering for photon mapping. We introduce a new non-photorealistic rendering pass and integrate it within the standard photon mapping pipeline. Our framework contains three passes. The photon tracing pass casts photons to exploit a three-dimensional scene and holds photon information in a data structure developed specifically for painterly rendering. The non-photorealistic rendering pass constructs the sparse map and informative map which preserve the attributes of strokes required for 3D painterly rendering. In this pass, we also employ an impressionistic line integral convolution algorithm producing various types of painterly rendering results. Finally, the rendering pass renders photorealistic objects generating the final image consisting of the scene with both artistic painterly styles and global illumination visual effects. We implemented our framework and developed an experimental rendering system on a personal computer environment. To the best of our knowledge, the proposed framework is the first that can take advantages of both global illumination and 3D painterly rendering environment. Experimental results and rendered images demonstrate the feasibility of our framework.

Watermark recovery from two-dimensional views of a three-dimensional object using texture or silhouette information

We describe a novel framework for watermarking 3-D objects using their texture or silhouette information. Unlike most conventional 3-D object-watermarking techniques, for which both insertion and extraction of the mark are performed on the object itself (3-D/3-D approach), we propose an asymmetric 3-D/2-D procedure. It consists in watermarking 3-D objects and retrieving the mark from rendering 2-D images or videos having used the 3-D synthetic object, thus protecting the visual representations of the object. Two 3-D object watermarking schemes are presented, a texture-based approach for 3-D photorealistic objects and a silhouette-based approach for 3-D CAD (computer-assisted design) objects.

Passive navigation from image sequences by use of a volumetric approach.

The paper describes a volumetric approach to depth estimation for robot navigation with use of only an approximately calibrated translating camera. Our approach is related to techniques for photo-realistic object reconstruction but with the emphasis on issues associated with navigation. The technique performs three-dimensional matching by a process of image interpolation and can adjust for errors in camera position. The reconstruction is achieved from a small angular range of scene views, and the technique is demonstrated to be insensitive to large errors in the camera positions. The ability to correct for more critical errors such as the camera orientation is shown to significantly improve the algorithm's performance. Our technique is demonstrated on real image sequences and compares favorably with techniques based on optical flow.