Objects In Scene Research Articles

Spatial-Temporal Knowledge-Embedded Transformer for Video Scene Graph Generation.

Video scene graph generation (VidSGG) aims to identify objects in visual scenes and infer their relationships for a given video. It requires not only a comprehensive understanding of each object scattered on the whole scene but also a deep dive into their temporal motions and interactions. Inherently, object pairs and their relationships enjoy spatial co-occurrence correlations within each image and temporal consistency/transition correlations across different images, which can serve as prior knowledge to facilitate VidSGG model learning and inference. In this work, we propose a spatial-temporal knowledge-embedded transformer (STKET) that incorporates the prior spatial-temporal knowledge into the multi-head cross-attention mechanism to learn more representative relationship representations. Specifically, we first learn spatial co-occurrence and temporal transition correlations in a statistical manner. Then, we design spatial and temporal knowledge-embedded layers that introduce the multi-head cross-attention mechanism to fully explore the interaction between visual representation and the knowledge to generate spatial- and temporal-embedded representations, respectively. Finally, we aggregate these representations for each subject-object pair to predict the final semantic labels and their relationships. Extensive experiments show that STKET outperforms current competing algorithms by a large margin, e.g., improving the mR@50 by 8.1%, 4.7%, and 2.1% on different settings over current algorithms.

Siamese-DETR for Generic Multi-Object Tracking.

The ability to detect and track the dynamic objects in different scenes is fundamental to real-world applications, e.g., autonomous driving and robot navigation. However, traditional Multi-Object Tracking (MOT) is limited to track objects belonging to the pre-defined closed-set categories. Recently, Generic MOT (GMOT) is proposed to track interested objects beyond pre-defined categories and it can be divided into Open-Vocabulary MOT (OVMOT) and Template-Image-based MOT (TIMOT). Taking the consideration that the expensive well pre-trained (vision-)language model and fine-grained category annotations are required to train OVMOT models, in this paper, we focus on TIMOT and propose a simple but effective method, Siamese-DETR. Only the commonly used detection datasets (e.g., COCO) are required for training. Different from existing TIMOT methods, which train a Single Object Tracking (SOT) based detector to detect interested objects and then apply a data association based MOT tracker to get the trajectories, we leverage the inherent object queries in DETR variants. Specifically: 1) The multi-scale object queries are designed based on the given template image, which are effective for detecting different scales of objects with the same category as the template image; 2) A dynamic matching training strategy is introduced to train Siamese-DETR on commonly used detection datasets, which takes full advantage of provided annotations; 3) The online tracking pipeline is simplified through a tracking-by-query manner by incorporating the tracked boxes in the previous frame as additional query boxes. The complex data association is replaced with the much simpler Non-Maximum Suppression (NMS). Extensive experimental results show that Siamese-DETR surpasses existing MOT methods on GMOT-40 dataset by a large margin.

An airport apron ground service surveillance algorithm based on improved YOLO network

<abstract> <p>To assure operational safety in the airport apron area and track the process of ground service, it is necessary to analyze key targets and their activities in the airport apron surveillance videos. This research shows an activity identification algorithm for ground service objects in an airport apron area and proposes an improved YOLOv5 algorithm to increase the precision of small object detection by introducing an SPD-Conv (spath-to-depth-Conv) block in YOLOv5's backbone layer. The improved algorithm can efficiently extract the information features of small-sized objects, medium-sized objects, and moving objects in large scenes, and it achieves effective detection of activities of ground service in the apron area. The experimental results show that the detection average precision of all objects is more than 90%, and the whole class mean average precision (mAP) is 98.7%. At the same time, the original model was converted to TensorRT and OpenVINO format models, which increased the inference efficiency of the GPU and CPU by 55.3 and 137.1%, respectively.</p> </abstract>

Eye tracking evidence for the reinstatement of emotionally negative and neutral memories.

Recent eye tracking studies have linked gaze reinstatement-when eye movements from encoding are reinstated during retrieval-with memory performance. In this study, we investigated whether gaze reinstatement is influenced by the affective salience of information stored in memory, using an adaptation of the emotion-induced memory trade-off paradigm. Participants learned word-scene pairs, where scenes were composed of negative or neutral objects located on the left or right side of neutral backgrounds. This allowed us to measure gaze reinstatement during scene memory tests based on whether people looked at the side of the screen where the object had been located. Across two experiments, we behaviorally replicated the emotion-induced memory trade-off effect, in that negative object memory was better than neutral object memory at the expense of background memory. Furthermore, we found evidence that gaze reinstatement was related to recognition memory for the object and background scene components. This effect was generally comparable for negative and neutral memories, although the effects of valence varied somewhat between the two experiments. Together, these findings suggest that gaze reinstatement occurs independently of the processes contributing to the emotion-induced memory trade-off effect.

Image Object and Scene Recognition Based on Improved Convolutional Neural Network

In recent years, due to the continuous optimization of network structure and the emergence of large-scale data, Convolutional neural network has made breakthroughs in a series of applications of computer vision. Based on this, the Convolutional neural network is improved and optimized. The improved convolutional neural network is introduced into image Object detection and scene recognition, and image object detection is carried out by combining sliding window fusion and Convolutional neural network. The image scene recognition model is constructed by using potential object area recognition and Convolutional neural network Transfer learning. Using different data sets to verify the algorithm, the research results show that in Group1 and Group2, the error rate of the multi column convolutional neural network fused by sliding window is reduced by about 25% compared with the single column convolutional neural network. As the group with the smallest decrease in error rate, Group3 also achieved a 9% decrease in error rate. The fitness rate of object detection algorithm is gradually stable after 7 runs, reaching about 9.8%, and its operation effect is obviously better than other algorithms. The multi column convolutional neural network fused by sliding window is more adaptive to the training data set, and gets better recognition effect in the algorithm operation. However, the image scene recognition model based on potential object area recognition algorithm and Convolutional neural network has good convergence. The average recognition time for image scenes is 1.5356s. The recognition speed is fast and stable, which can effectively solve the problem of multi-scale image scene recognition.

Point-to-Pixel Prompting for Point Cloud Analysis With Pre-Trained Image Models.

Nowadays, pre-training big models on large-scale datasets has achieved great success and dominated many downstream tasks in natural language processing and 2D vision, while pre-training in 3D vision is still under development. In this paper, we provide a new perspective of transferring the pre-trained knowledge from 2D domain to 3D domain with Point-to-Pixel Prompting in data space and Pixel-to-Point distillation in feature space, exploiting shared knowledge in images and point clouds that display the same visual world. Following the principle of prompting engineering, Point-to-Pixel Prompting transforms point clouds into colorful images with geometry-preserved projection and geometry-aware coloring. Then the pre-trained image models can be directly implemented for point cloud tasks without structural changes or weight modifications. With projection correspondence in feature space, Pixel-to-Point distillation further regards pre-trained image models as the teacher model and distills pre-trained 2D knowledge to student point cloud models, remarkably enhancing inference efficiency and model capacity for point cloud analysis. We conduct extensive experiments in both object classification and scene segmentation under various settings to demonstrate the superiority of our method. In object classification, we reveal the important scale-up trend of Point-to-Pixel Prompting and attain 90.3% accuracy on ScanObjectNN dataset, surpassing previous literature by a large margin. In scene-level semantic segmentation, our method outperforms traditional 3D analysis approaches and shows competitive capacity in dense prediction tasks.

PIG: Prompt Images Guidance for Night-Time Scene Parsing.

Night-time scene parsing aims to extract pixel-level semantic information in night images, aiding downstream tasks in understanding scene object distribution. Due to limited labeled night image datasets, unsupervised domain adaptation (UDA) has become the predominant method for studying night scenes. UDA typically relies on paired day-night image pairs to guide adaptation, but this approach hampers dataset construction and restricts generalization across night scenes in different datasets. Moreover, UDA, focusing on network architecture and training strategies, faces difficulties in handling classes with few domain similarities. In this paper, we leverage Prompt Images Guidance (PIG) to enhance UDA with supplementary night knowledge. We propose a Night-Focused Network (NFNet) to learn night-specific features from both target domain images and prompt images. To generate high-quality pseudo-labels, we propose Pseudo-label Fusion via Domain Similarity Guidance (FDSG). Classes with fewer domain similarities are predicted by NFNet, which excels in parsing night features, while classes with more domain similarities are predicted by UDA, which has rich labeled semantics. Additionally, we propose two data augmentation strategies: the Prompt Mixture Strategy (PMS) and the Alternate Mask Strategy (AMS), aimed at mitigating the overfitting of the NFNet to a few prompt images. We conduct extensive experiments on four night-time datasets: NightCity, NightCity+, Dark Zurich, and ACDC. The results indicate that utilizing PIG can enhance the parsing accuracy of UDA. The code is available at https://github.com/qiurui4shu/PIG.

Exploring Ragdoll Physics in Crowd Disaster Simulation

Abstract Objective This study explores the feasibility of applying ragdoll physics, a procedural animation, and related game-engine technologies to create real-time simulations for investigating crowd disasters. Methods We harnessed the power of the Unity Game Engine to develop real-time simulation for the dynamics of a crowd disaster with rendering of scene objects and user interface (UI) elements. Humanoid agents with trigger colliders were linked to color schemes representing impact forces sustained with lethal thresholds ranging from 500 to 550 pounds force for males and 450 to 500 pounds force for females. The simulation enables users to manipulate parameters such as population, crowd density, directional movement and forces applied. Results Our simulation showcases an animated real-time depiction of a crowd disaster and enables visualization of casualties, both injured and dead, based upon a quantitative evaluation of the forces sustained. Conclusions Utilization of the Unity Game Engine for the development of visually-engaging real-time crowd disaster simulation has promising outcomes. This novel approach for modeling and analysis of crowd dynamics may provide valuable insights with applications in mass gatherings, crowd management, safety, and preparedness.

Semi-automatic A.I.-Specified Stereographic Scene Rendering Techniques Across Various Displays

Semi-automatic, A.I.-specified stereographic scene processing is achieved by extending Blender with ChatGPT. We can generate objects and set scenes in Blender semi-automatically by invoking appropriate text prompts. Stereography techniques include anaglyphy, chromastereoscopy, and side-by-side display using binocular viewing. One can create a similar scene by copying and pasting the prompt used. It cannot always work identically, but it behaves roughly the same. First, we established Skybox and added scene Objects (Camera, Cube, etc.) in 3D view, materials, and animation with prompt. Stereographic images are created semiautomatically and compared across some qualities. The goal of this research is to validate those various stereographic techniques as rendered automatically in Blender.

Attention Network with Outdoor Illumination Variation Prior for Spectral Reconstruction from RGB Images

Hyperspectral images (HSIs) are widely used to identify and characterize objects in scenes of interest, but they are associated with high acquisition costs and low spatial resolutions. With the development of deep learning, HSI reconstruction from low-cost and high-spatial-resolution RGB images has attracted widespread attention. It is an inexpensive way to obtain HSIs via the spectral reconstruction (SR) of RGB data. However, due to a lack of consideration of outdoor solar illumination variation in existing reconstruction methods, the accuracy of outdoor SR remains limited. In this paper, we present an attention neural network based on an adaptive weighted attention network (AWAN), which considers outdoor solar illumination variation by prior illumination information being introduced into the network through a basic 2D block. To verify our network, we conduct experiments on our Variational Illumination Hyperspectral (VIHS) dataset, which is composed of natural HSIs and corresponding RGB and illumination data. The raw HSIs are taken on a portable HS camera, and RGB images are resampled directly from the corresponding HSIs, which are not affected by illumination under CIE-1964 Standard Illuminant. Illumination data are acquired with an outdoor illumination measuring device (IMD). Compared to other methods and the reconstructed results not considering solar illumination variation, our reconstruction results have higher accuracy and perform well in similarity evaluations and classifications using supervised and unsupervised methods.

Memanifestasi Deiksis dalam Film “Ngeri-Ngeri Sedap” Karya Bene Dion Rajagukguk: Analisis Pragmatik

This research aims to analyze the use of deixis in the film "Ngeri-Ngeri Sedap" from a pragmatic perspective. Deixis is an important aspect in linguistic studies which is related to the use of signs that refer to the context of space and time. The film text analysis method is used to identify and analyze the use of deixis in the context of situations involving characters, location, time, and interactions between characters. The research results show that the use of deixis in this film aims to create a communicative effect and build relationships between the characters and the audience, by using personal, time, place, atmosphere, and social deixis. The use of pointing signs, such as body gestures and the placement of objects in scenes, is used to clarify the spatial relationships between characters and objects in the story. Pronouns are also often used to replace previously introduced references, maintaining cohesion in the narrative. In addition, the effective use of time-telling words helps the audience understand the sequence of events and establish chronology in the story. This research provides deeper insight into the use of language in communicative situations in the context of films, as well as a contribution to the development of pragmatic theory and an understanding of the audience's experience.

Animation Scene Object Recognition and Modeling Based on Computer Vision Technology

Animation Scene Object Recognition and Modeling Based on Computer Vision Technology

Monocular 3D object detection for construction scene analysis

AbstractThree‐dimensional (3D) object detection, that is, localizing and classifying all critical objects in a 3D space, is essential for downstream construction scene analysis tasks. However, accurate instance segmentation, few 2D object segmentation and 3D object detection data sets, high‐quality feature representations for depth estimation, and limited 3D cues from a single red‐green‐blue (RGB) image pose significant challenges to 3D object detection and severely hinder its practical applications. In response to these challenges, an improved cascade‐based network with a transformer backbone and a boundary‐patch‐refinement method is proposed to build hierarchical features and refine object boundaries, resulting in better results in 2D object detection and instance segmentation. Furthermore, a novel self‐supervised monocular depth learning method is proposed to extract better feature representations for depth estimation from construction site video data with unknown camera parameters. Additionally, a pseudo‐LiDAR point cloud method and a 3D object detection method with a density‐based clustering algorithm are proposed to detect 3D objects in a construction scene without help from 3D labels, which will serve as a good foundation for other downstream 3D tasks. Finally, the proposed model is evaluated for object instance segmentation and depth estimation on the moving objects in construction sites (MOCS) and construction scene data sets. It brings a 9.16% gain in terms of mean average precision (mAP) for object detection and a 4.92% gain in mask mAP for object instance segmentation. The average order accuracy and relative mean error for depth estimation are improved by 0.94% and 60.56%, respectively. This study aims to overcome the challenges and limitations of 3D object detection and facilitate practical applications in construction scene analysis.

Enhanced visual SLAM for construction robots by efficient integration of dynamic object segmentation and scene semantics

With the increasing adoption of autonomous mobile robots in the construction industry, accurate localization and mapping in dynamic construction environments have become paramount. This is typically tackled via Simultaneous Localization and Mapping (SLAM) techniques. Primarily designed for static environments, traditional SLAM systems struggle to maintain robustness and accuracy in dynamic settings. To address this challenge, this study presents an enhanced visual SLAM system specifically tailored for dynamic construction environments. The proposed system, named vSLAM-Con, introduces an adaptive dynamic object segmentation method, utilizing an innovative AD-keyframes selection mechanism grounded on optical flow magnitude to diminish computational overhead while preserving competitive tracking accuracy. Additionally, a semantic-based feature update process is developed, leveraging scene understanding and continuous observation to augment the reliability of tracking features. This system's performance, evaluated on both an established public benchmark and a custom construction dataset, shows substantial improvements over the baseline and competitive results with the state-of-the-art algorithms. More importantly, it largely reduces the processing time compared to state-of-the-arts, demonstrating robust tracking performance even under highly dynamic conditions. The findings highlight the system's potential to contribute significantly to autonomous robotics in construction, offering more accurate navigation and interaction capabilities in complex, ever-changing environments.

Automated Reconstruction of Existing Building Interior Scene BIMs Using a Feature-Enhanced Point Transformer and an Octree

Building information models (BIMs) offer advantages, such as visualization and collaboration, making them widely used in the management of existing buildings. Currently, most BIMs for existing indoor spaces are manually created, consuming a significant amount of manpower and time, severely impacting the efficiency of building operations and maintenance management. To address this issue, this study proposes an automated reconstruction method for an indoor scene BIM based on a feature-enhanced point transformer and an octree. This method enhances the semantic segmentation performance of point clouds by using feature position encoding to strengthen the point transformer network. Subsequently, the data are partitioned into multiple segments using an octree, collecting the geometric and spatial information of individual objects in the indoor scene. Finally, the BIM is automatically reconstructed using Dynamo in Revit. The research results indicate that the proposed feature-enhanced point transformer algorithm achieves a high segmentation accuracy of 71.3% mIoU on the S3DIS dataset. The BIM automatically generated from the field point cloud data, when compared to the original data, has an average error of ±1.276 mm, demonstrating a good reconstruction quality. This method achieves the high-precision, automated reconstruction of the indoor BIM for existing buildings, avoiding extensive manual operations and promoting the application of BIMs for the maintenance processes of existing buildings.

Moving scene object tracking method based on deep convolutional neural network

The effect of target tracking is not ideal when facing various complex tracking scenarios such as non-rigid deformation of target, frequent occlusion, clutter of target background and interference of similar objects. In this paper, the feature based on deep convolutional neural network is used for target tracking in moving scenes, and a sliding window target segmentation method is proposed to study the impact of data normalization and data set expansion on the final result. In order to select more distinguishing features, principal component analysis is used to process the features of Deep Convolution Neural Network (DCNN), and the features of different network layers of DCNN are compared. The feature coding algorithm is studied, and the extracted DCNN features are encoded by Fisher Vectors algorithm, and compared with the locality-constrained linear encoding technique. Experiments show that the feature based on deep convolutional neural network in this paper can obtain higher accuracy than the traditional feature fusion method. According to the result analysis, the tracking accuracy of deep convolutional neural network algorithm is improved under the condition of illumination variation. In the case of local occlusion, the tracking accuracy is also improved.

ScaNeRF: Scalable Bundle-Adjusting Neural Radiance Fields for Large-Scale Scene Rendering

High-quality large-scale scene rendering requires a scalable representation and accurate camera poses. This research combines tile-based hybrid neural fields with parallel distributive optimization to improve bundle-adjusting neural radiance fields. The proposed method scales with a divide-and-conquer strategy. We partition scenes into tiles, each with a multi-resolution hash feature grid and shallow chained diffuse and specular multilayer perceptrons (MLPs). Tiles unify foreground and background via a spatial contraction function that allows both distant objects in outdoor scenes and planar reflections as virtual images outside the tile. Decomposing appearance with the specular MLP allows a specular-aware warping loss to provide a second optimization path for camera poses. We apply the alternating direction method of multipliers (ADMM) to achieve consensus among camera poses while maintaining parallel tile optimization. Experimental results show that our method outperforms state-of-the-art neural scene rendering method quality by 5%--10% in PSNR, maintaining sharp distant objects and view-dependent reflections across six indoor and outdoor scenes.

DIRECT IMAGE RECONSTRUCTION IN MULTI-ELEMENT INTERFEROMETRY

The Ukrainian VLBI system of decameter radio telescopes URAN successfully solves many scientific problems, but implementing aperture synthesis technology has a number of difficulties. One of them is significant phase distortions in this range caused by an inhomogeneous propagation environment. Therefore, the task arose to develop an alternative technology with the conventional name "Interferovision", which would allow us to restore radio images with a limited number of antennas, operate with broadband signals, not require flatness of the objects scene, and have an extended field of view. A method of direct image reconstruction when observing objects in space, using a multi-element interferometer, is proposed. This method is based on a physically based principle that is similar to holography. The wave front is registered by the antennas of the interferometer, and further processing is equivalent to its playback in reverse and registration of the resulting spatial interference image. There are no special requirements for the radiation of individual points of the source, except for its delta correlation. A finite frequency band is considered, and each point can be characterized by its own spectrum, that is, its own autocorrelation function of radiation. The resolution of the direct reconstruction method depends on its width. With the quasi-monochromatic approximation, the autocorrelation functions of the radiation of all the source points degenerate into sinusoids, and the restoration of the image becomes possible only with the use of the Fourier transformation. A theoretical justification of the method for spaces of different dimensions has been obtained. Interferometric systems of the same rank with many antennas are reduced to a same canonical form with fixed number of virtual antennas placed at the origin of the coordinates and at unit distances on the coordinate axes. The ambiguity of the obtained solution is eliminated by using an additional antenna. A simulation of the proposed method of direct image reconstruction for two-dimensional space was carried out. Despite the low conditionality of the system for estimating the distance, when it increases, the angular characteristics are preserved. Therefore, the method is promising for restoring radio images of space radio sources.

Visadapt: Catadioptric adaptive camera for scenes of variable density of visual information

This paper presents the design method of a multi-resolution camera, named Visadapt. It is made of a conventional compact camera with a sensor and a lens pointed to a new deformable mirror so that the mirror in a flat state is parallel to the image plane. The main novelty of the latter mirror, to our knowledge, is the ability to control automatically strokes of several millimeters. This allows Visadapt to capture scenes with a spatially variable density of visual information. A grid of linear actuators, set underneath the mirror surface, deforms the mirror to reach the desired shape computed to capture several areas of different resolutions. Mechanical simulations are allowed to iterate on Visadapt’s design, to reduce the geometrical distortions in the images. Evaluations made on an actual prototype of Visadapt show that, by adapting the mirror shape, this camera can magnify a scene object up to 20%, even off-centered in the field-of-view, while still perceiving the rest of the scene.

Neural Radiance Fields-Based 3D Reconstruction of Power Transmission Lines Using Progressive Motion Sequence Images

To address the fuzzy reconstruction effect on distant objects in unbounded scenes and the difficulty in feature matching caused by the thin structure of power lines in images, this paper proposes a novel image-based method for the reconstruction of power transmission lines (PTLs). The dataset used in this paper comprises PTL progressive motion sequence datasets, constructed by a visual acquisition system carried by a developed Flying–walking Power Line Inspection Robot (FPLIR). This system captures close-distance and continuous images of power lines. The study introduces PL-NeRF, that is, an enhanced method based on the Neural Radiance Fields (NeRF) method for reconstructing PTLs. The highlights of PL-NeRF include (1) compressing the unbounded scene of PTLs by exploiting the spatial compression of normal L∞; (2) encoding the direction and position of the sample points through Integrated Position Encoding (IPE) and Hash Encoding (HE), respectively. Compared to existing methods, the proposed method demonstrates good performance in 3D reconstruction, with fidelity indicators of PSNR = 29, SSIM = 0.871, and LPIPS = 0.087. Experimental results highlight that the combination of PL-NeRF with progressive motion sequence images ensures the integrity and continuity of PTLs, improving the efficiency and accuracy of image-based reconstructions. In the future, this method could be widely applied for efficient and accurate 3D reconstruction and inspection of PTLs, providing a strong foundation for automated monitoring of transmission corridors and digital power engineering.