Objects In Scene Research Articles

Attention Network with Outdoor Illumination Variation Prior for Spectral Reconstruction from RGB Images

Hyperspectral images (HSIs) are widely used to identify and characterize objects in scenes of interest, but they are associated with high acquisition costs and low spatial resolutions. With the development of deep learning, HSI reconstruction from low-cost and high-spatial-resolution RGB images has attracted widespread attention. It is an inexpensive way to obtain HSIs via the spectral reconstruction (SR) of RGB data. However, due to a lack of consideration of outdoor solar illumination variation in existing reconstruction methods, the accuracy of outdoor SR remains limited. In this paper, we present an attention neural network based on an adaptive weighted attention network (AWAN), which considers outdoor solar illumination variation by prior illumination information being introduced into the network through a basic 2D block. To verify our network, we conduct experiments on our Variational Illumination Hyperspectral (VIHS) dataset, which is composed of natural HSIs and corresponding RGB and illumination data. The raw HSIs are taken on a portable HS camera, and RGB images are resampled directly from the corresponding HSIs, which are not affected by illumination under CIE-1964 Standard Illuminant. Illumination data are acquired with an outdoor illumination measuring device (IMD). Compared to other methods and the reconstructed results not considering solar illumination variation, our reconstruction results have higher accuracy and perform well in similarity evaluations and classifications using supervised and unsupervised methods.

Memanifestasi Deiksis dalam Film “Ngeri-Ngeri Sedap” Karya Bene Dion Rajagukguk: Analisis Pragmatik

This research aims to analyze the use of deixis in the film "Ngeri-Ngeri Sedap" from a pragmatic perspective. Deixis is an important aspect in linguistic studies which is related to the use of signs that refer to the context of space and time. The film text analysis method is used to identify and analyze the use of deixis in the context of situations involving characters, location, time, and interactions between characters. The research results show that the use of deixis in this film aims to create a communicative effect and build relationships between the characters and the audience, by using personal, time, place, atmosphere, and social deixis. The use of pointing signs, such as body gestures and the placement of objects in scenes, is used to clarify the spatial relationships between characters and objects in the story. Pronouns are also often used to replace previously introduced references, maintaining cohesion in the narrative. In addition, the effective use of time-telling words helps the audience understand the sequence of events and establish chronology in the story. This research provides deeper insight into the use of language in communicative situations in the context of films, as well as a contribution to the development of pragmatic theory and an understanding of the audience's experience.

Animation Scene Object Recognition and Modeling Based on Computer Vision Technology

Animation Scene Object Recognition and Modeling Based on Computer Vision Technology

Monocular 3D object detection for construction scene analysis

AbstractThree‐dimensional (3D) object detection, that is, localizing and classifying all critical objects in a 3D space, is essential for downstream construction scene analysis tasks. However, accurate instance segmentation, few 2D object segmentation and 3D object detection data sets, high‐quality feature representations for depth estimation, and limited 3D cues from a single red‐green‐blue (RGB) image pose significant challenges to 3D object detection and severely hinder its practical applications. In response to these challenges, an improved cascade‐based network with a transformer backbone and a boundary‐patch‐refinement method is proposed to build hierarchical features and refine object boundaries, resulting in better results in 2D object detection and instance segmentation. Furthermore, a novel self‐supervised monocular depth learning method is proposed to extract better feature representations for depth estimation from construction site video data with unknown camera parameters. Additionally, a pseudo‐LiDAR point cloud method and a 3D object detection method with a density‐based clustering algorithm are proposed to detect 3D objects in a construction scene without help from 3D labels, which will serve as a good foundation for other downstream 3D tasks. Finally, the proposed model is evaluated for object instance segmentation and depth estimation on the moving objects in construction sites (MOCS) and construction scene data sets. It brings a 9.16% gain in terms of mean average precision (mAP) for object detection and a 4.92% gain in mask mAP for object instance segmentation. The average order accuracy and relative mean error for depth estimation are improved by 0.94% and 60.56%, respectively. This study aims to overcome the challenges and limitations of 3D object detection and facilitate practical applications in construction scene analysis.

Enhanced visual SLAM for construction robots by efficient integration of dynamic object segmentation and scene semantics

With the increasing adoption of autonomous mobile robots in the construction industry, accurate localization and mapping in dynamic construction environments have become paramount. This is typically tackled via Simultaneous Localization and Mapping (SLAM) techniques. Primarily designed for static environments, traditional SLAM systems struggle to maintain robustness and accuracy in dynamic settings. To address this challenge, this study presents an enhanced visual SLAM system specifically tailored for dynamic construction environments. The proposed system, named vSLAM-Con, introduces an adaptive dynamic object segmentation method, utilizing an innovative AD-keyframes selection mechanism grounded on optical flow magnitude to diminish computational overhead while preserving competitive tracking accuracy. Additionally, a semantic-based feature update process is developed, leveraging scene understanding and continuous observation to augment the reliability of tracking features. This system's performance, evaluated on both an established public benchmark and a custom construction dataset, shows substantial improvements over the baseline and competitive results with the state-of-the-art algorithms. More importantly, it largely reduces the processing time compared to state-of-the-arts, demonstrating robust tracking performance even under highly dynamic conditions. The findings highlight the system's potential to contribute significantly to autonomous robotics in construction, offering more accurate navigation and interaction capabilities in complex, ever-changing environments.

Automated Reconstruction of Existing Building Interior Scene BIMs Using a Feature-Enhanced Point Transformer and an Octree

Building information models (BIMs) offer advantages, such as visualization and collaboration, making them widely used in the management of existing buildings. Currently, most BIMs for existing indoor spaces are manually created, consuming a significant amount of manpower and time, severely impacting the efficiency of building operations and maintenance management. To address this issue, this study proposes an automated reconstruction method for an indoor scene BIM based on a feature-enhanced point transformer and an octree. This method enhances the semantic segmentation performance of point clouds by using feature position encoding to strengthen the point transformer network. Subsequently, the data are partitioned into multiple segments using an octree, collecting the geometric and spatial information of individual objects in the indoor scene. Finally, the BIM is automatically reconstructed using Dynamo in Revit. The research results indicate that the proposed feature-enhanced point transformer algorithm achieves a high segmentation accuracy of 71.3% mIoU on the S3DIS dataset. The BIM automatically generated from the field point cloud data, when compared to the original data, has an average error of ±1.276 mm, demonstrating a good reconstruction quality. This method achieves the high-precision, automated reconstruction of the indoor BIM for existing buildings, avoiding extensive manual operations and promoting the application of BIMs for the maintenance processes of existing buildings.

Moving scene object tracking method based on deep convolutional neural network

The effect of target tracking is not ideal when facing various complex tracking scenarios such as non-rigid deformation of target, frequent occlusion, clutter of target background and interference of similar objects. In this paper, the feature based on deep convolutional neural network is used for target tracking in moving scenes, and a sliding window target segmentation method is proposed to study the impact of data normalization and data set expansion on the final result. In order to select more distinguishing features, principal component analysis is used to process the features of Deep Convolution Neural Network (DCNN), and the features of different network layers of DCNN are compared. The feature coding algorithm is studied, and the extracted DCNN features are encoded by Fisher Vectors algorithm, and compared with the locality-constrained linear encoding technique. Experiments show that the feature based on deep convolutional neural network in this paper can obtain higher accuracy than the traditional feature fusion method. According to the result analysis, the tracking accuracy of deep convolutional neural network algorithm is improved under the condition of illumination variation. In the case of local occlusion, the tracking accuracy is also improved.

ScaNeRF: Scalable Bundle-Adjusting Neural Radiance Fields for Large-Scale Scene Rendering

High-quality large-scale scene rendering requires a scalable representation and accurate camera poses. This research combines tile-based hybrid neural fields with parallel distributive optimization to improve bundle-adjusting neural radiance fields. The proposed method scales with a divide-and-conquer strategy. We partition scenes into tiles, each with a multi-resolution hash feature grid and shallow chained diffuse and specular multilayer perceptrons (MLPs). Tiles unify foreground and background via a spatial contraction function that allows both distant objects in outdoor scenes and planar reflections as virtual images outside the tile. Decomposing appearance with the specular MLP allows a specular-aware warping loss to provide a second optimization path for camera poses. We apply the alternating direction method of multipliers (ADMM) to achieve consensus among camera poses while maintaining parallel tile optimization. Experimental results show that our method outperforms state-of-the-art neural scene rendering method quality by 5%--10% in PSNR, maintaining sharp distant objects and view-dependent reflections across six indoor and outdoor scenes.

DIRECT IMAGE RECONSTRUCTION IN MULTI-ELEMENT INTERFEROMETRY

The Ukrainian VLBI system of decameter radio telescopes URAN successfully solves many scientific problems, but implementing aperture synthesis technology has a number of difficulties. One of them is significant phase distortions in this range caused by an inhomogeneous propagation environment. Therefore, the task arose to develop an alternative technology with the conventional name "Interferovision", which would allow us to restore radio images with a limited number of antennas, operate with broadband signals, not require flatness of the objects scene, and have an extended field of view. A method of direct image reconstruction when observing objects in space, using a multi-element interferometer, is proposed. This method is based on a physically based principle that is similar to holography. The wave front is registered by the antennas of the interferometer, and further processing is equivalent to its playback in reverse and registration of the resulting spatial interference image. There are no special requirements for the radiation of individual points of the source, except for its delta correlation. A finite frequency band is considered, and each point can be characterized by its own spectrum, that is, its own autocorrelation function of radiation. The resolution of the direct reconstruction method depends on its width. With the quasi-monochromatic approximation, the autocorrelation functions of the radiation of all the source points degenerate into sinusoids, and the restoration of the image becomes possible only with the use of the Fourier transformation. A theoretical justification of the method for spaces of different dimensions has been obtained. Interferometric systems of the same rank with many antennas are reduced to a same canonical form with fixed number of virtual antennas placed at the origin of the coordinates and at unit distances on the coordinate axes. The ambiguity of the obtained solution is eliminated by using an additional antenna. A simulation of the proposed method of direct image reconstruction for two-dimensional space was carried out. Despite the low conditionality of the system for estimating the distance, when it increases, the angular characteristics are preserved. Therefore, the method is promising for restoring radio images of space radio sources.

Visadapt: Catadioptric adaptive camera for scenes of variable density of visual information

This paper presents the design method of a multi-resolution camera, named Visadapt. It is made of a conventional compact camera with a sensor and a lens pointed to a new deformable mirror so that the mirror in a flat state is parallel to the image plane. The main novelty of the latter mirror, to our knowledge, is the ability to control automatically strokes of several millimeters. This allows Visadapt to capture scenes with a spatially variable density of visual information. A grid of linear actuators, set underneath the mirror surface, deforms the mirror to reach the desired shape computed to capture several areas of different resolutions. Mechanical simulations are allowed to iterate on Visadapt’s design, to reduce the geometrical distortions in the images. Evaluations made on an actual prototype of Visadapt show that, by adapting the mirror shape, this camera can magnify a scene object up to 20%, even off-centered in the field-of-view, while still perceiving the rest of the scene.

Neural Radiance Fields-Based 3D Reconstruction of Power Transmission Lines Using Progressive Motion Sequence Images

To address the fuzzy reconstruction effect on distant objects in unbounded scenes and the difficulty in feature matching caused by the thin structure of power lines in images, this paper proposes a novel image-based method for the reconstruction of power transmission lines (PTLs). The dataset used in this paper comprises PTL progressive motion sequence datasets, constructed by a visual acquisition system carried by a developed Flying–walking Power Line Inspection Robot (FPLIR). This system captures close-distance and continuous images of power lines. The study introduces PL-NeRF, that is, an enhanced method based on the Neural Radiance Fields (NeRF) method for reconstructing PTLs. The highlights of PL-NeRF include (1) compressing the unbounded scene of PTLs by exploiting the spatial compression of normal L∞; (2) encoding the direction and position of the sample points through Integrated Position Encoding (IPE) and Hash Encoding (HE), respectively. Compared to existing methods, the proposed method demonstrates good performance in 3D reconstruction, with fidelity indicators of PSNR = 29, SSIM = 0.871, and LPIPS = 0.087. Experimental results highlight that the combination of PL-NeRF with progressive motion sequence images ensures the integrity and continuity of PTLs, improving the efficiency and accuracy of image-based reconstructions. In the future, this method could be widely applied for efficient and accurate 3D reconstruction and inspection of PTLs, providing a strong foundation for automated monitoring of transmission corridors and digital power engineering.

MSA R-CNN: A comprehensive approach to remote sensing object detection and scene understanding

Remote sensing scene understanding is crucial for extracting valuable information from high-resolution images, including object detection and classification. Traditional object detection methods face challenges in handling the diverse scales, orientations, and complex backgrounds present in remote sensing data. In this paper, we propose a novel remote sensing scene understanding system called multiscale-attention R-CNN (MSA R-CNN), which incorporates a super multiscale feature extraction network (SMENet) for enhanced feature extraction from multiscale images, an adaptive dynamic inner lateral (ADIL) connection module to tackle information loss in feature pyramid networks (FPN), and a distributed lightweight attention module (DLAM) to refine feature information processing. Furthermore, a new dataset combining the DIOR and DOTA datasets is introduced to extract the background information of detected objects and evaluate the proposed system’s performance. MSA R-CNN achieved an mAP of 74.37% on the DIOR dataset when the gamma value was set to 0.2 and 81.97% on the DOTA dataset when the gamma value was set to 0.1 with the same learning rate, outperforming state-of-the-art models on both datasets. The proposed system demonstrates significant improvements in both object detection and background information extraction, providing a comprehensive solution for remote sensing scene understanding.

600-GHz Fourier imaging based on heterodyne detection at the 2nd sub-harmonic.

Fourier imaging is an indirect imaging method which records the diffraction pattern of the object scene coherently in the focal plane of the imaging system and reconstructs the image using computational resources. The spatial resolution, which can be reached, depends on one hand on the wavelength of the radiation, but also on the capability to measure - in the focal plane - Fourier components with high spatial wave-vectors. This leads to a conflicting situation at THz frequencies, because choosing a shorter wavelength for better resolution usually comes at the cost of less radiation power, concomitant with a loss of dynamic range, which limits the detection of higher Fourier components. Here, aiming at maintaining a high dynamic range and limiting the system costs, we adopt heterodyne detection at the 2nd sub-harmonic, working with continuous-wave (CW) radiation for object illumination at 600 GHz and local-oscillator (LO) radiation at 300 GHz. The detector is a single-pixel broad-band Si CMOS TeraFET equipped with substrate lenses on both the front- and backside for separate in-coupling of the waves. The entire scene is illuminated by the object wave, and the Fourier spectrum is recorded by raster scanning of the single-detector unit through the focal plane. With only 56 µW of power of the 600-GHz radiation, a dynamic range of 60 dB is reached, sufficient to detect the entire accessible Fourier space spectrum in the test measurements. We present a detailed comparison between plane-to-plane imaging and Fourier imaging, and show that, with both, a lateral spatial resolution of better than 0.5 mm, at the diffraction limit, is reached.

Cross-domain point cloud completion for multi-class indoor incomplete objects via class-conditional GAN inversion

Indoor point clouds from real-world scans are often incomplete and sparse due to limited observation views and severe occlusion between objects. Point cloud completion can restore missing parts to ensure data integrity, so that subsequent tasks like point cloud classification, segmentation, object detection and three-dimensional (3D) reconstruction can obtain more accurate and complete geometric and semantic information. Existing point cloud completion methods focus more on synthetic data or require separate modeling on each class of objects, which struggle to meet the inherent requirements of the completion task for real-world indoor scenes containing multi-class objects with significant pose variations. In this research, we present an efficient class-conditional generative adversarial network (GAN) inversion framework for the task of recovering incomplete objects in real-world indoor scenes, in which we first train a class-conditional GAN to learn rich shape priors for multi-class objects from synthetic data domain and then find an optimal latent code in GAN’s latent space that gives a complete shape that best recovers the given incomplete input from real-world data domain. The performance of our proposed method is evaluated on both real-world and synthetic datasets. In comparison to ShapeInversion (Zhang et al., 2021), our method demonstrates an improvement in F-score by 3.7 and 2.0 percentage points while reducing the chamfer distance (CD) by 4.15 and 1.1 on ScanNet dataset and CRN dataset, respectively. The developed code will be publicly available at: https://github.com/CNU-DLandCV-lab/cgan-inversion.

High-quality 3D shape recovery from scattering scenario via deep polarization neural networks

High-quality three-dimension(3D) shape recovery from scattering scenario is of great importance with various potential applications. However, in turbid water scenarios, the absorption and scattering by water and suspending particles have a great disturbance on the generation of image clarity, texture details, and pixel-wise polarized properties such as degree of polarization (DoP) and angle of polarization (AoP), which significantly affect the 3D reconstructed performance. In this study, we present a data-driven polarimetric method for 3D reconstruction of underwater objects in turbid scenes, leveraging deep learning image priors. An underwater descattering network is designed to fit the underwater transport function of target signal light and generate clear underwater images, which are utilized to enhance the texture and remove the backscattering light. Subsequently, accurate pixel-wise DoP and AoP of the object could be calculated using the clear image prior and underwater imaging model. Finally, utilizing the obtained clear image and polarimetric information, polarimetric 3D reconstruction of object in turbid underwater scenarios is achieved. Experimental results demonstrate that accurate 3D shape of objects in turbid water can be well-reconstructed. The proposed technique has great potential for advancing marine exploration research.

Semantic object processing is modulated by prior scene context

ABSTRACT Objects that are congruent with a scene are recognised more efficiently than objects that are incongruent. Further, semantic integration of incongruent objects elicits a stronger N300/N400 EEG component. Yet, the time course and mechanisms of how contextual information supports access to semantic object information is unclear. We used computational modelling and EEG to test how context influences semantic object processing. Using representational similarity analysis, we established that EEG patterns dissociated between objects in congruent or incongruent scenes from around 300 ms. By modelling the semantic processing of objects using independently normed properties, we confirm that the onset of semantic processing of both congruent and incongruent objects is similar (∼150 ms). Critically, after ∼275 ms, we discover a difference in the duration of semantic integration, lasting longer for incongruent compared to congruent objects. These results constrain our understanding of how contextual information supports access to semantic object information.

Visual coherence in augmented reality training systems considering aerospace specific features

In May 2022, Saudi Arabian Military Industries, a Saudi government agency, acquired an augmented reality training platform for pilots. In September, the Boeing Corporation began the development of an augmented reality pilot simulator. In November, a similar project was launched by BAE Systems, a leading British developer of aeronautical engineering. These facts allow us to confidently speak about the beginning of a new era of aviation simulators – simulators using the augmented reality technology. One of the promising advantages of this technology is the ability to safely simulate dangerous situations in the real world. A necessary condition for using this advantage is to ensure the visual coherence of augmented reality scenes: virtual objects must be indistinguishable from real ones. All the global IT leaders consider augmented reality as the subsequent surge of radical changes in digital electronics, so visual coherence is becoming a key issue for the future of IT, and in aerospace applications, visual coherence has already acquired practical significance. The Russian Federation lags far behind in studying the problems of visual coherence in general and for augmented reality flight simulators in particular: at the time of publication the authors managed to find only two papers on the subject in the Russian research space, while abroad their number is already approximately a thousand. The purpose of this review article is to create conditions for solving the problem. Visual coherence depends on many factors: lighting, color tone, shadows from virtual objects on real ones, mutual reflections, textures of virtual surfaces, optical aberrations, convergence and accommodation, etc. The article reviews the publications devoted to methods for assessing the conditions of illumination and color tone of a real scene and transferring them to virtual objects using various probes and by individual images, as well as by rendering virtual objects in augmented reality scenes, using neural networks.

Multi-Scale Indoor Scene Geometry Modeling Algorithm Based on Segmentation Results

Due to the numerous objects with regular structures in indoor environments, identifying and modeling the regular objects in scenes aids indoor robots in sensing unknown environments. Typically, point cloud preprocessing can obtain highly complete object segmentation results in scenes which can be utilized as the objects for geometric analysis and modeling, thus ensuring modeling accuracy and speed. However, due to the lack of a complete object model, it is not possible to recognize and model segmented objects through matching methods. To achieve a greater understanding of scene point clouds, this paper proposes a direct geometric modeling algorithm based on segmentation results, which focuses on extracting regular geometries in the scene, rather than objects with geometric details or combinations of multiple primitives. This paper suggests using simpler geometric models to describe the corresponding point cloud data. By fully utilizing the surface structure information of segmented objects, the paper analyzes the types of faces and their relationships to classify regular geometric objects into two categories: planar and curved. Different types of geometric objects are fitted using random sampling consistency algorithms with type classification results as prior knowledge, and segmented results are modeled through a combination of size information associated with directed bounding boxes. For indoor scenes with occlusion and stacking, utilizing a higher-level semantic expression can effectively simplify the scene, complete scene abstraction and structural modeling, and aid indoor robots’ understanding and further operation in unknown environments.

Self-training based augmented reality for robust 3D object registration and task assistance

Typical augmented reality (AR) technologies superimpose virtual information onto the real environment for task guidance and assistance by tracking fiducial markers. However, these have narrow recognition ranges due to the markers, and it is difficult to track markers when occlusion occurs. Therefore, it is challenging to seamlessly register real objects in AR scenes captured by the camera without requiring fiducial markers. Deep learning–based approaches to 3D object registration and task assistance have recently indicated promising directions for superimposing 3D virtual objects onto corresponding real objects or their surroundings seamlessly. However, since supervised learning-based object registration requires many pose annotations on images for the locations and orientations of real objects, this process is a time-consuming and labor-intensive task. Thus, this paper proposes a new self-training based augmented reality method for seamless real object registration and task assistance without time-consuming pose annotations. As the industrial components or objects have their 3D geometric models, the proposed approach utilizes them as their synthetic data. The automatically generated synthetic dataset is used for training the deep learning model, which outputs a colored coordinate map of objects to be detected. 2D–3D key point matching between the 2D colored coordinate map and 3D colored coordinate model is then performed for 3D object registration in an AR environment. In particular, self-training is performed for fine-tuning the deep learning model by generating pseudo-labels with the highest probability for mask intersection over union (IoU) in the unlabeled real object dataset. We compared the proposed approach with previous studies to confirm its effectiveness and advantages. Also, we performed ablation studies by training a custom dataset based on various industrial 3D virtual models. Finally, we present AR-based task guidance and assistance, such as industrial robot manipulation.

A YOLO-NL object detector for real-time detection

In recent years, YOLO object detection models have undergone significant advancement due to the success of novel deep convolutional networks. The success of these YOLO models is often attributed to their use of guidance techniques, such as expertly tailored deeper backbone and meticulously crafted detector head, which provides effective mechanisms to tradeoff between accuracy and efficiency. However, these sluggish-reasoning models are not capable of handling false detection and negative phenomena, facing challenges include improving the robustness of scaled objects detection against occlude and densely sophisticated scenarios.To address these limitations, we propose a novel object detector, You Only Look Once and None Left (YOLO-NL). Our model includes a novel global dynamic label assignment strategy, which allocates labels for specific anchors to maintain a balance between higher precision detection and finer localization. To enhance the detection capability of multi-scale objects in complex scenes, we separately upgrade CSPNet and PANet using the shortest-longest gradient strategy and self-attention mechanism. To meet the need for fast inference, we propose the Rep-CSPNet network using the reparameterization method to convert residual convolutions to ghost linear operations. Additionally, we accelerate the feature extraction process by deploying the serial SSPP structure. The proposed model is robust to scale objects against negative effectives such as dust, dense, ambiguous, and obstructed scenes. YOLO-NL achieved a mAP of 52.9% on the COCO 2017 test dataset, exhibiting a significant improvement of 2.64% compared to the baseline YOLOX. It is worth noting that YOLO-NL can perform high-accuracy and high-speed face mask detection in real-life scenarios. The YOLO-NL model was employed on self-built FMD and large open-source datasets, and the results show that it outperforms the other state-of-the-art methods, achieving 98.8% accuracy while maintaining 130 FPS.