Scene Model Research Articles

Failure process and three-dimensional motions of mining-induced Jianshanying landslide in China observed by optical, LiDAR and SAR datasets

ABSTRACT The occurrence of collapses and landslides due to underground mining has its unique failure mechanism, especially in the Karst mountainous regions of China. Spaceborne and airborne remote sensing observations provide rapid and effective tools for assessing surface changes and monitoring surface deformation of such landslides. In this study, we take the Jianshanying landslide, a typical mining-induced and fast-deformed landslide, as an example, and reveal the failure mechanism of such landslide by investigating the historical surface displacement. First, the complete evolution of the landslide surface was investigated from its original state to the overall sliding. The data include the satellite and Unmanned Aerial Vehicle (UAV) optical images, UAV three-dimensional (3-D) real scene models, high-resolution Light Detection and Ranging (LiDAR) DEM, and field survey. The results show that the head region entered the high deformation stage after 2019, the maximum deformation rate was 12.3 m/yr. The landslide morphology was formed after the overall slide occurred in September 2020. Then, the pre-event 3-D surface deformation after the landslide entered the high deformation stage was recovered using Interferometric Synthetic Aperture Radar (InSAR), differential DEM, and SAR/optical offset-tracking techniques. The vertical deformation was recovered around −30 m from 2019 to 2020. In particular, we solved the problem of unequal accuracy of SAR and optical offset-tracking observations in 3-D deformation inversion by employing the Helmert variance component estimation method. The maximum deformation was 6 m and 3 m within 4 months in the NS and EW directions, respectively. Finally, we revealed the failure mechanism of the Jianshanying landslide based on the disparity of horizontal and vertical deformation. That is, underground mining causes a significant subsidence of the rear part of the landslide body, resulting in different stress changes in the rear and front parts of the landslide body, which eventually led to sliding of the front part of the slope along the free surface. This work investigates and monitors the typical underground mining-induced Jianshanying landslide by using multi-sensor remote sensing approaches to trace the pre-event surface motions and to reveal its failure mechanism.

Compositional Scene Representation Learning via Reconstruction: A Survey.

Visual scenes are composed of visual concepts and have the property of combinatorial explosion. An important reason for humans to efficiently learn from diverse visual scenes is the ability of compositional perception, and it is desirable for artificial intelligence to have similar abilities. Compositional scene representation learning is a task that enables such abilities. In recent years, various methods have been proposed to apply deep neural networks, which have been proven to be advantageous in representation learning, to learn compositional scene representations via reconstruction, advancing this research direction into the deep learning era. Learning via reconstruction is advantageous because it may utilize massive unlabeled data and avoid costly and laborious data annotation. In this survey, we first outline the current progress on reconstruction-based compositional scene representation learning with deep neural networks, including development history and categorizations of existing methods from the perspectives of the modeling of visual scenes and the inference of scene representations; then provide benchmarks, including an open source toolbox to reproduce the benchmark experiments, of representative methods that consider the most extensively studied problem setting and form the foundation for other methods; and finally discuss the limitations of existing methods and future directions of this research topic.

Naturalistic Scene Modelling: Deep Learning with Insights from Biology

Advances in machine learning coupled with the abundances of training data has facilitated the deep learning era, which has demonstrated its ability and effectiveness in solving complex detection and recognition problems. In general application areas with elements of machine learning have seen exponential growth with promising new and sophisticated solutions to complex learning problems. In computer vision, the challenge related to the detection of known objects in a scene is a thing of the past. With the tremendous increase in detection accuracies, some close to that of human detection, there are several areas still lagging in computer vision and machine learning where improvements may call for more architectural designs. In this paper, we propose a physiologically inspired model for scene understanding that encodes three key components: object location, size and category. Our aim is to develop an energy efficient artificial intelligent model for naturalistic scene understanding capable of deploying on a low power neuromorphic hardware. We have reviewed recent advances in deep learning architecture that have taken inspiration from human or primate learning systems and provided direct to future advancement on deep learning with inspiration from physiological experiments. Upon a review of areas that have benefitted from deep learning, we provide recommendations for enhancing those areas that might have stalled or grinded to a halt with little or no significant improvement.

Technology in the Administration of Justice: Forensic Scene Research in International Approach

Technology begins to play a leading role in the administration of justice, in particular in case of complex proceedings, to which basic forensic methods, techniques and tools have not yet been applied. Humans as an end users of a computer software more and more often use highly specialized software in daily work, thanks to which a number of further analyses can be performed. Such operations, the vast majority of which have been forwarded from competent foreign institutions, are conducted by domestic centres with the application of the latest achievements in forensic science. Recent efforts aimed at introducing commercial services (also made available for use by public institutions) to the market of solutions for the study of sceneprompted the author to scientifically analyse the use of modern technologies in forensic science, with particular emphasis on the crime scene (being the place where traces are discovered and evidence is secured) for the purposes of proceedings conducted by competent law enforcement institutions. Thus, the study can be deemed original and its purpose is to fill the cognitive gap in the indicated scope.Individual sections of the article are devoted to the indication of the role that photography plays in forensic science, issues related to satellite imaging, creation of threedimensional reconstruction models of crime scenes using scanning technology, with an emphasis on the role played by artificial intelligence algorithms in this process as well. As a research thesis, an assumption was made that scene research (its detailed scope, correctness, duration, the possibility of conducting analyses, drawing conclusions), necessary to adequately secure the crime scene, is not independent of the progress of technology. The study includes both theoretical considerations, for which an analysis of Polish and foreign language sources was used, as well as focuses on presenting practical possibilities of applying the described tools. The issue raised remains up-to-date, in particular given the need to seek international opportunities to use the latest, constantly developing technologies for the purpose of conducting criminal proceedings. This focus contributes to the presentation of a comprehensive study, for which a qualitative descriptive method supported by a critical analysis of sources has been used. The article also contains final conclusions and research postulates, including de lege ferendalegislative suggestions.

A Multi-Channel Parallel Keypoint Fusion Framework for Human Pose Estimation

Although modeling self-attention can significantly reduce computational complexity, human pose estimation performance is still affected by occlusion and background noise, and undifferentiated feature fusion leads to significant information loss. To address these issues, we propose a novel human pose estimation framework called DatPose (deformable convolution and attention for human pose estimation), which combines deformable convolution and self-attention to relieve these issues. Considering that the keypoints of the human body are mostly distributed at the edge of the human body, we adopt the deformable convolution strategy to obtain the low-level feature information of the image. Our proposed method leverages visual cues to capture detailed keypoint information, which we embed into the Transformer encoder to learn the keypoint constraints. More importantly, we designed a multi-channel two-way parallel module with self-attention and convolution fusion to enhance the weight of the keypoints in visual cues. In order to strengthen the implicit relationship of fusion, we attempt to generate keypoint tokens to the visual cues of the fusion module and transformers, respectively. Our experimental results on the COCO and MPII datasets show that performing the keypoint fusion module improves keypoint information. Extensive experiments and visual analysis demonstrate the robustness of our model in complex scenes and our framework outperforms popular lightweight networks in human pose estimation.

Performance Evaluation of You Only Look Once v4 in Road Anomaly Detection and Visual Simultaneous Localisation and Mapping for Autonomous Vehicles

The proliferation of autonomous vehicles (AVs) emphasises the pressing need to navigate challenging road networks riddled with anomalies like unapproved speed bumps, potholes, and other hazardous conditions, particularly in low- and middle-income countries. These anomalies not only contribute to driving stress, vehicle damage, and financial implications for users but also elevate the risk of accidents. A significant hurdle for AV deployment is the vehicle’s environmental awareness and the capacity to localise effectively without excessive dependence on pre-defined maps in dynamically evolving contexts. Addressing this overarching challenge, this paper introduces a specialised deep learning model, leveraging YOLO v4, which profiles road surfaces by pinpointing defects, demonstrating a mean average precision (mAP@0.5) of 95.34%. Concurrently, a comprehensive solution—RA-SLAM, which is an enhanced Visual Simultaneous Localisation and Mapping (V-SLAM) mechanism for road scene modeling, integrated with the YOLO v4 algorithm—was developed. This approach precisely detects road anomalies, further refining V-SLAM through a keypoint aggregation algorithm. Collectively, these advancements underscore the potential for a holistic integration into AV’s intelligent navigation systems, ensuring safer and more efficient traversal across intricate road terrains.

DRL-Based Dynamic Destroy Approaches for Agile-Satellite Mission Planning

Agile-satellite mission planning is a crucial issue in the construction of satellite constellations. The large scale of remote sensing missions and the high complexity of constraints in agile-satellite mission planning pose challenges in the search for an optimal solution. To tackle the issue, a dynamic destroy deep-reinforcement learning (D3RL) model is designed to facilitate subsequent optimization operations via adaptive destruction to the existing solutions. Specifically, we first perform a clustering and embedding operation to reconstruct tasks into a clustering graph, thereby improving data utilization. Secondly, the D3RL model is established based on graph attention networks (GATs) to enhance the search efficiency for optimal solutions. Moreover, we present two applications of the D3RL model for intensive scenes: the deep-reinforcement learning (DRL) method and the D3RL-based large-neighborhood search method (DRL-LNS). Experimental simulation results illustrate that the D3RL-based approaches outperform the competition in terms of solutions’ quality and computational efficiency, particularly in more challenging large-scale scenarios. DRL-LNS outperforms ALNS with an average scheduling rate improvement of approximately 11% in Area instances. In contrast, the DRL approach performs better in World scenarios, with an average scheduling rate that is around 8% higher than that of ALNS.

Thermal Imaging Detection System: A Case Study for Indoor Environments.

Currently, there is an increasing need for reliable mechanisms for automatically detecting and localizing people-from performing a people-flow analysis in museums and controlling smart homes to guarding hazardous areas like railway platforms. A method for detecting people using FLIR Lepton 3.5 thermal cameras and Raspberry Pi 3B+ computers was developed. The method creates a control and capture library for the Lepton 3.5 and a new person-detection technique that uses the state-of-the-art YOLO (You Only Look Once) real-time object detector based on deep neural networks. A thermal unit with an automated configuration using Ansible encapsulated in a custom 3D-printed enclosure was used. The unit has applications in simple thermal detection based on the modeling of complex scenes with polygonal boundaries and multiple thermal camera monitoring. An easily deployable person-detection and -localization system based on thermal imaging that supports multiple cameras and can serve as an input for other systems that take actions by knowing the positions of people in monitored environments was created. The thermal detection system was tested on a people-flow analysis performed in the Czech National Museum in Prague. The contribution of the presented method is the development of a small and simple detection system that is easily mountable with wide indoor as well as outdoor applications. The novelty of the system is in the utilization of the YOLO model for thermal data.

Advances in Data‐Driven Analysis and Synthesis of 3D Indoor Scenes

AbstractThis report surveys advances in deep learning‐based modelling techniques that address four different 3D indoor scene analysis tasks, as well as synthesis of 3D indoor scenes. We describe different kinds of representations for indoor scenes, various indoor scene datasets available for research in the aforementioned areas, and discuss notable works employing machine learning models for such scene modelling tasks based on these representations. Specifically, we focus on the analysis and synthesis of 3D indoor scenes. With respect to analysis, we focus on four basic scene understanding tasks – 3D object detection, 3D scene segmentation, 3D scene reconstruction and 3D scene similarity. And for synthesis, we mainly discuss neural scene synthesis works, though also highlighting model‐driven methods that allow for human‐centric, progressive scene synthesis. We identify the challenges involved in modelling scenes for these tasks and the kind of machinery that needs to be developed to adapt to the data representation, and the task setting in general. For each of these tasks, we provide a comprehensive summary of the state‐of‐the‐art works across different axes such as the choice of data representation, backbone, evaluation metric, input, output and so on, providing an organized review of the literature. Towards the end, we discuss some interesting research directions that have the potential to make a direct impact on the way users interact and engage with these virtual scene models, making them an integral part of the metaverse.

Improved Lightweight Multi-Target Recognition Model for Live Streaming Scenes

Nowadays, the commercial potential of live e-commerce is being continuously explored, and machine vision algorithms are gradually attracting the attention of marketers and researchers. During live streaming, the visuals can be effectively captured by algorithms, thereby providing additional data support. This paper aims to consider the diversity of live streaming devices and proposes an extremely lightweight and high-precision model to meet different requirements in live streaming scenarios. Building upon yolov5s, we incorporate the MobileNetV3 module and the CA attention mechanism to optimize the model. Furthermore, we construct a multi-object dataset specific to live streaming scenarios, including anchor facial expressions and commodities. A series of experiments have demonstrated that our model realized a 0.4% improvement in accuracy compared to the original model, while reducing its weight to 10.52%.

Underwater Depth Estimation via Stereo Adaptation Networks

With the fast development and wide application of stereo depth estimation, adequate high-quality stereo training data with groundtruth depth information plays an important role, but is not easily acquired in underwater environments. Therefore, satisfactory performance of depth estimation is difficult to achieve in underwater environments. In addition, the domain gap also leads to the failure of directly applying existing models of terrestrial scene to underwater scene. Therefore, this paper proposes a novel underwater depth estimation network which can infer depth maps from real underwater stereo images in an adaptation manner. The proposed learning pipeline mainly contains three different adaptation modules, i.e., style adaptation, semantic adaptation and disparity range adaptation, to progressively adapt a terrestrial depth estimation model to the underwater domain. Specifically, due to the lack of underwater training data, we first propose a depth-aware stereo image translation network to synthesize stylized underwater stereo images from terrestrial dataset, thus benefiting the effective training of depth estimation network. Then, considering the weak generalization to the real underwater data when only trained on the above synthetic data, we present a self-ensembling semantic adaptation for depth estimation network to minimize the semantic domain discrepancy between synthetic and real underwater data. Meanwhile, we design a disparity range adaptation module to address the problem of disparity range miss-match between both data, thus obtaining more accurate depth predictions for large-disparity-span underwater images. Experimental results show that by integrating the proposed adaptation modules into the off-the-shelf depth estimation backbones, our method successfully achieves superior performance of underwater depth estimation compared to other state-of-the-art methods.

Digitalization of Traffic Scenes in Support of Intelligent Transportation Applications

Digitalization of real-world traffic scenes is a fundamental task in development of digital twins of road transportation. However, the existing digitalization approaches are either expensive in equipment costs or inapplicable to collect granular level data of traffic scenes. This study proposed a vision-based method for real-time digitalization of traffic scenes through modeling and merging the road infrastructure (static components) and road users (dynamic components) progressively. Specifically, the former is reconstructed by leveraging unmanned aerial vehicles (UAVs) and structure from motion; and the latter is digitized via using roadside surveillance videos and a new reconstruction process through applying deep learning and view geometry. Last, the digital model of the traffic scene is built by merging the digital models of static and dynamic components. A field experiment was performed to evaluate the performance of the proposed method. The results showed that the traffic scene can be successfully digitalized by the proposed method with promising accuracy, thus signifying the method’s potential for the development of the digital twins of road transportation in support of intelligent transportation applications.

Assessing Precision and Dependability of Reconstructed Three-Dimensional Modeling for Vehicles at Crash Scenes using Unmanned Aircraft System

This study focuses on the accuracy assessment of 3D reconstructions of crime scenes using Unmanned Aircraft Systems (UAS) and Terrestrial Laser Scanners (TLS) data for forensic crash investigation. Forensic crash investigation involves meticulously analyzing physical evidence, vehicles, and human factors in road collisions to determine the sequence of events. Preserving the original state of the crash scene before cleaning is essential for accurate forensic analysis. However, this preservation process can disrupt normal activities and demand considerable time. Geomatic technology, specifically UAS or drones, offers a potential solution for efficient and precise forensic mapping. The application of UAS technology enables swift data collection, leading to cost savings, enhanced safety, and data utilization. This study aims to assess the suitability of UAS techniques for forensic mapping, encompassing both relative and absolute accuracy. This research uses a UAS to rapidly and comprehensively capture evidence from a simulated crash site using predefined flight paths. The acquired image data is then processed utilizing Agisoft Metashape software, generating a detailed 3D model of the crash scene. This model can be enriched with annotations, measurements, and pertinent information. A comparative analysis is performed by preparing a table that contrasts the absolute and relative accuracy of UAS-collected data with that obtained from TLS, which serves as a benchmark. The results reveal that the UAS demonstrates a relative accuracy Root Mean Square Error (RMSE) of approximately ±4.1 cm compared to TLS. Concerning absolute precision, the UAS-produced RMSE values are determined as ±0.20719 for the X coordinate, ±0.164 for the Y coordinate, and ±0.001584 for the Z coordinate compared to GNSS data, which functions as the benchmark. The utilization of UAS technology offers a non-invasive measurement approach that eliminates direct physical contact between the operator and the documented object. This non-intrusive method ensures the preservation of the original scene characteristics and has shown its superiority over conventional approaches in managing crash scenes. Overall, this study underscores the potential of UAS technology in accurately reconstructing crime scenes for forensic investigation purposes.

Surveillance Video Georeference Method Based on Real Scene Model with Geometry Priors

With the comprehensive promotion of digital construction in China, cameras scattered throughout the country are of great significance in obtaining first-hand data. However, their potential role is limited due to the lack of georeference information on current surveillance cameras. Provided surveillance camera images and real scenes are combined and given georeference information, this problem can be solved, allowing cameras to generate significant social benefits. This article proposed an accurate registration method based on misalignment calibration and least squares matching between real scene and surveillance camera images to address this issue. Firstly, it is necessary to convert the navigation coordinate system from which cameras obtain data to the photogrammetric coordinate system and then solve for the misalignment and internal orientation elements of the camera. Then, accurate registration is achieved using the least squares matching on pyramid images. The experiment obtained surrounding image data of two common scenes with lens pitch angles of 45°, 55°, 65°, 75°, and 85° using the surveillance camera and obtained a 3D real scene model of each scene using a low-altitude aircraft. The experiment results show that the proposed method in this paper can achieve the expected goals of accurately matching real scene and surveillance camera images and assigning georeference information. Through extensive data analysis, the success rate and accuracy rate of registration are 98.1% and 97.06%, respectively.

Разработка системы полуавтоматического моделирования трехмерных изображений

Descriptive geometry is one of the main disciplines of the general engineering cycle. It presents methods of accurate representation of spatial objects on a plane, as well as the identification of geometric shapes of figures from given images. That is, this discipline is designed to give future engineers the knowledge and skills to build and read drawings.
 By causing the intensive work of students' spatial thinking, descriptive geometry contributes to the development of spatial representation. In descriptive geometry, a flat image of a spatial object is called a plot (drawing).
 Students studying descriptive geometry get acquainted with examples of practical use of the theoretical provisions of descriptive geometry in course and diploma design.
 However, it is much easier to perceive drawings in volume if they can be rotated and viewed from different sides. For these purposes, you can not do without third-party software. Visualization of objects in the presence of a three-dimensional model is provided by Unity. But to create a three–dimensional model according to the drawing, other software is required - for example, Blender. Therefore, the authors set a goal: to create a semi-automatic visualization system for two-dimensional drawings in space based on Blender software and to develop comprehensive and simple documentation that will allow even people far from programming to use the program.
 It is known that Blender is a powerful program for creating three-dimensional graphics that allows you to create various objects, scenes, animations, as well as edit them. Blender is a free and open source program available to anyone interested in three-dimensional modeling and computer graphics creation.
 Among the Blender tools for creating three-dimensional models are powerful tools for modeling, sculpting, animation and visualization of objects and scenes. In Blender, you can also draw textures and create 3D animations using animation tools such as key frames.

Towards real-world haze removal with uncorrelated graph model

The scene restoration effect of the dehazing algorithms usually suffers from double quality interference. First, the dehazing methods based on the atmospheric scattering model (ASM) lead to light loss. Second, the synthetic data used for supervised learning may produce apparent data deviation from the real hazy scene, which seriously weakens the generalization ability of the model. To address the above problems, we propose an uncorrelated graph dehazing model for real-world scenes. The process firstly eliminates the correlation in the representation space by establishing a directed acyclic graph with isolated points, thus enhancing the generalization performance of the model. Secondly, to suppress over-exposure that may occur on real-world data, the proposed model training is performed with a combination of light control. This work improves the ASM and constructs a new dataset applicable to natural haze scenes. Finally, the effectiveness of the proposed method can be verified through multiple experimental comparisons.

Automatic extraction and reconstruction of a 3D wireframe of an indoor scene from semantic point clouds

ABSTRACT Accurate indoor 3D models are essential for building administration and applications in digital city construction and operation. Developing an automatic and accurate method to reconstruct an indoor model with semantics is a challenge in complex indoor environments. Our method focuses on the permanent structure based on a weak Manhattan world assumption, and we propose a pipeline to reconstruct indoor models. First, the proposed method extracts boundary primitives from semantic point clouds, such as floors, walls, ceilings, windows, and doors. The primitives of the building boundary are aligned to generate the boundaries of the indoor scene, which contains the structure of the horizontal plane and height change in the vertical direction. Then, an optimization algorithm is applied to optimize the geometric relationships among all features based on their categories after the classification process. The heights of feature points are captured and optimized according to their neighborhoods. Finally, a 3D wireframe model of the indoor scene is reconstructed based on the 3D feature information. Experiments on three different datasets demonstrate that the proposed method can be used to effectively reconstruct 3D wireframe models of indoor scenes with high accuracy.

Vehicle Localization in a Completed City-Scale 3D Scene Using Aerial Images and an On-Board Stereo Camera

Simultaneous Localization and Mapping (SLAM) forms the foundation of vehicle localization in autonomous driving. Utilizing high-precision 3D scene maps as prior information in vehicle localization greatly assists in the navigation of autonomous vehicles within large-scale 3D scene models. However, generating high-precision maps is complex and costly, posing challenges to commercialization. As a result, a global localization system that employs low-precision, city-scale 3D scene maps reconstructed by unmanned aerial vehicles (UAVs) is proposed to optimize visual positioning for vehicles. To address the discrepancies in image information caused by differing aerial and ground perspectives, this paper introduces a wall complementarity algorithm based on the geometric structure of buildings to refine the city-scale 3D scene. A 3D-to-3D feature registration algorithm is developed to determine vehicle location by integrating the optimized city-scale 3D scene with the local scene generated by an onboard stereo camera. Through simulation experiments conducted in a computer graphics (CG) simulator, the results indicate that utilizing a completed low-precision scene model enables achieving a vehicle localization accuracy with an average error of 3.91 m, which is close to the 3.27 m error obtained using the high-precision map. This validates the effectiveness of the proposed algorithm. The system demonstrates the feasibility of utilizing low-precision city-scale 3D scene maps generated by unmanned aerial vehicles (UAVs) for vehicle localization in large-scale scenes.

Reconstructing 3D model of accident scene using drone image processing

At the current stage, an investigation technique on the accident takes a longer time and this causes longer traffic congestion. The aim of this study is to reconstruct a 3D model of an accident scene using an unmanned aerial vehicle (UAV). The flight parameters that have been chosen are the circular method, the double grid method, and the single grid method. All these designs can produce a good 3D model to achieve the study’s objective. The methodology in this study is divided into 4 phases which are preliminary work, data acquisition, data processing, and data analysis. The main results of this study are the 3D model of the accident scene, an orthophoto map layout, and an accuracy assessment of a 3D model of reconstructed accident scene. All these parameters will be tested on accuracy based on the root mean square error (RMSE) value, comparing the UAV data and site measurement data. This objective has been tested for 10 different types of processing and different types of flight parameters. The best result among all the methods is the circular method 5 meters with ground control point (GCP) since this method has the least RMSE value which is 0.047 m. UAVs can replace the site measurement to reconstruct the accident scene.

Time-optimal trajectory planning for autonomous parking of tractor-semitrailer vehicle based on piecewise Gauss pseudospectral method

For the autonomous parking problem of the tractor-semitrailer vehicle, a piecewise Gauss pseudospectral method is proposed as a trajectory planning scheme in this paper. Firstly, the trajectory planning problem is transformed into an optimal control problem by establishing a vehicle kinematics model for the low-velocity scene, the dynamic constraints including the physical constraints, the collision avoidance constraints, and the endpoint constraints. The optimal control problem are discretized into a nonlinear programming problem by piecewise Gauss pseudospectral method according to the movement area during parking process. The shortest time to complete parking is taken as the objective function. The Gauss pseudospectral method is further adopted to solve the nonlinear programming problem in the two parking stages. Three parallel parking conditions with different lengths of parking spaces and three vertical parking conditions with different widths of parking spaces are set for simulation verification based on Matlab/Simulink. The simulation results show that the proposed method can uniformly and effectively solve the trajectory planning problem of the tractor-semitrailer vehicle. Compared with the traditional pseudospectral method, a shorter parking time and a significantly higher convergence rate can be got with the method, which further indicates that the proposed method has good effectiveness and robustness.