Dynamic Scene Analysis Research Articles

CROWD VIDEO SEQUENCES PROCESSING METHODS FOR DETERMINING THE CROWD MOTION PATTERNS

Nowadays, homogeneous objects clusters motion is one of the most important and rapidly developing computer vision and machine learning application. In this paper, we consider the crowd motion patterns determination by using motion maps that we calculate with FlowNet, a neural network examining motion of objects in a video sequence. This approach allows us to get information on the crowd direction and velocity with relation to other objects of scene, which plays the key role in behavior analysis and security establishment. Besides, we consider methods for preliminary video sequence processing, including frame combination, to estimate motion maps more precisely and improve the effectiveness of the dynamic scenes analysis.

Optical Flow for Spike Camera with Hierarchical Spatial-Temporal Spike Fusion

As an emerging neuromorphic camera with an asynchronous working mechanism, spike camera shows good potential for high-speed vision tasks. Each pixel in spike camera accumulates photons persistently and fires a spike whenever the accumulation exceeds a threshold. Such high-frequency fine-granularity photon recording facilitates the analysis and recovery of dynamic scenes with high-speed motion. This paper considers the optical flow estimation problem for spike cameras. Due to the Poisson nature of incoming photons, the occurrence of spikes is random and fluctuating, making conventional image matching inefficient. We propose a Hierarchical Spatial-Temporal (HiST) fusion module for spike representation to pursue reliable feature matching and develop a robust optical flow network, dubbed as HiST-SFlow. The HiST extracts features at multiple moments and hierarchically fuses the spatial-temporal information. We also propose an intra-moment filtering module to further extract the feature and suppress the influence of randomness in spikes. A scene loss is proposed to ensure that this hierarchical representation recovers the essential visual information in the scene. Experimental results demonstrate that the proposed method achieves state-of-the-art performance compared with the existing methods. The source codes are available at https://github.com/ruizhao26/HiST-SFlow.

A real-time wearable AR system for egocentric vision on the edge

Real-time performance is critical for Augmented Reality (AR) systems as it directly affects responsiveness and enables the timely rendering of virtual content superimposed on real scenes. In this context, we present the DARLENE wearable AR system, analysing its specifications, overall architecture and core algorithmic components. DARLENE comprises AR glasses and a wearable computing node responsible for several time-critical computation tasks. These include computer vision modules developed for the real-time analysis of dynamic scenes supporting functionalities for instance segmentation, tracking and pose estimation. To meet real-time requirements in limited resources, concrete algorithmic adaptations and design choices are introduced. The proposed system further supports real-time video streaming and interconnection with external IoT nodes. To improve user experience, a novel approach is proposed for the adaptive rendering of AR content by considering the user’s stress level, the context of use and the environmental conditions for adjusting the level of presented information towards enhancing their situational awareness. Through extensive experiments, we evaluate the performance of individual components and end-to-end pipelines. As the proposed system targets time-critical security applications where it can be used to enhance police officers’ situational awareness, further experimental results involving end users are reported with respect to overall user experience, workload and evaluation of situational awareness.

Long-term Human Participation Assessment In Collaborative Learning Environments Using Dynamic Scene Analysis

Long-term Human Participation Assessment In Collaborative Learning Environments Using Dynamic Scene Analysis

PREFACE: TECHNICAL COMMISSION II

Abstract. ISPRS Technical Commission II focuses, at various scales, on geometric, radiometric and multi-temporal aspects of image- and range-based 3D surveying and modeling. Specifically, Commission II deals with image orientation, point cloud generation and processing, 3D feature extraction, dynamic and static scene analysis, sensor and data fusion, and machine learning for geospatial data analysis and big data techniques for massive data processing. Applications in the fields of mapping, infrastructure monitoring, heritage studies, space exploration, underwater photogrammetry and environmental engineering are also considered.The Volume related to Commission II contains 108 papers published in the ISPRS Archives and 20 published in the ISPRS Annals. The papers in Archives were accepted from among 111 abstract submissions and 59 full papers. Of the 108 Archives papers, 32 were full paper submissions and 76 were abstract submissions. The 20 papers in the Annals were selected from among 59 full paper submissions after going through a peer review process.In the aforementioned areas of research, the papers in this volume discuss the challenges and needs, and introduce novel photogrammetric solutions that depict the latest developments in the field.There has been a wide range of coverage of these topics and point cloud generation and processing has been the most active coverage with close to 38% of the accepted papers. This is followed by machine/deep learning methods with 31.8% that provide solutions to the semantic enrichment of images and 3D data. Research in heritage and underwater studies is represented with 9% and 6%, respectively, of the accepted papers.We believe these volumes nicely recap the state-of-the-art, current trends and possible applications of photogrammetry on a wide range of topics with a nice overview of the future research directions.On behalf of Technical Commission II, we would like to thank the local organizers of the 2021 ISPRS Congress, the members of the international program committee, all Working Group officers and all reviewers for their hard organizational work and efforts in the paper reviewing process.

PREFACE: TECHNICAL COMMISSION II

Abstract. ISPRS Technical Commission II focuses, at various scales, on geometric, radiometric and multi-temporal aspects of image- and range-based 3D surveying and modeling. Specifically, Commission II deals with image orientation, point cloud generation and processing, 3D feature extraction, dynamic and static scene analysis, sensor and data fusion, and machine learning for geospatial data analysis and big data techniques for massive data processing. Applications in the fields of mapping, infrastructure monitoring, heritage studies, space exploration, underwater photogrammetry and environmental engineering are also considered.The Volume related to Commission II contains 108 papers published in the ISPRS Archives and 20 published in the ISPRS Annals. The papers in Archives were accepted from among 111 abstract submissions and 59 full papers. Of the 108 Archives papers, 32 were full paper submissions and 76 were abstract submissions. The 20 papers in the Annals were selected from among 59 full paper submissions after going through a peer review process.In the aforementioned areas of research, the papers in this volume discuss the challenges and needs, and introduce novel photogrammetric solutions that depict the latest developments in the field.There has been a wide range of coverage of these topics and point cloud generation and processing has been the most active coverage with close to 38% of the accepted papers. This is followed by machine/deep learning methods with 31.8% that provide solutions to the semantic enrichment of images and 3D data. Research in heritage and underwater studies is represented with 9% and 6%, respectively, of the accepted papers.We believe these volumes nicely recap the state-of-the-art, current trends and possible applications of photogrammetry on a wide range of topics with a nice overview of the future research directions.On behalf of Technical Commission II, we would like to thank the local organizers of the 2021 ISPRS Congress, the members of the international program committee, all Working Group officers and all reviewers for their hard organizational work and efforts in the paper reviewing process.

A DIC-assisted fringe projection profilometry for high-speed 3D shape, displacement and deformation measurement of textured surfaces

High-speed three-dimensional (3D) shape measurement techniques based on fringe projection profilometry (FPP) have undergone huge advances over the past two decades. However, accurate 3D displacement mapping and deformation analysis of dynamic scenes using FPP remains an unsolved problem. Because fringe patterns are projected rather than attached on the tested surfaces, the full-field point-to-point correspondence cannot be accurately established between any two 3D shape results. To deal with this challenge, a DIC-assisted FPP for high-speed 3D shape, displacement and deformation measurement of textured surfaces is proposed. Firstly, a high-speed 3D shape measurement system is adopted using our recently proposed robust and efficient Gray-coded coding strategy, which can accurately reconstruct full-field shape of discontinuous surfaces with rich texture information. Then, the modulation-based method is proposed to retrieve high-quality texture map from three phase-shifting fringe patterns, which can eliminate the adverse influence of the nonuniform and time-varying ambient light. By matching the retrieved surface texture images at difference states using DIC, accurate point tracking between the measured 3D shape data can be fulfilled, leading to precise 3D displacement and deformation measurement. Experiments have verified that the proposed method can achieve 3D shape, displacement and deformation measurement of dynamic scenes at a frame rate of 542 fps without any extra expense of hardware or calibration for the FPP system. The presented method is reliable and promising for further 3D displacement mapping and deformation analysis of dynamic scenes using FPP.

Robust 3D Reconstruction of Dynamic Scenes From Single-Photon Lidar Using Beta-Divergences.

In this article, we present a new algorithm for fast, online 3D reconstruction of dynamic scenes using times of arrival of photons recorded by single-photon detector arrays. One of the main challenges in 3D imaging using single-photon lidar in practical applications is the presence of strong ambient illumination which corrupts the data and can jeopardize the detection of peaks/surface in the signals. This background noise not only complicates the observation model classically used for 3D reconstruction but also the estimation procedure which requires iterative methods. In this work, we consider a new similarity measure for robust depth estimation, which allows us to use a simple observation model and a non-iterative estimation procedure while being robust to mis-specification of the background illumination model. This choice leads to a computationally attractive depth estimation procedure without significant degradation of the reconstruction performance. This new depth estimation procedure is coupled with a spatio-temporal model to capture the natural correlation between neighboring pixels and successive frames for dynamic scene analysis. The resulting online inference process is scalable and well suited for parallel implementation. The benefits of the proposed method are demonstrated through a series of experiments conducted with simulated and real single-photon lidar videos, allowing the analysis of dynamic scenes at 325 m observed under extreme ambient illumination conditions.

A Biologically Motivated, Proto-Object-Based Audiovisual Saliency Model

The natural environment and our interaction with it are essentially multisensory, where we may deploy visual, tactile and/or auditory senses to perceive, learn and interact with our environment. Our objective in this study is to develop a scene analysis algorithm using multisensory information, specifically vision and audio. We develop a proto-object-based audiovisual saliency map (AVSM) for the analysis of dynamic natural scenes. A specialized audiovisual camera with 360∘ field of view, capable of locating sound direction, is used to collect spatiotemporally aligned audiovisual data. We demonstrate that the performance of a proto-object-based audiovisual saliency map in detecting and localizing salient objects/events is in agreement with human judgment. In addition, the proto-object-based AVSM that we compute as a linear combination of visual and auditory feature conspicuity maps captures a higher number of valid salient events compared to unisensory saliency maps. Such an algorithm can be useful in surveillance, robotic navigation, video compression and related applications.

Drosophila-inspired 3D moving object detection based on point clouds

3D moving object detection is one of the most critical tasks in dynamic scene analysis. In this paper, we propose a novel Drosophila-inspired 3D moving object detection method using Lidar sensors. According to the theory of elementary motion detector, we have developed a motion detector based on the shallow visual neural pathway of Drosophila. This detector is sensitive to the movement of objects and can well suppress background noise. Designing neural circuits with different connection modes, the approach searches for motion areas in a coarse-to-fine fashion and extracts point clouds of each motion area to form moving object proposals. An improved 3D object detection network is then used to estimate the point clouds of each proposal and efficiently generates the 3D bounding boxes and the object categories. We evaluate the proposed approach on the widely-used KITTI benchmark, and state-of-the-art performance was obtained by using the proposed approach on the task of motion detection.

Fast online 3D reconstruction of dynamic scenes from individual single-photon detection events.

In this paper, we present an algorithm for online 3D reconstruction of dynamic scenes using individual times of arrival (ToA) of photons recorded by single-photon detector arrays. One of the main challenges in 3D imaging using single-photon Lidar is the integration time required to build ToA histograms and reconstruct reliably 3D profiles in the presence of non-negligible ambient illumination. This long integration time also prevents the analysis of rapid dynamic scenes using existing techniques. We propose a new method which does not rely on the construction of ToA histograms but allows, for the first time, individual detection events to be processed online, in a parallel manner in different pixels, while accounting for the intrinsic spatiotemporal structure of dynamic scenes. Adopting a Bayesian approach, a Bayesian model is constructed to capture the dynamics of the 3D profile and an approximate inference scheme based on assumed density filtering is proposed, yielding a fast and robust reconstruction algorithm able to process efficiently thousands to millions of frames, as usually recorded using single-photon detectors. The performance of the proposed method, able to process hundreds of frames per second, is assessed using a series of experiments conducted with static and dynamic 3D scenes and the results obtained pave the way to a new family of real-time 3D reconstruction solutions.

Track-Before-Detect Framework-Based Vehicle Monocular Vision Sensors

This paper proposes a Track-before-Detect framework for a multibody motion segmentation (named TbD-SfM). Our contribution relies on a tightly coupled tracking before detection strategy intended to reduce the complexity of existing Multibody Structure from Motion approaches. Efforts were done towards an algorithm variant closer and aimed to a further embedded implementation for dynamic scene analysis while enhancing processing time performances. This generic motion segmentation approach can be transposed to several transportation sensor systems since no constraints are considered on segmented motions (6-DOF model). The tracking scheme is analyzed and its performance is evaluated under thorough experimental conditions including full-scale driving scenarios from known and available datasets. Results on challenging scenarios including the presence of multiple and simultaneous moving objects observed from a moving camera are reported and discussed.

MAGiC: A Multimodal Framework for Analysing Gaze in Dyadic Communication.

The analysis of dynamic scenes has been a challenging domain in eye tracking research. This study presents a framework, named MAGiC, for analyzing gaze contact and gaze aversion in face-to-face communication. MAGiC provides an environment that is able to detect and track the conversation partner’s face automatically, overlay gaze data on top of the face video, and incorporate speech by means of speech-act annotation. Specifically, MAGiC integrates eye tracking data for gaze, audio data for speech segmentation, and video data for face tracking. MAGiC is an open source framework and its usage is demonstrated via publicly available video content and wiki pages. We explored the capabilities of MAGiC through a pilot study and showed that it facilitates the analysis of dynamic gaze data by reducing the annotation effort and the time spent for manual analysis of video data.

THERMAL TEXTURE GENERATION AND 3D MODEL RECONSTRUCTION USING SFM AND GAN

Abstract. Realistic 3D models with textures representing thermal emission of the object are widely used in such fields as dynamic scene analysis, autonomous driving, and video surveillance. Structure from Motion (SfM) methods provide a robust approach for the generation of textured 3D models in the visible range. Still, automatic generation of 3D models from the infrared imagery is challenging due to an absence of the feature points and low sensor resolution. Recent advances in Generative Adversarial Networks (GAN) have proved that they can perform complex image-to-image transformations such as a transformation of day to night and generation of imagery in a different spectral range. In this paper, we propose a novel method for generation of realistic 3D models with thermal textures using the SfM pipeline and GAN. The proposed method uses visible range images as an input. The images are processed in two ways. Firstly, they are used for point matching and dense point cloud generation. Secondly, the images are fed into a GAN that performs the transformation from the visible range to the thermal range. We evaluate the proposed method using real infrared imagery captured with a FLIR ONE PRO camera. We generated a dataset with 2000 pairs of real images captured in thermal and visible range. The dataset is used to train the GAN network and to generate 3D models using SfM. The evaluation of the generated 3D models and infrared textures proved that they are similar to the ground truth model in both thermal emissivity and geometrical shape.

Система операторов для пространственно-временного анализа динамических сцен

The rapid growth in the volume of information and the need for its comprehensive analysis lead to the development of new methods of working with multidimensional and space-time data. To work with such data, Qualitative Spatial Reasoning is often used to extract (produce) new knowledge based on facts established in one way or another. Despite the variety of formal analysis systems available, sets of operators do not allow for the identification of complex space-time relationships between objects. Existing software tools, such as SparQ, GQR, QAT, CLP (QS), are focused on analyzing one aspect and are applicable mainly for interval analysis of time series of events and production of conclusions about some spatial relationships. In practice, tools are usually used to analyze simple relationships in simple scenes. The lack of interfaces limits their combined use with quantitative analysis tools, necessary, for example, to establish primary facts, and use within the framework of the concepts of 4D (space-time), 5D (space-time and cost) and multi-D (multidimensional) modeling. This paper proposes a system of topological, metric, directional and temporal operators for complex spatial-temporal analysis of dynamic scenes. This system allows combined usage of qualitative and quantitative analysis methods, which is essential not only for determining initial facts, but also for producing new knowledge based on these facts. The system of operators proposed is deemed prospective for problems of spatial-temporal (4D) modeling and planning of industrial projects, and particularly for specifying and detecting of non-trivial conflicts in project schedules.

Situational awareness assessment of hearing protection

Uncompromised situational awareness has become a critical component of hearing protection devices (HPDs). Situational awareness is a complex psychological phenomenon, which consists of several perceptual and cognitive factors. This study presents data from three experiments designed to analyze the performance of HPDs on measures of situational awareness and determine how multiple factors may impact in-the-field performance. Data are presented for four hearing protection devices on measures of sound localization, distance perception, spatial segregation, speech intelligibility, and dynamic scene analysis. Baseline (open-ear) data is also presented to contrast with the HPDs. Small differences in fine localization-discrimination and the number of large quadrant errors were observed, but these differences between devices were not observed in the more complex scene analysis task. A performance index was developed to determine overall performance of each HPD compared to open-ear performance. This index combined...

Reconstructing the 3D shapes of droplets in glass microchannels with application to Bretherton’s problem

The outstanding potential of absorbance imaging for the 3D shape analysis of dynamic scenes in multiphase flows is demonstrated and experimentally confirmed. The experimental procedures and workflows have been extended and applied to the 3D shape analysis of droplets moving in microchannels. Using this technique, the problem of measuring the thickness of the lubrication layer and the droplet geometry with sub-\(\upmu\)m resolution could be solved. The obtained data generally conform to models based on Bretherton’s theory and allow its extension even for non-circular microchannel geometries.

Bilayer sparse topic model for scene analysis in imbalanced surveillance videos.

Dynamic scene analysis has become a popular research area especially in video surveillance. The goal of this paper is to mine semantic motion patterns and detect abnormalities deviating from normal ones occurring in complex dynamic scenarios. To address this problem, we propose a data-driven and scene-independent approach, namely, Bilayer sparse topic model (BiSTM), where a given surveillance video is represented by a word-document hierarchical generative process. In this BiSTM, motion patterns are treated as latent topics sparsely distributed over low-level motion vectors, whereas a video clip can be sparsely reconstructed by a mixture of topics (motion pattern). In addition to capture the characteristic of extreme imbalance between numerous typical normal activities and few rare abnormalities in surveillance video data, a one-class constraint is directly imposed on the distribution of documents as a discriminant priori. By jointly learning topics and one-class document representation within a discriminative framework, the topic (pattern) space is more specific and explicit. An effective alternative iteration algorithm is presented for the model learning. Experimental results and comparisons on various public data sets demonstrate the promise of the proposed approach.

Motion-based segmentation of objects using overlapping temporal windows

Motion segmentation refers to the problem of separating the objects in a video sequence according to their motion. It is a fundamental problem of computer vision, since various systems focusing on the analysis of dynamic scenes include motion segmentation algorithms. In this paper we present a novel approach, where a video shot is temporally divided in successive and overlapping windows and motion segmentation is performed on each window respectively. This attribute renders the algorithm suitable even for long video sequences. In the last stage of the algorithm the segmentation results for every window are aggregated into a final segmentation. The presented algorithm can handle effectively asynchronous trajectories on each window even when they have no temporal intersection. The evaluation of the proposed algorithm on the Berkeley motion segmentation benchmark demonstrates its scalability and accuracy compared to the state of the art.

TOWARDS 4D VIRTUAL CITY RECONSTRUCTION FROM LIDAR POINT CLOUD SEQUENCES

Abstract. In this paper we propose a joint approach on virtual city reconstruction and dynamic scene analysis based on point cloud sequences of a single car-mounted Rotating Multi-Beam (RMB) Lidar sensor. The aim of the addressed work is to create 4D spatio-temporal models of large dynamic urban scenes containing various moving and static objects. Standalone RMB Lidar devices have been frequently applied in robot navigation tasks and proved to be efficient in moving object detection and recognition. However, they have not been widely exploited yet for geometric approximation of ground surfaces and building facades due to the sparseness and inhomogeneous density of the individual point cloud scans. In our approach we propose an automatic registration method of the consecutive scans without any additional sensor information such as IMU, and introduce a process for simultaneously extracting reconstructed surfaces, motion information and objects from the registered dense point cloud completed with point time stamp information.