Sequence Of Range Images Research Articles

Multi exposure fusion for high dynamic range imaging via multi-channel gradient tensor

Multi-exposure fusion (MEF) is an effective technique for directly fusing a sequence of low dynamic range (LDR) images from a high dynamic range (HDR) natural scene. The goal is to generate an information enriched LDR image. Despite its effectiveness, current MEF methods often encounter issues such as detail loss and color degradation. Additionally, existing algorithms often struggle to balance image quality and computation time, particularly for large-sized images. This paper introduces an innovative MEF algorithm that address these challenges, offering improved performance and computational time across all image sizes. The algorithm employs a multi-channel gradient tensor on RGB images to effectively capture the contrast information among the three channels. This mechanism allows an edge-preserving image filter to maintain edges while smoothing weight maps. To enhance computational efficiency, the algorithm uses a fast approximation method suitable for large sized images. Our comprehensive experimental results demonstrate that the proposed method outperforms existing MEF techniques both quantitatively and qualitatively. Furthermore, our method reduces computational time by approximately 30% compared to the most recent state-of-the-art techniques.

SeqOT: A Spatial–Temporal Transformer Network for Place Recognition Using Sequential LiDAR Data

Place recognition is an important component for autonomous vehicles to achieve loop closing or global localization. In this paper, we tackle the problem of place recognition based on sequential 3D LiDAR scans obtained by an onboard LiDAR sensor. We propose a transformer-based network named SeqOT to exploit the temporal and spatial information provided by sequential range images generated from the LiDAR data. It uses multi-scale transformers to generate a global descriptor for each sequence of LiDAR range images in an end-to-end fashion. During online operation, our SeqOT finds similar places by matching such descriptors between the current query sequence and those stored in the map. We evaluate our approach on four datasets collected with different types of LiDAR sensors in different environments. The experimental results show that our method outperforms the state-of-the-art LiDAR-based place recognition methods and generalizes well across different environments. Furthermore, our method operates online faster than the frame rate of the sensor. The implementation of our method is released as open source at: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/BIT-MJY/SeqOT</uri> .

HDR image-based deep learning approach for automatic detection of split defects on sheet metal stamping parts

Sheet metal stamping is widely used for high-volume production. Despite the wide adoption, it can lead to defects in the manufactured components, making their quality unacceptable. Because of the variety of defects that can occur on the final product, human inspectors are frequently employed to detect them. However, they can be unreliable and costly, particularly at speeds that match the stamping rate. In this paper, we propose an automatic inspection framework for the stamping process that is based on computer vision and deep learning techniques. The low cost, remote sensing capability and simple implementation mean that it can be easily deployed in an industrial setting. A particular focus of this research is to account for the harsh lighting conditions and the highly reflective nature of products found in manufacturing environments that affect optical sensing techniques by making it difficult to capture the details of a scene. High dynamic range images can capture details of an environment in harsh lighting conditions, and in the context of this work, can capture highly reflective metals found in sheet metal stamping manufacturing. Building on this imaging technique, we propose a framework including a deep learning model to detect defects in sheet metal stamping parts. To test the framework, sheet metal ‘Nakajima’ samples were pressed with an industrial stamping press. Then optimally exposed, sequence of exposures, tone-mapped and high dynamic range images of the samples were used to train convolutional neural network-based detectors. Analysis of the resulting models showed that high dynamic range image-based models achieved substantially higher accuracy and minimal false-positive predictions.

A Novel Coding Architecture for Multi-Line LiDAR Point Clouds Based on Clustering and Convolutional LSTM Network

Light detection and ranging (LiDAR) plays an indispensable role in autonomous driving technologies, such as localization, map building, navigation and object avoidance. However, due to the vast amount of data, transmission and storage could become an important bottleneck. In this article, we propose a novel compression architecture for multi-line LiDAR point cloud sequences based on clustering and convolutional long short-term memory (LSTM) networks. LiDAR point clouds are structured, which provides an opportunity to convert the 3D data to 2D array, represented as range images. Thus, we cast the 3D point clouds compression as a range image sequence compression problem. Inspired by the high efficiency video coding (HEVC) algorithm, we design a novel compression framework for LiDAR data that includes two main techniques: intra-prediction and inter-prediction. For intra-frames, inspired by the depth modeling modes (DMM) adopted in 3D-HEVC, we develop a clustering-based intra-prediction technique, which can utilize the spatial structure characteristics of point clouds to remove the spatial redundancy. For inter-frames, we design a prediction network model using convolutional LSTM cells. The network model is capable of predicting future inter-frames using the encoded intra-frames. As a result, temporal redundancy can be removed. Experiments on the KITTI dataset demonstrate that the proposed method achieves an impressive compression ratio (CR), with 4.10% at millimeter precision, which means the point clouds can compress to nearly 1/25 of their original size. Additionally, compared with the well-known octree, Google Draco, and MPEG TMC13 methods, our algorithm yields better performance in compression ratio.

De-ghosting in High Dynamic Range Imaging Based on Intensity Scaling Cue

A High Dynamic Range (HDR) image produced from a sequence of low dynamic range (LDR) images can contain motion artefacts (ghosting) if the scene contains moving objects. Conventional d ...

An Unsupervised Real-Time Framework of Human Pose Tracking From Range Image Sequences

Pose tracking from range image sequences remains a difficult task due to strong noise and serious self-occlusion of human body. Existing work either rely on extremely large and precisely annotated datasets, or rely on accurate human mesh model and GPU acceleration. In this paper, we propose an unsupervised real-time framework of pose tracking from range image sequences. Our framework consists of a visible hybrid model (VHM), a componentwise correspondence optimization (CCO) and a dynamic database lookup (DDL). VHM consists of component sphere sets and component visible spherical point sets which exhibits both simplicity and high accuracy. CCO converts the matching between VHM and input point cloud into several subproblems regarding local rotations of components and a global translation of body abdominal joint, each of which has an efficient closed form solution. DDL is designed to recover correct pose when tracking fails, which effectively mitigates accumulative error during tracking. Experiments on $\mathsf{SMMC}$ , $\mathsf{PDT}$ , $\mathsf{EVAL}$ datasets indicate that our framework not only achieves better or competitive precision compared with state-of-the-art methods, but also produces real-time efficiency in personal computers without GPU acceleration.

Snapshot High Dynamic Range Imaging via Sparse Representations and Feature Learning

Bracketed High Dynamic Range (HDR) imaging architectures acquire a sequence of Low Dynamic Range (LDR) images in order to either produce a HDR image or an “optimally” exposed LDR image, achieving impressive results under static camera and scene conditions. However, in real world conditions, ghost-like artifacts and noise effects limit the quality of HDR reconstruction. We address these limitations by introducing a post-acquisition snapshot HDR enhancement scheme that generates a bracketed sequence from a small set of LDR images, and in the extreme case, directly from a single exposure. We achieve this goal via a sparse-based approach where transformations between differently exposed images are encoded through a dictionary learning process, while we learn appropriate features by employing a stacked sparse autoencoder (SSAE) based framework. Via experiments with real images, we demonstrate the improved performance of our method over the state-of-the-art, while our single-shot based HDR formulation provides a novel paradigm for the enhancement of LDR imaging and video sequences.

BUILDING FACADE MODELING UNDER LINE FEATURE CONSTRAINT BASED ON CLOSE-RANGE IMAGES

Abstract. To solve existing problems in modeling facade of building merely with point feature based on close-range images , a new method for modeling building facade under line feature constraint is proposed in this paper. Firstly, Camera parameters and sparse spatial point clouds data were restored using the SFM , and 3D dense point clouds were generated with MVS; Secondly, the line features were detected based on the gradient direction , those detected line features were fit considering directions and lengths , then line features were matched under multiple types of constraints and extracted from multi-image sequence. At last, final facade mesh of a building was triangulated with point cloud and line features. The experiment shows that this method can effectively reconstruct the geometric facade of buildings using the advantages of combining point and line features of the close - range image sequence，especially in restoring the contour information of the facade of buildings.

Generating high dynamic range images using a sky imager

One way to optimize their operation of Concentrated Solar Power (CSP) plants is to obtain accurate short-term forecasts of Direct Normal Irradiance (DNI). To do so, the use of sky imagers is developing. They should provide high-quality images, especially in the circumsolar area, where pixels are usually saturated. To reduce the saturated area around the Sun, High Dynamic Range (HDR) imaging is needed. This paper describes the steps required to obtain HDR images from a sequence of low dynamic range images acquired using a sky imager. Since the HDR image will be used to study the sky radiance and the circumsolar area, a particular care is taken to maintain a linear sensor’s response throughout the process.

Robust surface tracking in range image sequences

A novel robust method for surface tracking in range-image sequences is presented which combines a clustering method based on surface models with a particle-filter-based 2-D affine-motion estimator. Segmented regions obtained at previous time steps are used to create seed areas by comparing measured depth values with those obtained from surface-model fitting. The seed areas are further refined using a motion-probability region estimated by the particle-filter-based tracker through prediction of future states. This helps resolving ambiguities that arise when surfaces belonging to different objects are in physical contact with each other, for example during hand-object manipulations. Region growing allows recovering the complete segment area. The obtained segmented regions are then used to improve the predictions of the tracker for the next frame. The algorithm runs in quasi real-time and uses on-line learning, eliminating the need to have a priori knowledge about the surface being tracked. We apply the method to in-house depth videos acquired with both time-of-flight and structured-light sensors, demonstrating object tracking in real-world scenarios, and we compare the results with those of an ICP-based tracker.

Direct three‐dimensional head pose estimation from Kinect‐type sensors

A direct method for recovering three-dimensional (3D) head motion parameters from a sequence of range images acquired by Kinect sensors is presented. Based on the range images, a new version of the ...

Recent Progress in Human Face Detection, Tracking and Recognition

Facial recognition is an important research topic in biometrics and has wide applications in pattern recognition and computer vision. This paper aims at providing interested readers with recent progresses in human face detection, tracking, video-based face recognition, and 3D+2D hybrid face recognition. For this purpose, it mainly focuses on those state-of-the-arts methods and technologies that emerged in recent previous few years. Most existing methods in this area are still-image based which do not utilize motion information; whereas most video-based methods work only in 2D video sequences, which are subject to pose and illumination variations. The recent emergence of 3D video cameras capable of producing range image sequences and corresponding texture image sequences simultaneously allows for the possibility of facial recognition in a 3D+2D video-based scenario. In view of this fact, a scheme of face detection, tracking and recognition process in 3D video-based manner is also proposed in this paper with further concerns addressed.

Automatic techniques for 3D reconstruction of critical workplace body postures from range imaging data

The paper shows techniques for the determination of structured motion parameters from range camera image sequences. The core contribution of the work presented here is the development of an integrated least squares 3D tracking approach based on amplitude and range image sequences to calculate dense 3D motion vector fields. Geometric primitives of a human body model are fitted to time series of range camera point clouds using these vector fields as additional information. Body poses and motion information for individual body parts are derived from the model fit. On the basis of these pose and motion parameters, critical body postures are detected. The primary aim of the study is to automate ergonomic studies for risk assessments regulated by law, identifying harmful movements and awkward body postures in a workplace.

Graphic-processing-unit-accelerated real-time exposure fusion method using pixel-level optimal exposure criterion

High dynamic range (HDR) imaging is an important and challenging research topic in computational photography. A simple but effective image fusion method is proposed to accomplish the multi-exposure image composition in both static and dynamic scenes. The foundation of the proposed method is an experiential criterion that optimizes the exposure that occurs at a dramatic alteration point in the low dynamic range image sequence (LDRI). To extract these well-exposed pixel vectors, each pixel curve formed by the pixel vectors at same position along all frames in the LDRIs is first preprocessed by the chord length parameterization. Then a single high-quality pseudo-HDR image can be extracted directly and efficiently from the LDRIs using a pixel-level fusion index matrix derived from the first- and second-order difference quotients of the preprocessed pixel curves. The main advantage of the proposed method is its use of a single independent pixel in computing. It is highly parallel, allowing a graphic processing unit-based, real-time implementation. The experiments on various scenes discussed here indicate that the proposed exposure fusion method can combine a large image sequence with 10 megapixels into a visually compelling pseudo-HDR image at a rate of 30 frames/s on a consumer hardware.

Range Flow in Varying Illumination: Algorithms and Comparisons

We extend estimation of range flow to handle brightness changes in image data caused by inhomogeneous illumination. Standard range flow computes 3D velocity fields using both range and intensity image sequences. Toward this end, range flow estimation combines a depth change model with a brightness constancy model. However, local brightness is generally not preserved when object surfaces rotate relative to the camera or the light sources, or when surfaces move in inhomogeneous illumination. We describe and investigate different approaches to handle such brightness changes. A straightforward approach is to prefilter the intensity data such that brightness changes are suppressed, for instance, by a highpass or a homomorphic filter. Such prefiltering may, though, reduce the signal-to-noise ratio. An alternative novel approach is to replace the brightness constancy model by 1) a gradient constancy model, or 2) by a combination of gradient and brightness constancy constraints used earlier successfully for optical flow, or 3) by a physics-based brightness change model. In performance tests, the standard version and the novel versions of range flow estimation are investigated using prefiltered or nonprefiltered synthetic data with available ground truth. Furthermore, the influences of additive Gaussian noise and simulated shot noise are investigated. Finally, we compare all range flow estimators on real data.

Hand-held 3D scanning based on coarse and fine registration of multiple range images

A hand-held 3D scanning technique is proposed to reconstruct 3D models of real objects. A sequence of range images captured from a hand-held stereo camera is automatically registered to a reference coordinate system. The automated scanning process consists of two states, coarse and fine registration. At the beginning, scanning process starts at the fine registration state. A fast and accurate registration refinement technique is used to align range images in a pair-wise manner. If the refinement technique fails, the process changes to the coarse registration state. A feature-based coarse registration technique is proposed to find correspondences between the last successful frame and the current frame. If the coarse registration successes, the process returns to the fine registration state again. A fast point-to-plane refinement technique is employed to do shape-based registration. After the shape-based alignment, a texture-based refinement technique matches texture features to enhance visual appearance of the reconstructed models. Through a graphic and video display, a human operator adjusts the pose of the camera to change the view of the next acquisition. Experimental results show that 3D models of real objects are reconstructed from sequences of range images.

1A2-G05 距離画像時系列データを用いた平面上のデブリ衝突痕形状・位置同定

This study proposes a method to autonomously identify sizes and locations of the small impact craters on planar surface such as the one of spacecraft's solar array. The method is realized with the 2D-scanning range sensor in 6DOF motion over the measuring surface, and its motion parameters should be estimated to determine each crater's location as well as its geometry. In order to precisely extract the geometrical property of the craters from the range images, two invariant parameters based on 3D curvature is utilized. Also, the sensor's 6DOF velocities are estimated from the range flow data, which is computed from two successive range images. The validity of the proposed method is demonstrated through a numerical simulation using synthetic range image sequence.

4021 カメラおよび距離画像を用いた,着陸機運動・特徴点位置同時推定(S61-1 宇宙システムの誘導・制御(1),21世紀地球環境革命の機械工学:人・マイクロナノ・エネルギー・環境)

This paper proposes a simultaneous estimation method for planetary lander's motion parameters and 3-D locations of the scene points on planetary surface using CCD camera and scanning range sensor. First, the relationship between the lander's motion and the scene points' motion in camera and range image sequence is derived, and described as a couple of state and observation equations. Secondly, an estimation scheme utilizing two Extended Kalman Filter (EKF) algorithms is derived for the lnader's 6DOF motion parameters and the scene points' 3D locations. Finally, the effectiveness of the proposed method is demonstrated through numerical simulation.

Flying Laser Range Sensor for Large-Scale Site-Modeling and Its Applications in Bayon Digital Archival Project

We have been conducting a project to digitize the Bayon temple, located at the center of Angkor-Thom in the kingdom of Cambodia. This is a huge structure, more than 150 meters long on all sides and up to 45 meters high. Digitizing such a large-scale object in fine detail requires developing new types of sensors for obtaining data of various kinds related to irregular positions such as the very high parts of the structure occluded from the ground. In this article, we present a sensing system with a moving platform, referred to as the Flying Laser Range Sensor (FLRS), for obtaining data related to these high structures from above them. The FLRS, suspended beneath a balloon, can be maneuvered freely in the sky and can measure structures invisible from the ground. The obtained data, however, has some distortion due to the movement of the sensor during the scanning process. In order to remedy this issue, we have developed several new rectification algorithms for the FLRS. One method is an extension of the 3D alignment algorithm to estimate not only rotation and translation but also motion parameters. This algorithm compares range data of overlapping regions from ground-based sensors and our FLRS. Another method accurately estimates the FLRS's position by combining range data and image sequences from a video camera mounted on the FLRS. We evaluate these algorithms using a IS-based method and verify that both methods achieve much higher accuracy than previous methods.

Model-Based Tracking by Classification in a Tiny Discrete Pose Space

A method is presented for tracking 3D objects as they transform rigidly in space within a sparse range image sequence. The method operates in discrete space and exploits the coherence across image frames that results from the relationship between known bounds on the object's velocity and the sensor frame rate. These motion bounds allow the interframe transformation space to be reduced to a reasonable and indeed tiny size, comprising only tens or hundreds of possible states. The tracking problem is in this way cast into a classification framework, effectively trading off localization precision for runtime efficiency and robustness. The method has been implemented and tested extensively on a variety of freeform objects within a sparse range data stream comprising only a few hundred points per image. It has been shown to compare favorably against continuous domain Iterative Closest Point (ICP) tracking methods, performing both more efficiently and more robustly. A hybrid method has also been implemented that executes a small number of ICP iterations following the initial discrete classification phase. This hybrid method is both more efficient than the ICP alone and more robust than either the discrete classification method or the ICP separately.