Inter-frame Registration Research Articles

An effective point cloud registration method for three-dimensional reconstruction of pressure piping

Abstract At present, industrial scenes with sparse features and weak textures are widely encountered, and the three-dimensional reconstruction of such scenes is a recognized problem. Pressure pipelines have a wide range of applications in fields such as petroleum engineering, chemical engineering, and hydropower station engineering. However, there is no mature solution for the three-dimensional reconstruction of pressure pipes. The main reason is that the typical scenes in which pressure pipes are found also have relatively few features and textures. Traditional three-dimensional reconstruction algorithms based on feature extraction are largely ineffective for such scenes that are lacking in features. In view of the above problems, this paper proposes an improved interframe registration algorithm based on point cloud fitting with cylinder axis vector constraints. By incorporating geometric feature parameters of a cylindrical pressure pipeline, specifically the axis vector of the cylinder, to constrain the traditional iterative closest point algorithm, the accuracy of point cloud registration can be improved in scenarios lacking features and textures, and some environmental uncertainties can be overcome. Finally, using actual laser point cloud data collected from pressure pipelines, the proposed fitting-based point cloud registration algorithm with cylinder axis vector constraints is tested. The experimental results show that under the same conditions, compared with other open-source point cloud registration algorithms, the proposed method can achieve higher registration accuracy. Moreover, integrating this algorithm into an open-source three-dimensional reconstruction algorithm framework can lead to better reconstruction results.

S2-PCM: Super-Resolution Structural Point Cloud Matching for High-Accuracy Video-SAR Image Registration

In this paper, the super-resolution structural point cloud matching (S2-PCM) framework is proposed for video synthetic aperture radar (SAR) inter-frame registration, which consists of a feature recurrence super-resolution network (FRSR-Net), structural point cloud extraction network (SPCE-Net) and robust point matching network (RPM-Net). FRSR-Net is implemented by integrating the feature recurrence structure and residual dense block (RDB) for super-resolution enhancement, SPCE-Net is implemented by training a U-Net with data augmentation, and RPM-Net is applied for robust point cloud matching. Experimental results show that compared with the classical SIFT-like algorithms, S2-PCM achieves higher registration accuracy for video-SAR images under diverse evaluation metrics, such as mutual information (MI), normalized mutual information (NMI), entropy correlation coefficient (ECC), structural similarity (SSIM), etc. The proposed FRSR-Net can significantly improve the quality of video-SAR images and point cloud extraction accuracy. Combining FRSR-Net with S2-PCM, we can obtain higher inter-frame registration accuracy, which is crucial for moving target detection and shadow tracking.

Space-based infrared aerial target detection method via interframe registration and spatial local contrast

Infrared aerial target detection on a space-based platform is of great importance in the field of aerospace. However, the intensities of clutter may be much stronger than those of aerial targets in space-based infrared images, which causes issues in accurate aerial target detection. Existing detection methods cannot adapt well to such a situation, and often receive results with missed detections and false alarms since they are prone to enhance clutter rather than the target. To tackle such limitations, we propose a two-stage Interframe Registration and Spatial Local Contrast (IFR-SLC) based method for space-based infrared aerial target detection. In the first stage, the interframe registration model is constructed to suppress strong clutter and highlight the target. In the second stage, the spatial local contrast model is introduced to further suppress the residual background and enhance the target, which eventually generates the saliency map with only the aerial target remaining. Target extraction is conducted by setting an adaptive threshold. Five space-based sequences with different backgrounds were selected for evaluation, and experimental results indicated better performances of IFR-SLC compared with those of space-based or state-of-the-art detection methods.

Robust Infrared Maritime Target Detection via Anti-Jitter Spatial–Temporal Trajectory Consistency

When detecting diverse infrared (IR) small maritime targets on complicated scenes, the existing methods get into trouble and unsatisfactory performance. The main reasons are as follows: 1) affected by target characteristics and ambient temperature and so on, both bright and dark targets may exist on IR maritime images in practical application and 2) spatial information and temporal correlation of targets are not fully excavated. To these problems, we propose a robust anti-jitter spatial–temporal trajectory consistency (ASTTC) method, the main idea of which is to improve detection accuracy using single-frame detection followed by multi-frame decision. First, we innovatively design adaptive local gradient variation (ALGV) descriptor, which combines local dissimilarity measure with gradient magnitude distribution to enhance the local contrast for both bright and dark targets so that the suspected targets can be robustly extracted. For multi-frame decision, interframe displacement correction is necessary to eliminate the interference of IR imager motion and vibration for target codeword. We use pyramid optical flow to track feature point extracted by Shi-Tomasi to capture interframe registration coefficients. Then, the target position is corrected in the spatial–temporal domain. Finally, a robust spatial–temporal trajectory descriptor (STTD), which achieves target encoding and target trajectory consistency measurement, is designed to further confirm real targets and eliminate false targets. Experiments conducted on various real IR maritime image sequences demonstrate the applicability and robustness of ASTTC, which performs more satisfactorily than the existing methods in detection accuracy.

Thermal Infrared Video Stabilization for Aerial Monitoring of Active Wildfires

Measuring wildland fire behavior is essential for fire science and fire management. Aerial thermal infrared (TIR) imaging provides outstanding opportunities to acquire such information remotely. Variables such as fire rate of spread (ROS), fire radiative power (FRP), and fireline intensity may be measured explicitly both in time and space, providing the necessary data to study the response of fire behavior to weather, vegetation, topography, and firefighting efforts. However, raw TIR imagery acquired by unmanned aerial vehicles (UAVs) requires stabilization and georeferencing before any other processing can be performed. Aerial video usually suffers from instabilities produced by sensor movement. This problem is especially acute near an active wildfire due to fire-generated turbulence. Furthermore, the nature of fire TIR video presents some specific challenges that hinder robust interframe registration. Therefore, this article presents a software-based video stabilization algorithm specifically designed for TIR imagery of forest fires. After a comparative analysis of existing image registration algorithms, the KAZE feature-matching method was selected and accompanied by pre- and postprocessing modules. These included foreground histogram equalization and a multireference framework designed to increase the algorithm's robustness in the presence of missing or faulty frames. The performance of the proposed algorithm was validated in a total of nine video sequences acquired during field fire experiments. The proposed algorithm yielded a registration accuracy between 10 and 1000× higher than other tested methods, returned 10× more meaningful feature matches, and proved robust in the presence of faulty video frames. The ability to automatically cancel camera movement for every frame in a video sequence solves a key limitation in data processing pipelines and opens the door to a number of systematic fire behavior experimental analyses. Moreover, a completely automated process supports the development of decision support tools that can operate in real time during an emergency.

FIFNET: A convolutional neural network for motion-based multiframe super-resolution using fusion of interpolated frames

We present a novel motion-based multiframe image super-resolution (SR) algorithm using a convolutional neural network (CNN) that fuses multiple interpolated input frames to produce an SR output. We refer to the proposed CNN and associated preprocessing as the Fusion of Interpolated Frames Network (FIFNET). We believe this is the first such CNN approach in the literature to perform motion-based multiframe SR by fusing multiple input frames in a single network. We study the FIFNET using translational interframe motion with both fixed and random frame shifts. The input to the network is a sequence of interpolated and aligned frames. One key innovation is that we compute subpixel interframe registration information for each interpolated pixel and feed this into the network as additional input channels. We demonstrate that this subpixel registration information is critical to network performance. We also employ a realistic camera-specific optical transfer function model that accounts for diffraction and detector integration when generating training data. We present a number of experimental results to demonstrate the efficacy of the proposed FIFNET using both simulated and real camera data. The real data come directly from a camera and are not artificially downsampled or degraded. In the quantitative results with simulated data, we show that the FIFNET performs favorably in comparison to the benchmark methods tested.

Image Similarity Metrics Suitable for Infrared Video Stabilization during Active Wildfire Monitoring: A Comparative Analysis

Aerial Thermal Infrared (TIR) imagery has demonstrated tremendous potential to monitor active forest fires and acquire detailed information about fire behavior. However, aerial video is usually unstable and requires inter-frame registration before further processing. Measurement of image misalignment is an essential operation for video stabilization. Misalignment can usually be estimated through image similarity, although image similarity metrics are also sensitive to other factors such as changes in the scene and lighting conditions. Therefore, this article presents a thorough analysis of image similarity measurement techniques useful for inter-frame registration in wildfire thermal video. Image similarity metrics most commonly and successfully employed in other fields were surveyed, adapted, benchmarked and compared. We investigated their response to different camera movement components as well as recording frequency and natural variations in fire, background and ambient conditions. The study was conducted in real video from six fire experimental scenarios, ranging from laboratory tests to large-scale controlled burns. Both Global and Local Sensitivity Analyses (GSA and LSA, respectively) were performed using state-of-the-art techniques. Based on the obtained results, two different similarity metrics are proposed to satisfy two different needs. A normalized version of Mutual Information is recommended as cost function during registration, whereas 2D correlation performed the best as quality control metric after registration. These results provide a sound basis for image alignment measurement and open the door to further developments in image registration, motion estimation and video stabilization for aerial monitoring of active wildland fires.

Non-uniformity correction of airborne infrared detection system based on inter-frame registration

The predecessor of China Optics was China Optics and Applied Optics Digest, which was founded in 1985. At that time, it was the only retrieval journal in the field of optics in China. At the end of 2008, China Optics and Applied Optics Digest was renamed

Statistical Scene-Based Non-Uniformity Correction Method with Interframe Registration.

The non-uniform response in infrared focal plane array (IRFPA) detectors inevitably produces corrupted images with a fixed-pattern noise. In this paper, we present a novel and adaptive scene-based non-uniformity correction (NUC) method called Correction method with Statistical scene-based and Interframe Registration (CSIR), which realizes low delay calculation of correction coefficient for infrared image. This method combines the statistical method and registration method to achieve a better NUC performance. Specifically, CSIR estimates the gain coefficient with statistical method to give registration method an appropriate initial value. This combination method not only reduces the need of interactive pictures, which means lower time delay, but also achieves better performance compared to the statistical method and other single registration methods. To verify this, real non-uniformity infrared image sequences collected by ourselves were used, and the advantage of CSIR was compared thoroughly on frame number (corresponding to delay time) and accuracy. The results show that the proposed method could achieve a significantly fast and reliable fixed-pattern noise reduction with the effective gain and offset.

Indoor Li-DAR 3D mapping algorithm with semantic-based registration and optimization

The method proposed in this paper using a two-dimensional Li-DAR which moves in six degrees of freedom to construct a three-dimensional point cloud map of the laser traversed environment which includes point cloud feature extraction and registration, global optimization and back-end optimization algorithm, and the constructed 3D point cloud map and the laser trajectory are given. First, the hardware platform of the simultaneous localization and 3D mapping system based on Li-DAR is introduced; then, a semantic-based point cloud feature extraction algorithm is proposed according to the scale invariance of the laser point cloud, the point clouds are registered using the equivalence relation of triangles, and the motion of the laser is calculated between two consecutive scans. Then, a global optimization algorithm is proposed to reduce the cumulative error caused by inter-frame registration. The general map optimization is used to optimize the pose of the Li-DAR, and the comparison results are given. Finally, the three-dimensional point cloud of extraction, registration, laser trajectory, as well as the final 3D point cloud is given. Experimental results show that the proposed Li-DAR-based SLAM system can accurately estimate the trajectory of the Li-DAR and construct a high-quality 3D point cloud in real time. The relative accuracy in the indoor environment is about 2%.

Acute reperfusion without recanalization: Serial assessment of collaterals within 6 h of using perfusion-weighted magnetic resonance imaging

Acute reperfusion despite persistent arterial occlusion may occur in up to 30% of ischemic stroke patients. Recruitment of leptomeningeal collaterals may explain this phenomenon. Using dynamic susceptibility-contrast perfusion imaging (DSC-PI), we assessed acute changes in collateral flow among patients without recanalization. From a multicenter prospective database (I-KNOW), 46 patients with magnetic resonance angiography visible occlusion in whom both reperfusion and recanalization were assessed within 6 h of onset were identified. Maps of collateral flow at arterial, capillary and late venous phases were generated from DSC-PI through inter-frame registration, baseline signal subtraction and temporal summation, and graded blind to all other relevant clinical and radiological data using the Higashida scale. Flow direction and the acute evolution of collaterals were evaluated against the reperfusion status. Among patients without recanalization (n = 33), flow direction remained retrograde. Collateral grades significantly improved between admission and acute follow-up in patients who reperfused (OR: 4.57; 95% CI: 1.1–22.7; p = 0.048), but not in those without reperfusion (OR: 1.34; 95% CI: 0.4–4.5; p = 0.623). Our study confirmed that acute reperfusion without recanalization is associated with a significant improvement of retrograde collateral flow. DSC-PI can detect acute changes in collateral flow, and may help evaluate novel treatments targeting leptomeningeal collaterals.

Robust super-resolution by fusion of interpolated frames for color and grayscale images

Multi-frame super-resolution (SR) processing seeks to overcome undersampling issues that can lead to undesirable aliasing artifacts. The key to effective multi-frame SR is accurate subpixel inter-frame registration. This accurate registration is challenging when the motion does not obey a simple global translational model and may include local motion. SR processing is further complicated when the camera uses a division-of-focal-plane (DoFP) sensor, such as the Bayer color filter array. Various aspects of these SR challenges have been previously investigated. Fast SR algorithms tend to have difficulty accommodating complex motion and DoFP sensors. Furthermore, methods that can tolerate these complexities tend to be iterative in nature and may not be amenable to real-time processing. In this paper, we present a new fast approach for performing SR in the presence of these challenging imaging conditions. We refer to the new approach as Fusion of Interpolated Frames (FIF) SR. The FIF SR method decouples the demosaicing, interpolation, and restoration steps to simplify the algorithm. Frames are first individually demosaiced and interpolated to the desired resolution. Next, FIF uses a novel weighted sum of the interpolated frames to fuse them into an improved resolution estimate. Finally, restoration is applied to deconvolve the modeled system PSF. The proposed FIF approach has a lower computational complexity than most iterative methods, making it a candidate for real-time implementation. We provide a detailed description of the FIF SR method and show experimental results using synthetic and real datasets in both constrained and complex imaging scenarios. The experiments include airborne grayscale imagery and Bayer color array images with affine background motion plus local motion.

Random noise filtering method based on the inter-frame registration

One of the problems to be solved in image processing is how to eliminate image noise effectively. In this work, we brought forward a random noise filtering method based on the inter-frame registration. Firstly, we calculated the relative displacement of the adjacent frames by a registration algorithm. Then we divided the image into the overlapping area and the non-overlapping area according to the relative displacement. Finally, we do noise reduction processing for these two areas respectively. The experiments results indicate that the proposed method can reduce noise in both spatial and time domain of video images. The main advantage is that it cannot only remove noise, but also effectively protect the image edge and detail information. Besides, it not only maintains the de-noising effect of traditional inter-frame algorithm, but also is suitable for moving targets. It has better real-time performance and wider application range.

A Infrared Small Moving Object Extraction Method in the Context of Complex Background Motion

The detection of small moving target in the context of complex background is a difficult issue. A method combining interframe differential registration and adaptive wiener filtering aimed to suppress background to detect moving object in complex background is proposed. The fixed background in the fore-and-aft frames can be filtered out by the interframe registration which preserves the moving target, parts of background and noise due to interframe movement and the gray-scale fluctuation. On one hand the complex background is estimated by an adaptive wiener filter, and the background suppression leaves the high-frequency regions containing the moving target in image. On the other hand, most of the high-frequency regions corresponding to non-target area are eliminated by the inter-frame registration in the differential images. The motion of target is continual in image sequences, while the position of the leaked background is relatively fixed and the noise is of small size. The fusion of the background suppression and inter-frame registration makes the discrimination of targets, background and noise possible. The small moving target is detected by trajectory association based on its interframe trajectory continuity. Experiment results verify the feasibility of the method.

Time-of-Flight camera based 3D point cloud reconstruction of a car

Modern car wash systems are generally controlled by light barriers and power measurement sensors. These sensors are directly mounted on the movable parts of the system which result in a slow regulation, leading to non-optimal performance, e.g. in terms of energy, water and detergent consumption or regarding the quality of the result.This paper presents an alternative approach, based on online automatic acquisition of 3D vehicle models, thus allowing for an a priori optimization of the washing process in order to achieve a better overall performance of the car wash system. Technically, the approach uses merged and synchronized data acquired by a multiple Time-of-Flight (ToF) camera setup. The proposed processing concept handles the range data sets acquired online, and performs data preprocessing, registration and fusion, as well as geometry extraction. The realized system is based on Graphics Processing Units (GPUs) in order to achieve a sufficient temporal processing performance.Furthermore, details are given showing how to solve the main challenges related to point cloud data processing, especially due to low image resolution, the influences of specific environment conditions, e.g. variations in coating and material reflectivity. The data registration is twofold, on the one hand the multiple ToF images are registered using the pre-calibrated camera system (extrinsic parameters), whereas the inter-frame registration is performed using a point-to-plane based iterative closest point (ICP) on the other hand. Finally, the merged data is integrated in a GPU data structure to allow a fast and efficient 3D point cloud reconstruction.

Improved interframe registration based nonuniformity correction for focal plane arrays

In this paper, an improved interframe registration based nonuniformity correction algorithm for focal plane arrays is proposed. The method simultaneously estimates detector parameters and carries out the nonuniformity correction by minimizing the mean square error between the two properly registered image frames. A new masked phase correlation algorithm is introduced to obtain reliable shift estimates in the presence of fixed pattern noise. The use of an outliers exclusion scheme, together with a variable step size strategy, could not only promote the correction precision considerably, but also eliminate ghosting artifacts effectively. The performance of the proposed algorithm is evaluated with clean infrared image sequences with simulated nonuniformity and real pattern noise. We also apply the method to a real-time imaging system to show how effective it is in reducing noise and the ghosting artifacts.

Scene-based nonuniformity correction algorithm based on interframe registration

In this paper, we present a simple and effective scene-based nonuniformity correction (NUC) method for infrared focal plane arrays based on interframe registration. This method estimates the global translation between two adjacent frames and minimizes the mean square error between the two properly registered images to make any two detectors with the same scene produce the same output value. In this way, the accumulation of the registration error can be avoided and the NUC can be achieved. The advantages of the proposed algorithm lie in its low computational complexity and storage requirements and ability to capture temporal drifts in the nonuniformity parameters. The performance of the proposed technique is thoroughly studied with infrared image sequences with simulated nonuniformity and infrared imagery with real nonuniformity. It shows a significantly fast and reliable fixed-pattern noise reduction and obtains an effective frame-by-frame adaptive estimation of each detector's gain and offset.

An efficient image-registration method based on probability density and global parallax

This paper is concerned with the problem of image registration of video sequence, to meet the accuracy and real-time requirements for video analysis and processing. Despite of the existence of sophisticated registration algorithms, it is still problematic to register images precisely with low computation complexity due to the lack of stable features and effective matching method. In this paper, a probability density gradient based interest point detector is employed to extract stable point features precisely. And a robust technique, namely, global parallax histogram based filter is proposed to discard outliers in the initial candidate matches set found by classical correlation method. The registration matrix can then be accurately and precisely estimated using a well adapted criterion. Several field tests are performed on complex and challenging images to assess the performance, including comparison to conventional algorithms, and both inter-frame registration results and statistical analysis of video sequence. These simulations validate the improvement of proposed method in accuracy and efficiency, and the robustness against camera motions, illumination variations, acquirement conditions, moving objects and image noise.

Model‐based registration for dynamic cardiac perfusion MRI

To assess the accuracy of a model-based approach for registration of myocardial dynamic contrast-enhanced (DCE)-MRI corrupted by respiratory motion. Ten patients were scanned for myocardial perfusion on 3T or 1.5T scanners, and short- and long-axis slices were acquired. Interframe registration was done using an iterative model-based method in conjunction with a mean square difference metric. The method was tested by comparing the absolute motion before and after registration, as determined from manually registered images. Regional flow indices of myocardium calculated from the manually registered data were compared with those obtained with the model-based registration technique. The mean absolute motion of the heart for the short-axis data sets over all the time frames decreased from 5.3+/-5.2 mm (3.3+/-3.1 pixels) to 0.8+/-1.3 mm (0.5+/-0.7 pixels) in the vertical direction, and from 3.0+/-3.7 mm (1.7+/-2.1 pixels) to 0.9+/-1.2 mm (0.5+/-0.7 pixels) in the horizontal direction. A mean absolute improvement of 77% over all the data sets was observed in the estimation of the regional perfusion flow indices of the tissue as compared to those obtained from manual registration. Similar results were obtained with two-chamber-view long-axis data sets. The model-based registration method for DCE cardiac data is comparable to manual registration and offers a unique registration method that reduces errors in the quantification of myocardial perfusion parameters as compared to those obtained from manual registration.