Image Triplets Research Articles

RECTIFICATION AND ROBUST MATCHING USING ORIENTED IMAGE TRIPLETS FOR MINIMALLY INVASIVE SURGERY

Stereo endoscopes for minimally invasive surgery have been available on the market for several years and are well established in some areas. In practice, they offer a stereoscopic view to the surgeon but are not yet intended for 3D measurements. However, using current knowledge about the camera system and the difficult conditions in object space, it is possible to reconstruct a highly accurate surface model of the current endoscopic view. In particular for medical interventions, a highly reliable point cloud and real-time computation are required. To obtain good reliability, a miniaturised trinocular camera system is introduced that reduces the amount of outliers. To reduce computation time, an approach for generation of rectified image triplets and their corresponding interior and exterior camera parameters has been developed. With these modified and parameterised images it is possible to directly process 3D measurements in object space. Accordingly, an efficient semi-global optimisation is implemented by the authors. In this paper the special camera system, the rectification approach and the applied methodology of matching in rectified image triplets are explained. Finally, first results are presented. In conclusion, the trinocular camera system provides more reliable point clouds than a binocular one, especially for areas with repetitive or poor texture. Currently, the benefit of the third camera is not as great as desired.

Visualization of acceleration in multiphase fluid interactions.

Probing the dynamics of structures in turbid media is important for understanding the internal forces that drive the time evolution of many fluid systems; the breakup of fuel injection sprays is a prime example. We demonstrate a three-pulse configuration for time-gated ballistic imaging, applied to a turbulent, steady spray allowing the acquisition of time-correlated image data. Coupled with targeted region-matching analysis, the detected image triplets are used to generate time-resolved velocity and acceleration vectors representing motion and forces involved in spray development.

Blind Stereoscopy of the Coronal Magnetic Field

We test the feasibility of 3D coronal-loop tracing in stereoscopic EUV image pairs, with the ultimate goal of enabling efficient 3D reconstruction of the coronal magnetic field that drives flares and coronal mass ejections (CMEs). We developed an automated code designed to perform triangulation of coronal loops in pairs (or triplets) of EUV images recorded from different perspectives. The automated (or blind) stereoscopy code includes three major tasks: (i) automated pattern recognition of coronal loops in EUV images, (ii) automated pairing of corresponding loop patterns from two different aspect angles, and (iii) stereoscopic triangulation of 3D loop coordinates. We perform tests with simulated stereoscopic EUV images and quantify the accuracy of all three procedures. In addition we test the performance of the blind stereoscopy code as a function of the spacecraft-separation angle and as a function of the spatial resolution. We also test the sensitivity to magnetic non-potentiality. The automated code developed here can be used for analysis of existing {\sl Solar TErrestrial RElationship Observatory (STEREO)} data, but primarily serves for a design study of a future mission with dedicated diagnostics of non-potential magnetic fields. For a pixel size of 0.6\arcsec (corresponding to the {\sl Solar Dynamics Observatory (SDO) Atmospheric Imaging Assembly (AIA)} spatial resolution of 1.4\arcsec), we find an optimum spacecraft-separation angle of $\alpha_s \approx 5^\circ$.

Weakly Semi-Supervised Deep Learning for Multi-Label Image Annotation

In this paper, we study leveraging both weakly labeled images and unlabeled images for multi-label image annotation. Motivated by the recent advance in deep learning, we propose an approach called we akly se mi-supervised d eep learning for multi-label image annotation (WeSed). In WeSed, a novel weakly weighted pairwise ranking loss is effectively utilized to handle weakly labeled images, while a triplet similarity loss is employed to harness unlabeled images. WeSed enables us to train deep convolutional neural network (CNN) with images from social networks where images are either only weakly labeled with several labels or unlabeled. We also design an efficient algorithm to sample high-quality image triplets from large image datasets to fine-tune the CNN. WeSed is evaluated on benchmark datasets for multi-label annotation. The experiments demonstrate the effectiveness of our proposed approach and show that the leverage of the weakly labeled images and unlabeled images leads to a significantly better performance.

Spectral index utility for summer urban heating analysis

The land surface temperature (LST) retrieval in an urban environment by thermal remote sensing is a widespread scientific topic and several studies have been made to point out the correspondence between spectral indices and LST pattern. This work evaluates the potential of the spectral indices introduced in the literature, more than 30, by assessing their correlation with the summer LST in the heterogeneous urban area of Rome, Italy, considering two different triplets of images acquired during 2009 and 2011 by the Landsat Thematic Mapper (TM). The spectral indices have been divided into vegetation and built up-soil indices employing the reflective TM bands, and then ranked on the basis of their linear and monotonic relationship with the LST. Vegetation indices have a strong negative correlation with LST: vegetation area (VA), nonlinear index, modified soil adjusted vegetation index exhibit a greater Pearson and Spearman correlation coefficient with LST. The more useful spectral indices for built up and soil analysis, exhibiting a greater positive correlation with LST, are the impervious surface area (ISA), the bare soil index, the index-based built-up index, and the normalized difference built-up index. Interesting indications of the impact on the spectral index performance of specific land-cover classes embedded in an urban environment, such as the bare soil and the water classes were pointed out; for example, the reduction of the ISA and VA capability to display the full dynamic range of the LST pattern.

Understanding the Influence of Assimilating Subsets of Enhanced Atmospheric Motion Vectors on Numerical Analyses and Forecasts of Tropical Cyclone Track and Intensity with an Ensemble Kalman Filter

Abstract Recent studies have shown that assimilating enhanced satellite-derived atmospheric motion vectors (AMVs) has improved mesoscale forecast of tropical cyclones (TC) track and intensity. The authors conduct data-denial experiments to understand where the TC analyses and forecasts benefit the most from the enhanced AMV information using an ensemble Kalman filter and the Weather Research and Forecasting Model. The Cooperative Institute for Meteorological Satellite Studies at the University of Wisconsin provides enhanced AMV datasets with higher density and temporal resolution using shorter-interval image triplets for the duration of Typhoon Sinlaku and Hurricane Ike (both 2008). These AMV datasets are then spatially and vertically subsetted to create six parallel cycled assimilation-forecast experiments for each TC: all AMVs; AMVs withheld between 100 and 350 hPa (upper layer), between 350 and 700 hPa (middle layer), and between 700 and 999 hPa (lower layer); and only AMVs within (interior) and outside (exterior) 1000-km radius of the TC center. All AMV subsets are found to be useful in some capacity. The interior and upper-layer AMVs are particularly crucial for improving initial TC position, intensity, and the three-dimensional wind structure along with their forecasts. Compared with denying interior or exterior AMVs, withholding AMVs in different tropospheric layers had less impact on TC intensity and size forecasts. The ensemble forecast is less certain (larger spread) in providing accurate TC track, intensity, and size when upper-layer AMVs or interior AMVs are withheld. This information could be useful to potential targeting scenarios, such as activating and focusing satellite rapid-scan operations, and decisions regarding observing system assessments and deployments.

Tightly-coupled stereo visual-inertial navigation using point and line features.

This paper presents a novel approach for estimating the ego-motion of a vehicle in dynamic and unknown environments using tightly-coupled inertial and visual sensors. To improve the accuracy and robustness, we exploit the combination of point and line features to aid navigation. The mathematical framework is based on trifocal geometry among image triplets, which is simple and unified for point and line features. For the fusion algorithm design, we employ the Extended Kalman Filter (EKF) for error state prediction and covariance propagation, and the Sigma Point Kalman Filter (SPKF) for robust measurement updating in the presence of high nonlinearities. The outdoor and indoor experiments show that the combination of point and line features improves the estimation accuracy and robustness compared to the algorithm using point features alone.

DEM GENERATION WITH SHORT BASE LENGTH PLEIADES TRIPLET

Abstract. An image triplet of Pleiades images covering the area of Zonguldak, Turkey has been investigated. The height to base relation of the first to the last image is just 1:4.5 and for the first and the second image 1:9. This is quite below the usual height to base relation of 1:1.6 for a typical stereo pair of space images. The corresponding small angle of convergence influences the possible vertical accuracy, but images with such a small angle of convergence are more similar to each other as images with larger convergence angles. This enables a better image matching, improving the vertical accuracy and compensating partially the influence of poor intersection geometry. Even over forest areas no matching gaps occurred. Height models are generated with different base configurations and compared with a reference height model. Pleiades images are distributed with 50cm ground sampling distance instead of the physical size of 70cm, the image quality justifies this zooming and also the geometric results are in the range of other space images with originally 50cm GSD. The image orientation by bias corrected Rational Polynomial Coefficients (RPC) is leading with more as 160 ground control points (GCP) to root mean square (RMS) differences slightly below 1.0 GSD of the distributed images (0.5m GSD). Only negligible systematic errors have been identified. With the combination of the first and last image a standard deviation of the generated height model of 1.6m, respectively for flat terrain close to 1.0m has been reached in relation to a reference height model. The small angle of convergence is not as much influencing the height accuracy as according to simple geometric relation.

Semi-Global Matching of Pléiades Tri-Stereo Imagery to Generate Detailed Digital Topography for High-Alpine Regions

Nowadays, several sensors are capable of acquiring very high resolution (VHR) satellite images in stereo, and even in tri-stereo mode. These data can be efficiently exploited with photogrammetric methods, such as semi-global matching (SGM), to produce high quality 2.5D digital terrain information. Regarding the production of digital terrain models, since VHR (tri-)stereo satellite images are available at high temporal frequencies for nearly the whole globe, they can be considered an attractive alternative to restricted and expensive airborne laserscanning (ALS) data. In this study, SGM is applied to a Pléiades-1A image triplet, to generate an accurate and detailed 3D point cloud and a 2.5D digital surface model (DSM) for a high-alpine study area in South Tyrol with complex, snow- and ice-covered topography. Visual and quantitative comparison to an ALS reference DSM showed that there were only small elevation errors for large parts of the study area. Errors were higher in regions that were obscured by shade. This is one of the first studies demonstrating that SGM is not only capable of delivering reliable digital surface information for flat and hilly regions, but also for high-alpine areas with complex terrain.

Scan line optimization for Tri stereo planetary images

Abstract. In this paper, we propose a new scan line optimization method for matching the triplet of images. In the present paper, the triplets are initially matched using an area based local method. The cost is stored in a structure called as the Disparity Space Image (DSI). Using the global minimum of this cost the initial disparity is generated. Next the local minima are considered as potential matches where global minimum gives erroneous results. These local minima are used for optimization of disparity. As the method is a scanned line optimization, it use popularly resampled images. The experiment is performed using Terrain Mapping Camera images from the Chandrayaan-1 mission. In order to validate the result for accuracy, Lunar Orbiter Laser Altimeter dataset from Lunar Reconnaissance Orbiter mission is used. The method is again verified using standard Middlebury stereo dataset with ground truth. From experiments, it has been observed that using optimization technique for triplets, the total number of correct matches has increased by 5–10 % in comparison to direct methods. The method particularly gives good results at smooth regions, where dynamic programming and blockmatching gives limited accuracy.

Statistical learning of serial visual transitions by neurons in monkey inferotemporal cortex.

If monkeys repeatedly, over the course of weeks, view displays in which two images appear in fixed sequence, then neurons of inferotemporal cortex (ITC) come to exhibit prediction suppression. The response to the trailing image is weaker if it follows the leading image with which it was paired during training than if it follows some other leading image. Prediction suppression is a plausible neural mechanism for statistical learning of visual transitions such as has been demonstrated in behavioral studies of human infants and adults. However, in the human studies, subjects are exposed to continuous sequences in which the same image can be both predicted and predicting and statistical dependency can exist between nonadjacent items. The aim of the present study was to investigate whether prediction suppression in ITC develops under such circumstances. To resolve this issue, we exposed monkeys repeatedly to triplets of images presented in fixed order. The results indicate that prediction suppression can be induced by training not only with pairs of images but also with longer sequences.

An effective biomedical image retrieval framework in a fuzzy feature space employing Phase Congruency and GeoSOM

This paper presents a detailed study about a biomedical image retrieval framework by extracting Phase Congruency (PC) features from L*a*b* triplets of images (query, target) and representing them in fuzzy feature space. These features correspond to an edge-corner map of the given image. The resulting map is then processed by Scale Invariant Feature Transform (SIFT) to derive keypoints, that are invariant to affine transformations. The ensuing features were vector quantized to build a codebook of keypoints. The codebook was produced using a Spherical Self-Organizing Map (SOM) built with a geodesic data structure termed as GeoSOM. Then keypoints of the query image are mapped with the codebook and their occurrences are counted to formulate a histogram termed as Phase Congruency-based Bag of Keypoints (PC-BoK). This histogram is generated offline for target images and a similarity measure was performed with the query image to yield the nearest match based on a global fuzzy membership function. Exhaustive experiments of the proposed framework named as BIRS (Biomedical Image Retrieval System) were performed on a diverse medical image collection (NBIA, MESSIDOR, DRIVE). Finally, performance of BIRS demonstrates the advantage of the proposed image representation approach in terms of Precision (P)–Recall (R) parameters. Furthermore relative comparison of the proposed scheme with existing feature descriptors depicts improved P–R values. The proposed feature extraction and representation scheme was also robust against quantization errors.

DETECTION OF CRITICAL CAMERA CONFIGURATIONS FOR STRUCTURE FROM MOTION

Abstract. This paper deals with the detection of critical, i.e., poor or degenerate camera configurations, with a poor or undefined intersection geometry between views. This is the basis for a calibrated Structure from Motion (SfM) approach employing image triplets for complex, unordered image sets, e.g., obtained by combining terrestrial images and images from small Unmanned Aerial Systems (UAS). Poor intersection geometry results from a small ratio between the baseline length and the depth of the scene. If there is no baseline between views, the intersection geometry becomes undefined. Our approach can detect image pairs without or with a very weak baseline (motion degeneracy). For the detection we have developed various metrics and evaluated them by means of extensive experiments with about 1500 image pairs. The metrics are based on properties of the reconstructed 3D points, such as the roundness of the error ellipsoid. The detection of weak baselines is formulated as a classification problem using the metrics as features. Machine learning techniques are applied to improve the classification. By taking into account the critical camera configurations during the iterative composition of the image set, a complete, metric 3D reconstruction of the whole scene could be achieved also in this case. We sketch our approach for the orientation of unordered image sets and finally demonstrate that the approach is able to produce very accurate and reliable orientations.

A NEW APPROACH FOR AN INCREMENTAL ORIENTATION OF MICRO-UAV IMAGE SEQUENCES

Abstract. Civil applications for small size unmanned aerial vehicles (UAV) have become quite important in recent years and so have accurate orientation and navigation of these devices in unknown terrain. In this work we focus on on-line compatible positioning in facade observation based on monocular low resolution still images acquired by a camera mounted on a UAV. Also, a 3D point cloud of the facade is generated. This allows further processing steps, e.g. navigation assistance, collision avoidance or the evaluation of the point cloud density, verifying completeness of the data. To be able to deal with the increasing amount of observations and unknown parameters we implement an incremental bundle adjustment based on automatically determined tie points and sliding image triplets. The tripletwise orientation allows for an efficient double cross-check of the detected feature points and hence guarantees reliable initial values for the nonlinear bundle adjustment. The initial values are estimated within a convex formulation delivering a sound basis for the incremental adjustment. Our algorithm is evaluated by means of imagery we took of the facade of the Welfenschloss in Hannover, captured from a manually flown Microdrones md4-200 micro-UAV.We compare the orientation results of our approach with an approach in which initial values for the unknown object coordinates are computed algebraically.

Structure and motion from line correspondences: Representation, projection, initialization and sparse bundle adjustment

We address the problem of structure and motion from line correspondences, which ranges from the representation of lines, their projections and the initialization procedure to the final adjustment. The Cayley representation of spatial lines is developed, which is a nonlinear minimal parametrization circumventing the tiresome Plücker constraint. The relationships between different line representations are given. Based on these relationships, we derive a novel line projection function which is consistent with the previous results. After building the line observation model, we employ a closed-form solution for the first image triplet, then develop an incremental initialization approach to initialize the motion and structure parameters. Finally, the sparse bundle adjustment (SBA) is applied to refine the parameters, which updates the spatial lines by using the Cayley representation with an unconstrained optimization engine. The experiments show that the proposed algorithm outperforms the previous works both in efficiency and accuracy.

ON-LINE COMPATIBLE ORIENTATION OF A MICRO-UAV BASED ON IMAGE TRIPLETS

Abstract. In this paper we present a robust orientation approach for an imaging sensor flown on a micro-UAV based on image triplets. Our aim is to have the orientation available online, i.e. during image acquisition. The resulting point cloud and sensor orientations can then for instance be evaluated for navigation purposes of the UAV or to analyse the completeness of the point cloud. We use low quality imagery extracted from the downlink of an onboard PAL-camera. Trilinear constraints and cross-checked matches allow for a high robustness of the sensor orientation and the sparse 3D point cloud. In order to reach the goal of on-line processing given the large number of observations and unknowns, we make use of an incremental bundle adjustment. Estimated parameters are incrementally improved without explicitly considering previous observations. Our approach combines linear projective geometry for obtaining initial values using the trifocal tensor with non-linear perspective geometry for the estimation of the unknowns. This combination allows for a high precision of the estimation, while eliminating the need for initial values. We evaluate the performance of our approach by means of imagery we acquired of the facade of theWelfenschloss in Hannover, collected with a Microdrones md4-200 micro-UAV. The results are the orientation parameters of the images and a sparse 3D point cloud representing the object. They are compared to those from a commercial bundle adjustment software and analysed in terms of geometric precision.

Image annotation using high order statistics in non-Euclidean spaces

Automatic image annotation is a promising way to achieve more effective image retrieval and image analysis by using keywords associated to the image content. Due to the semantic gap between low-level visual features and high-level semantic concepts of an image, however, the performances of many existing algorithms are not so satisfactory. In this paper, a novel image classification scheme, named high order statistics based maximum a posterior (HOS-MAP), is proposed to deal with the issue of image annotation. To bridge the gap between human judgment and machine intelligence, the proposed scheme first constructs a dissimilarity representation for each image in a non-Euclidean space; then, the information of dissimilarity diffusion distribution for each image is achieved with respect to the high-order statistics of a triplet of nearest neighbor images; finally, a maximum a posteriori algorithm with the information of Gaussian Mixture Model and dissimilarity diffusion distribution is adopted to estimate the relevance between each annotation and an input un-annotated image. Experimental results on a general-purpose image database demonstrate the effectiveness and efficiency of the proposed automatic image annotation scheme.

A Coarse Elevation Map-based Registration Method for Super-resolution of Three-line Scanner Images

Three-line scanner imagery provides three overlapped images in an along-base direction, and creates a possible avenue to obtain higher quality images through the application of the super-resolution. Accurate co-registration of the three images is a key step for super-resolution. However, discontinuities and occlusions resulting from the 3D-to-2D projection cause mis-registration in traditional 2D-image-level co-registration methods. In this paper, we address this problem by introducing 3D information extracted from image triplets by using GPS/IMU data as an approximation. The core of the proposed method is to use a number of height layers derived from feature points and image partitions, in the form of a coarse elevation map (CEM), as a 3D constraint to restrict registration on the corresponding height. In terms of super-resolution, we also propose a tree-based fast adaptive template matching method for Knife-edge detection to fully automate the SRKE super-resolution algorithm. Experimental results show that the proposed method produces improved registered images and accordingly yields signifi cant resolution enhancement as compared to other methods.

ANALYTICAL SOLUTIONS OF SINGULAR ISOTHERMAL QUADRUPOLE LENS

Using analytical method, we study the Singular Isothermal Quadrupole (SIQ) lens system, which is the simplest lens model that can produce four images. In this case, the radial mass distribution is in accord with the profile of the Singular Isothermal Sphere (SIS) lens, and the tangential distribution is given by adding a quadrupole on the monopole component. The basic properties of the SIQ lens have been studied in this paper, including deflection potential, deflection angle, magnification, critical curve, caustic, pseudo-caustic and transition locus. Analytical solutions of the image positions and magnifications for the source on axes are derived. As have been found, naked cusps will appear when the relative intensity $k$ of quadrupole to monopole is larger than 0.6. According to the magnification invariant theory of the SIQ lens, the sum of the signed magnifications of the four images should be equal to unity \citep{dal98}. However, if a source lies in the naked cusp, the summed magnification of the left three images is smaller than the invariant 1. With this simple lens system, we study the situations that a point source infinitely approaches a cusp or a fold. The sum of magnifications of cusp image triplet is usually not equal to 0, and it is usually positive for major cusp while negative for minor cusp. Similarly, the sum of magnifications of fold image pair is usually neither equal to 0. Nevertheless, the cusp and fold relations are still equal to 0, in that the sum values are divided by infinite absolute magnifications by definition.

3D object reconstruction from multiple views using neural networks

Automatic determination of 3D information from digital images is a fundamental problem in digital photogrammetry and computer vision. The most difficult task in the problem is finding conjugate points in images. Despite the wealth information contained in stereo images, factors such as occlusions and discontinuities challenge several algorithms implemented to match conjugate points. On the other hand, multi-image matching has gained considerable attention recently since it accounts for changes in illumination conditions, camera positions, and surrounding features. Currently, a wide variety of techniques have been developed for multi-image matching. However, most of these techniques are based on one of two approaches: grouping outputs from stereo matching or looking for corresponding points in all views. The first approach results in scenarios where several false matches could be grouped. The second approach is limited to cases where points appear in all views with comparable intensity values. This paper introduces an alternative algorithm to match image points across several views. Our approach is based on locating all sets of corresponding points in all image triplets using neural networks then grouping those triplet sets that represent the same ground point. Radiometric and geometric properties are fed into a feedforward neural network designed to detect valid matches from invalid ones in all image triplets simultaneously. The detection rate of the neural network is more than 98 % and the false alarm rate is less than 2 %. Triplet matching sets are then combined if they share the same image point and their image rays have a valid intersection as measured through the colinearity equation. In order to assess the geometrical accuracy of the 3D points, the image coordinates are manually measured in the original images and object coordinates are computed and found comparable to those computed through the proposed algorithm.