Low-level Image Information Research Articles

Pixel-Perfect Structure-From-Motion With Featuremetric Refinement.

Finding local features that are repeatable across multiple views is a cornerstone of sparse 3D reconstruction. The classical image matching paradigm detects keypoints per-image once and for all, which can yield poorly-localized features and propagate large errors to the final geometry. In this paper, we refine two key steps of structure-from-motion by a direct alignment of low-level image information from multiple views: we first adjust the initial keypoint locations prior to any geometric estimation, and subsequently refine points and camera poses as a post-processing. This refinement is robust to large detection noise and appearance changes, as it optimizes a featuremetric error based on dense features predicted by a neural network. This significantly improves the accuracy of camera poses and scene geometry for a wide range of keypoint detectors, challenging viewing conditions, and off-the-shelf deep features. Our system easily scales to large image collections, enabling pixel-perfect crowd-sourced localization at scale. Our code is publicly available athttps://github.com/cvg/pixel-perfect-sfm as an add-on to the popular Structure-from-Motion software COLMAP.

Improving robustness for model discerning synthesis process of uranium oxide with unsupervised domain adaptation

The quantitative characterization of surface structures captured in scanning electron microscopy (SEM) images has proven to be effective for discerning provenance of an unknown nuclear material. Recently, many works have taken advantage of the powerful performance of convolutional neural networks (CNNs) to provide faster and more consistent characterization of surface structures. However, one inherent limitation of CNNs is their degradation in performance when encountering discrepancy between training and test datasets, which limits their use widely. The common discrepancy in an SEM image dataset occurs at low-level image information due to user-bias in selecting acquisition parameters and microscopes from different manufacturers. Therefore, in this study, we present a domain adaptation framework to improve robustness of CNNs against the discrepancy in low-level image information. Furthermore, our proposed approach makes use of only unlabeled test samples to adapt a pretrained model, which is more suitable for nuclear forensics application for which obtaining both training and test datasets simultaneously is a challenge due to data sensitivity. Through extensive experiments, we demonstrate that our proposed approach effectively improves the performance of a model by at least 18% when encountering domain discrepancy, and can be deployed in many CNN architectures.

A Novel Strategy for Extracting Richer Semantic Information Based on Fault Detection in Power Transmission Lines.

With the development of the smart grid, the traditional defect detection methods in transmission lines are gradually shifted to the combination of robots or drones and deep learning technology to realize the automatic detection of defects, avoiding the risks and computational costs of manual detection. Lightweight embedded devices such as drones and robots belong to small devices with limited computational resources, while deep learning mostly relies on deep neural networks with huge computational resources. And semantic features of deep networks are richer, which are also critical for accurately classifying morphologically similar defects for detection, helping to identify differences and classify transmission line components. Therefore, we propose a method to obtain advanced semantic features even in shallow networks. Combined with transfer learning, we change the image features (e.g., position and edge connectivity) under self-supervised learning during pre-training. This allows the pre-trained model to learn potential semantic feature representations rather than relying on low-level features. The pre-trained model then directs a shallow network to extract rich semantic features for downstream tasks. In addition, we introduce a category semantic fusion module (CSFM) to enhance feature fusion by utilizing channel attention to capture global and local information lost during compression and extraction. This module helps to obtain more category semantic information. Our experiments on a self-created transmission line defect dataset show the superiority of modifying low-level image information during pre-training when adjusting the number of network layers and embedding of the CSFM. The strategy demonstrates generalization on the publicly available PASCAL VOC dataset. Finally, compared with state-of-the-art methods on the synthetic fog insulator dataset (SFID), the strategy achieves comparable performance with much smaller network depths.

Context-aware saliency detection for image retargeting using convolutional neural networks

Image retargeting is the task of making images capable of being displayed on screens with different sizes. This work should be done so that high-level visual information and low-level features such as texture remain as intact as possible to the human visual system. At the same time, the output image may have different dimensions. Thus, simple methods such as scaling and cropping are not adequate for this purpose. In recent years, researchers have tried to improve the existing retargeting methods, and they have introduced new ones. However, a specific method cannot be utilized to retarget all types of images. In other words, different images require different retargeting methods. Image retargeting has a close relationship to image saliency detection, which is a relatively new image processing task. Earlier saliency detection methods were based on local and global but low-level image information. These methods are called bottom-up processes. On the other hand, newer approaches are top-down and mixed methods that consider the high level and semantic knowledge of the image too. In this paper, we introduce the proposed methods in both saliency detection and retargeting. For the saliency detection, the use of image context and semantic segmentation are examined, and a novel mixed bottom-up and top-down saliency detection method is introduced. After saliency detection, a modified version of an existing retargeting technique is utilized for retargeting the images. The results suggest that the proposed image retargeting pipeline has excellent performance compared to other tested methods. Also, the subjective evaluations on the Pascal dataset can be used as a retargeting quality assessment dataset for further research.

Automatic liver segmentation by integrating fully convolutional networks into active contour models.

Automatic and accurate three-dimensional (3D) segmentation of liver with severe diseases from computed tomography (CT) images is a challenging task. Fully convolutional networks (FCNs) have emerged as powerful tools for automatic semantic segmentation, with multiple potential applications in medical imaging. However, the use of a large receptive field and multiple pooling layers in the network leads to poor localization around object boundaries. The network usually makes pixel-wise prediction independently, making it difficult to respect local label consistency and enforce the smoothness of the object boundary. We have developed an automatic liver segmentation method based on a novel framework that integrates fully convolutional network predictions into active contour models (ACM). We use only a single network architecture to generate a pixel label map containing spatial regional information (foreground and background) as well as layered boundary information. We exploit the structured network outcome to define an external constraint force of active contour models. A unique property of the designed force is that both its strength and direction are adaptive to its position and relative distance to the object boundary. The resulting integrated active contour models have the advantages of incorporating both high-level and low-level image information simultaneously, while enforcing the smoothness of the contour. Because the external constraint force can push the evolving contour to the liver boundary and exists everywhere in the image domain, it allows us to place the initial contour far away from the liver boundary. It potentially allows us to control the evolution of the contour in order to preserve the topology of the liver. We have trained and evaluated our model on 73 liver CT scans from a clinic study. The integrated ACM model yields mean dice coefficients (DICE) 95.8±1.4 (%). Without further fine-tuning the network weights for two independent datasets, it yields mean DICE 96.2±0.9 (%) for the SLIVER07 training dataset, and mean DICE 94.3±2.7 (%) for the LiTS training dataset. In comparison with FCN alone model, the integrated ACM model yields improvements in terms of surface distance and DICE values for almost all the cases. Furthermore, the initialization of the active contour can be very far away from the liver boundary. Experimental results for segmenting livers (with severe diseases on CT images resulting in shape and density abnormalities) have revealed that our proposed model improves segmentation results in comparison with FCN alone. Without further fine-tuning the network weights for two independent datasets, the model is capable of handling image variations from different datasets due to its inherent deformable nature. It is relatively easy to integrate more advanced (either existing or future) FCN architecture into our framework to further improve the segmentation performance.

Implicit kernel sparse shape representation: a sparse-neighbors-based objection segmentation framework.

This paper introduces a new implicit-kernel-sparse-shape-representation-based object segmentation framework. Given an input object whose shape is similar to some of the elements in the training set, the proposed model can automatically find a cluster of implicit kernel sparse neighbors to approximately represent the input shape and guide the segmentation. A distance-constrained probabilistic definition together with a dualization energy term is developed to connect high-level shape representation and low-level image information. We theoretically prove that our model not only derives from two projected convex sets but is also equivalent to a sparse-reconstruction-error-based representation in the Hilbert space. Finally, a "wake-sleep"-based segmentation framework is applied to drive the evolutionary curve to recover the original shape of the object. We test our model on two public datasets. Numerical experiments on both synthetic images and real applications show the superior capabilities of the proposed framework.

Implicit kernel sparse shape representation: a sparse-neighbors-based objection segmentation framework

This paper introduces a new implicit-kernel-sparse-shape-representation-based object segmentation framework. Given an input object whose shape is similar to some of the elements in the training set, the proposed model can automatically find a cluster of implicit kernel sparse neighbors to approximately represent the input shape and guide the segmentation. A distance-constrained probabilistic definition together with a dualization energy term is developed to connect high-level shape representation and low-level image information. We theoretically prove that our model not only derives from two projected convex sets but is also equivalent to a sparse-reconstruction-error-based representation in the Hilbert space. Finally, a “wake–sleep”-based segmentation framework is applied to drive the evolutionary curve to recover the original shape of the object. We test our model on two public datasets. Numerical experiments on both synthetic images and real applications show the superior capabilities of the proposed framework.

Informative patches sampling for image classification by utilizing bottom-up and top-down information

In image classification based on bag of visual words framework, image patches used for creating image representations affect the classification performance significantly. However, currently, patches are sampled mainly based on processing low-level image information or just extracted regularly or randomly. These methods are not effective, because patches extracted through these approaches are not necessarily discriminative for image categorization. In this paper, we propose to utilize both bottom-up information through processing low-level image information and top-down information through exploring statistical properties of training image grids to extract image patches. In the proposed work, an input image is divided into regular grids, each of which is evaluated based on its bottom-up information and/or top-down information. Subsequently, every grid is assigned a saliency value based on its evaluation result, so that a saliency map can be created for the image. Finally, patch sampling from the input image is performed on the basis of the obtained saliency map. Furthermore, we propose a method to fuse these two kinds of information. The proposed methods are evaluated on both object categories and scene categories. Experiment results demonstrate their effectiveness.

A Genetic Algorithm for Automated Horizon Correlation Across Faults in Seismic Images

Finding corresponding seismic horizons which have been separated by a fault is still performed manually in geological interpretation of seismic images. The difficulties of automating this task are due to the small amount of local information typical for those images, resulting in a high degree of interpretation uncertainty. Our approach is based on a model consisting of geological and geometrical knowledge in order to support the low-level image information. Finding the geologically most probable matches of several horizons across a fault is a combinatorial optimization problem, which cannot be solved exhaustively since the number of combinations increases exponentially with the number of horizons. A genetic algorithm (GA) has been chosen as the most appropriate strategy to solve the optimization problem. Our implementation of a GA is adapted to this particular problem by introducing geological knowledge into the solution process. The results verify the suitability of the method and the appropriateness of the parameters chosen for the horizon correlation problem.