Large-scale Image Sets Research Articles

Distributed Manifold Hashing for Image Set Classification and Retrieval

Conventional image set methods typically learn from image sets stored in one location. However, in real-world applications, image sets are often distributed or collected across different positions. Learning from such distributed image sets presents a challenge that has not been studied thus far. Moreover, efficiency is seldom addressed in large-scale image set applications. To fulfill these gaps, this paper proposes Distributed Manifold Hashing (DMH), which models distributed image sets as a connected graph. DMH employs Riemannian manifold to effectively represent each image set and further suggests learning hash code for each image set to achieve efficient computation and storage. DMH is formally formulated as a distributed learning problem with local consistency constraint on global variables among neighbor nodes, and can be optimized in parallel. Extensive experiments on three benchmark datasets demonstrate that DMH achieves highly competitive accuracies in a distributed setting and provides faster classification and retrieval than state-of-the-arts.

Multiple Riemannian Kernel Hashing for Large-Scale Image Set Classification and Retrieval.

Conventional image set methods typically learn from small to medium-sized image set datasets. However, when applied to large-scale image set applications such as classification and retrieval, they face two primary challenges: 1) effectively modeling complex image sets; and 2) efficiently performing tasks. To address the above issues, we propose a novel Multiple Riemannian Kernel Hashing (MRKH) method that leverages the powerful capabilities of Riemannian manifold and Hashing on effective and efficient image set representation. MRKH considers multiple heterogeneous Riemannian manifolds to represent each image set. It introduces a multiple kernel learning framework designed to effectively combine statistics from multiple manifolds, and constructs kernels by selecting a small set of anchor points, enabling efficient scalability for large-scale applications. In addition, MRKH further exploits inter- and intra-modal semantic structure to enhance discrimination. Instead of employing continuous feature to represent each image set, MRKH suggests learning hash code for each image set, thereby achieving efficient computation and storage. We present an iterative algorithm with theoretical convergence guarantee to optimize MRKH, and the computational complexity is linear with the size of dataset. Extensive experiments on five image set benchmark datasets including three large-scale ones demonstrate the proposed method outperforms state-of-the-arts in accuracy and efficiency particularly in large-scale image set classification and retrieval.

Contribution of 3D reconstruction techniques to the animated restoration of historical scenes

Abstract Amid the escalating demand for heightened realism across diverse sectors such as film, television, gaming, tourism, virtual display, and the preservation of cultural heritage, there has been a notable advancement in reverse modeling technology that facilitates the extraction of three-dimensional models from image sequences. This study focuses on the classification of 3D reconstruction for the animation restoration of historical scenes. It delves into the technical pathways for the 3D reconstruction of such scenes, employing a clustering-based Scale Invariant Feature Transform (SIFT) feature-matching acceleration algorithm for the extraction and matching of image features. Subsequently, it integrates the Bundler method of camera self-calibration with the Patch-based Multi-view Stereo (PMVS) algorithm for dense reconstruction, culminating in the assembly and testing of the reconstructed historical scene. The acceleration time of the SIFT algorithm is higher than that of the standard algorithm when the number of images is 10. The standard algorithm’s acceleration time is the longest when there are more than 10 images. When the number of images is greater than 50, the acceleration ratio of sigma=60 is greater than that of sigma=120. This indicates that the clustering-based SIFT algorithm is suitable for accelerated matching of large-scale image sets. The acceleration effect can be improved by appropriately decreasing the value of sigma. The maximum difference between the five independent vectors in model 1 and model 2 is 3.18. After scaling model 2, the difference is narrowed to[ 0.01,0.03]. The small error of the model in scenes 3~9 and the clear graphic texture indicate that the 3D reconstruction model of the historical scene designed in this paper has high accuracy and provides a model reference for the animation restoration of the historical scene.

Image Manipulation Localization Using Multi-Scale Feature Fusion and Adaptive Edge Supervision

Image manipulation localization is a technique that can efficiently segment the tampered regions from a suspicious image. Existing work usually trains a detection model by fusing the features from diverse data streams, e.g., noise inconsistency, recompression inconsistency, and local inconsistency. They, however, ignore a fact that not all tampered images contain these data streams. As a result, high feature redundancy may cause a large number of false detection for tampered region. To address this problem, this paper designs an end-to-end high-confidence localization network architecture. First, deep convolutional neural networks are utilized to extract multi-scale feature sets from the RGB streams. We then design a semantic refined bi-directional feature integration module to fully fuse multi-scale adjacent features and significantly enhance feature representation. Subsequently, morphological operations are introduced to extract multi-scale edge information, which can efficiently reduce feature redundancy by generating wider high-resolution edges during image reconstructing. Finally, a deep semantic residual decoder is sequentially re-constructed by spreading deep semantic information into each decoding stage. The proposed method can not only improve the manipulation localization accuracy, but also guarantee the model robustness. Extensive experiments demonstrate that our method can obtain an effective performance in locating forged regions over different large-scale image sets, and outperforms most of state-of-the-art methods with higher localization accuracy and stronger robustness.

UAV-based bridge geometric shape measurement using automatic bridge component detection and distributed multi-view reconstruction

UAV cameras combined with image-based reconstruction are promising non-contact tools for bridge shape inspection while computational efficiency is low, particularly on large-scale image sets. This paper describes an efficient image-based reconstruction pipeline for bridge geometry measurements. A full bridge three-dimensional reconstruction task is decomposed into multiple distributed tasks of reconstructing bridge sub-models using sub-image sets. A deep learning-based object detection method combined with Euclidean clustering is proposed to automatically generate cropped sub-image sets. An image mask generation technique based on 3D-to-2D mesh projection is used to keep valid pixels for camera pose estimation. The proposed method was validated in a laboratory test for arch shape measurement and in a field test for suspension cable shape measurement. The results show significant improvement about computational efficiency without accuracy reduction, compared with conventional image reconstruction methods.

Global Color Consistency Correction for Large-Scale Images in 3-D Reconstruction

Global color consistency correction for multiview images in three-dimensional (3-D) reconstruction is an important problem. The color differences between the images will affect the result of dense matching, thereby reducing the geometric accuracy of the mesh model. Moreover, it will also affect the result of texture mapping, causing color differences in the textured model. The color correction method based on global optimization is mainly used to solve this problem. And existing methods usually use sparse matching points as the color correspondences, but the correction results are not accurate enough as a result of the sparsity of the points. Besides, their efficiency of solving large-scale images globally is low. This article proposes a novel color correction method to eliminate the color differences between large-scale multiview images effectively. The core idea of our method is to group images by graph partition algorithm, and then perform intragroup correction and intergroup correction in sequence. First, for each pair of images, we calculate the reliable matching regions around the sparse points as the color correspondences according to the local homography principle. Compared with sparse matching points, our strategy can achieve more accurate color correction results. Next, for large-scale images, we partition them into many groups. For each group of images, the correction parameters are solved to eliminate the color differences of the images included in the group. Finally, we eliminate the color differences between groups by intergroup correction to achieve overall color consistency. Experimental results on typical datasets demonstrate that the proposed method is better than the current representative methods. The proposed method shows better color consistency in the extreme cases, and also exhibits higher computational efficiency on large-scale image sets.

Learning to Adapt to Label-Scarce Image Domain via Angular Distance-Based Feature Alignment

Most recent domain adaptation (DA) methods deal with unsupervised setup, which requires numerous target images for training. However, constructing a large-scale image set of the target domain is occasionally much harder than preparing a smaller number of image and label pairs. To cope with the problem, a great attention is recently paid to supervised domain adaptation (SDA), which takes an extremely small amount of labeled target images for training (e.g., at most three examples per category). In the SDA setup, adapting deep networks towards target domain is very challenging due to the lack of target data, and we tackle this problem as follows. Given labeled images from source and target domains, we first extract deep features and project them to hyper-spherical space via l2-normalization. Afterwards, an additive angular margin loss is embedded so that deep features of both domains are compactly grouped on the basis of shared class prototypes. To further relieve domain discrepancy, a pairwise spherical feature alignment loss is incorporated. All of our loss functions are defined in the hyper-spherical space, and the advantage of each ingredient is analyzed in the literature. Comparative evaluation results demonstrate that the proposed approach is superior to existing SDA methods, achieving 60.7% (1-shot) and 64.4% (3-shot) average accuracies for the DomainNet benchmark dataset using the ResNet-34 backbone. In addition, by applying a semi-supervised learning scheme to a network initialized by our SDA method, we achieve the state-of-the-art performance on semi-supervised domain adaptation (SSDA) as well.

Robust hierarchical structure from motion for large-scale unstructured image sets

Structure from Motion (SfM) is key to mixed computer vision and photogrammetry applications. However, the fast-growing needs for large-scale SfM bring challenges to current SfM solutions. Unlike traditional global and incremental SfM solutions, hierarchical SfM approaches demonstrate promising potential in effectively reconstructing large-scale image sets by dividing the image set into multiple image clusters, reconstructing each cluster separately, and gradually merging partial models into a complete model. However, current hierarchical SfM approaches still suffer from the following problems: accurate image clustering without ancillary information; automatic quality evaluation of each reconstruction unit and unreliable partial reconstruction removal; effective and efficient reconstruction of each image cluster; robust and accurate cluster merging considering the merging order and the handling of images taken with the same camera but divided into different clusters. These unstable factors limit the robustness and accuracy of hierarchical SfM approaches on different unstructured image sets.To systematically improve the performance of hierarchical SfM, we propose a novel robust hierarchical structure from motion (RHSfM) method for large-scale image sets, which does not rely on any additional information, such as Global Positioning System (GPS) and Inertial Navigation System (INS). (1) We develop an automatic image clustering method based on image correlation and present a dynamic adjustment strategy, obtaining reliable image clustering results. (2) We remove the poor reconstructions by introducing multiple quality evaluation standards. (3) We put forward a fast incremental SfM algorithm that optimizes the image adding mode with an image pre-screening strategy and gets rid of the dependence by the proposed dynamic adjustment strategy. (4) We achieve accurate cluster merging by creating an optimal merging list and employing a stepwise global optimization strategy that merges structures first and then cameras. Significantly, the entire process is fully automated with only a few input parameters, and the final result is not sensitive to these parameters.We verify our method on various real image sets that cover different image conditions, different scenes, and different image scales, especially two large-scale image sets with 121,506 and 153,396 images, respectively. The experimental results reveal that our approach outperforms the state-of-the-art SfM systems Colmap, 3DF Samantha, and Metashape in terms of robustness, accuracy, and efficiency. In particular, only our method successfully reconstructed all the seven challenging datasets. For the five datasets that the other systems can also reconstruct, our method obtains the highest accuracy, which is 25 percent better than the best result of the comparable methods on average; for the remaining two datasets, the accuracy of our method is higher than 0.75 pixels. Moreover, the efficiency of our method is about 18, 4.85, and 0.25 times faster than Colmap, 3DF Samantha, and Metashape averagely on the experimental image sets, respectively. After all, our contribution provides a comprehensive and practical solution for large-scale SfM.

Efficient global color correction for large-scale multiple-view images in three-dimensional reconstruction

Consistent global color correction across multiple-view images in three-dimensional (3D) reconstruction is an important and challenging problem. The present work addresses this issue by proposing a novel global color correction method for multi-view images based on a spline curve remapping function. In contrast to existing methods, we obtain a series of optimal functions by minimizing the variance in the color values of all observations of every sparse point generated by the Structure from Motion (SfM) technique. We also find that adding only simple constraints to the spline is required to prevent the loss of image contrast and gradient information. The robustness of the proposed method is ensured by the adoption of strong geometric constraints between multi-view images. Finally, the applicability of the method to large-scale multiple-view images is facilitated by proposing a parallelizable hierarchical image color correction strategy based on a tree structure. The performance of the proposed method is compared with the performances of existing state-of-the-art methods when applied to several challenging datasets. The results indicate that the notable flexibility of the spline curve, along with the proposed optimization process and hierarchical strategy, not only enable the proposed method to perform well with challenging datasets, but also provide high computational efficiency when working with large-scale image sets.

Kill Two Birds With One Stone: Weakly-Supervised Neural Network for Image Annotation and Tag Refinement

The number of social images has exploded by the wide adoption of social networks, and people like to share their comments about them. These comments can be a description of the image, or some objects, attributes, scenes in it, which are normally used as the user-provided tags. However, it is well-known that user-provided tags are incomplete and imprecise to some extent. Directly using them can damage the performance of related applications, such as the image annotation and retrieval. In this paper, we propose to learn an image annotation model and refine the user-provided tags simultaneously in a weakly-supervised manner. The deep neural network is utilized as the image feature learning and backbone annotation model, while visual consistency, semantic dependency, and user-error sparsity are introduced as the constraints at the batch level to alleviate the tag noise. Therefore, our model is highly flexible and stable to handle large-scale image sets. Experimental results on two benchmark datasets indicate that our proposed model achieves the best performance compared to the state-of-the-art methods.

CNN-Based Joint Clustering and Representation Learning with Feature Drift Compensation for Large-Scale Image Data

Given a large unlabeled set of images, how to efficiently and effectively group them into clusters based on extracted visual representations remains a challenging problem. To address this problem, we propose a convolutional neural network (CNN) to jointly solve clustering and representation learning in an iterative manner. In the proposed method, given an input image set, we first randomly pick k samples and extract their features as initial cluster centroids using the proposed CNN with an initial model pre-trained from the ImageNet dataset. Mini-batch k-means is then performed to assign cluster labels to individual input samples for a mini-batch of images randomly sampled from the input image set until all images are processed. Subsequently, the proposed CNN simultaneously updates the parameters of the proposed CNN and the centroids of image clusters iteratively based on stochastic gradient descent. We also proposed a feature drift compensation scheme to mitigate the drift error caused by feature mismatch in representation learning. Experimental results demonstrate the proposed method outperforms start-of-the-art clustering schemes in terms of accuracy and storage complexity on large-scale image sets containing millions of images.

Revisiting Co-Saliency Detection: A Novel Approach Based on Two-Stage Multi-View Spectral Rotation Co-clustering.

With the goal of discovering the common and salient objects from the given image group, co-saliency detection has received tremendous research interest in recent years. However, as most of the existing co-saliency detection methods are performed based on the assumption that all the images in the given image group should contain co-salient objects in only one category, they can hardly be applied in practice, particularly for the large-scale image set obtained from the Internet. To address this problem, this paper revisits the co-saliency detection task and advances its development into a new phase, where the problem setting is generalized to allow the image group to contain objects in arbitrary number of categories and the algorithms need to simultaneously detect multi-class co-salient objects from such complex data. To solve this new challenge, we decompose it into two sub-problems, i.e., how to identify subgroups of relevant images and how to discover relevant co-salient objects from each subgroup, and propose a novel co-saliency detection framework to correspondingly address the two sub-problems via two-stage multi-view spectral rotation co-clustering. Comprehensive experiments on two publically available benchmarks demonstrate the effectiveness of the proposed approach. Notably, it can even outperform the state-of-the-art co-saliency detection methods, which are performed based on the image subgroups carefully separated by the human labor.

A dataset of images and morphological profiles of 30 000 small-molecule treatments using the Cell Painting assay

BackgroundLarge-scale image sets acquired by automated microscopy of perturbed samples enable a detailed comparison of cell states induced by each perturbation, such as a small molecule from a diverse library. Highly multiplexed measurements of cellular morphology can be extracted from each image and subsequently mined for a number of applications.FindingsThis microscopy dataset includes 919 265 five-channel fields of view, representing 30 616 tested compounds, available at “The Cell Image Library” (CIL) repository. It also includes data files containing morphological features derived from each cell in each image, both at the single-cell level and population-averaged (i.e., per-well) level; the image analysis workflows that generated the morphological features are also provided. Quality-control metrics are provided as metadata, indicating fields of view that are out-of-focus or containing highly fluorescent material or debris. Lastly, chemical annotations are supplied for the compound treatments applied.ConclusionsBecause computational algorithms and methods for handling single-cell morphological measurements are not yet routine, the dataset serves as a useful resource for the wider scientific community applying morphological (image-based) profiling. The dataset can be mined for many purposes, including small-molecule library enrichment and chemical mechanism-of-action studies, such as target identification. Integration with genetically perturbed datasets could enable identification of small-molecule mimetics of particular disease- or gene-related phenotypes that could be useful as probes or potential starting points for development of future therapeutics.

MapReduce-based clustering for near-duplicate image identification

In this paper, an effective algorithm is developed for tackling the problem of near-duplicate image identification from large-scale image sets, where the LLC (locality-constrained linear coding) method is seamlessly integrated with the maxIDF cut model to achieve more discriminative representations of images. By incorporating MapReduce framework for image clustering and pairwise merging, the near duplicates of images can be identified effectively from large-scale image sets. An intuitive strategy is also introduced to guide the process for parameter selection. Our experimental results on large-scale image sets have revealed that our algorithm can achieve significant improvement on both the accuracy rates and the computation efficiency as compared with other baseline methods.

Horror Image Recognition Based on Context-Aware Multi-Instance Learning.

Horror content sharing on the Web is a growing phenomenon that can interfere with our daily life and affect the mental health of those involved. As an important form of expression, horror images have their own characteristics that can evoke extreme emotions. In this paper, we present a novel context-aware multi-instance learning (CMIL) algorithm for horror image recognition. The CMIL algorithm identifies horror images and picks out the regions that cause the sensation of horror in these horror images. It obtains contextual cues among adjacent regions in an image using a random walk on a contextual graph. Borrowing the strength of the fuzzy support vector machine (FSVM), we define a heuristic optimization procedure based on the FSVM to search for the optimal classifier for the CMIL. To improve the initialization of the CMIL, we propose a novel visual saliency model based on the tensor analysis. The average saliency value of each segmented region is set as its initial fuzzy membership in the CMIL. The advantage of the tensor-based visual saliency model is that it not only adaptively selects features, but also dynamically determines fusion weights for saliency value combination from different feature subspaces. The effectiveness of the proposed CMIL model is demonstrated by its use in horror image recognition on two large-scale image sets collected from the Internet.

Local coding based matching kernel method for image classification.

This paper mainly focuses on how to effectively and efficiently measure visual similarity for local feature based representation. Among existing methods, metrics based on Bag of Visual Word (BoV) techniques are efficient and conceptually simple, at the expense of effectiveness. By contrast, kernel based metrics are more effective, but at the cost of greater computational complexity and increased storage requirements. We show that a unified visual matching framework can be developed to encompass both BoV and kernel based metrics, in which local kernel plays an important role between feature pairs or between features and their reconstruction. Generally, local kernels are defined using Euclidean distance or its derivatives, based either explicitly or implicitly on an assumption of Gaussian noise. However, local features such as SIFT and HoG often follow a heavy-tailed distribution which tends to undermine the motivation behind Euclidean metrics. Motivated by recent advances in feature coding techniques, a novel efficient local coding based matching kernel (LCMK) method is proposed. This exploits the manifold structures in Hilbert space derived from local kernels. The proposed method combines advantages of both BoV and kernel based metrics, and achieves a linear computational complexity. This enables efficient and scalable visual matching to be performed on large scale image sets. To evaluate the effectiveness of the proposed LCMK method, we conduct extensive experiments with widely used benchmark datasets, including 15-Scenes, Caltech101/256, PASCAL VOC 2007 and 2011 datasets. Experimental results confirm the effectiveness of the relatively efficient LCMK method.