Efficient Image Representation Research Articles

SIBS: A sparse encoder utilizing self-inspired bases for efficient image representation

Addressing the limitations of pre-defined dictionaries in image processing, this study introduces Self-Inspired Bases-based Sparse Encoder (SIBS), a novel approach that dynamically generates image-specific atoms by leveraging existing dictionaries and input data. Extensive numerical analysis and simulations demonstrate that SIBS enhances sparse coding performance in three key areas: improved quality resolution via precise image reconstruction, accelerated convergence speed, and competitive compression performance rivaling techniques like JPEG and JP2000. Experiments show that SIBS enhances quality levels and NMSE at the same number of non-zero coefficients and outperforms atoms learned directly from the image at low error levels. SIBS also effectively improves the performance of various learned and structured dictionaries by serving as auxiliary bases, enhancing quality resolution, convergence rate, and dictionary efficiency. Additionally, SIBS has potential as a competitive image compression method. The study’s discussion on SIBS limitations guides future research, including examining its performance with other greedy pursuit algorithms and reevaluating the encoder with other lossless and filter techniques.

Fast Depth Map Coding Algorithm for 3D-HEVC Based on Gradient Boosting Machine

Three-Dimensional High-Efficiency Video Coding (3D-HEVC) has been extensively researched due to its efficient compression and deep image representation, but encoding complexity continues to pose a difficulty. This is mainly attributed to redundancy in the coding unit (CU) recursive partitioning process and rate–distortion (RD) cost calculation, resulting in a complex encoding process. Therefore, enhancing encoding efficiency and reducing redundant computations are key objectives for optimizing 3D-HEVC. This paper introduces a fast-encoding method for 3D-HEVC, comprising an adaptive CU partitioning algorithm and a rapid rate–distortion-optimization (RDO) algorithm. Based on the ALV features extracted from each coding unit, a Gradient Boosting Machine (GBM) model is constructed to obtain the corresponding CU thresholds. These thresholds are compared with the ALV to further decide whether to continue dividing the coding unit. The RDO algorithm is used to optimize the RD cost calculation process, selecting the optimal prediction mode as much as possible. The simulation results show that this method saves 52.49% of complexity while ensuring good video quality.

STF-Trans: A two-stream spatiotemporal fusion transformer for very high resolution satellites images

Spatiotemporal satellite image fusion is regarded as an effective approach to address the limitations of a single optical sensor and generate data with high spatial and temporal resolutions. Thanks to the recent advances of Deep Learning, spatiotemporal fusion has gained a significant performance improvement. However, most models require at least three inputs, including a high resolution one at a prior date, to make the prediction at the desired date. Such a requirement is not always guaranteed in practice, in particular due to the high cost of high resolution image and bad atmospheric conditions. In the last few years, Transformers have achieved tremendous performance on several computer vision tasks compared to convolutional neural networks. Inspired by the new trend, we proposed an end-to-end two-stream SpatioTemporal fusion technique based on encoder–decoder transformer architecture, termed STF-Trans. To deal with a complexity and high-cost of data preparation, the proposed approach aims to fuse a coarse-resolution image with high temporal resolution, and a very high resolution Google Earth image, to produce a very high resolution image at the desired date. The proposed technique employs a two-stream convolutional network to capture both temporal and spatial features from the inputs, and then the encoder–decoder transformer captures long-range dependency of the latter and generates a more efficient image representation from the shallow features. Finally, a shallow network maps the features into the desired fused image. The experimental results show that the proposed method outperforms the current state-of-the-art techniques at both quantitative and qualitative evaluations.

Multiscale and multidirection depth map super resolution with semantic inference

AbstractDepth map super resolution has been paid much attention in 3D applications due to the limitation of depth sensors. Few textures in objects with clear contours along them is the most important characteristic of depth map. An efficient image representation should be directional, multiscale and anisotropic. From this we propose a novel multiscale and multidirection depth map super resolution framework with semantic inference to improve the quality of depth maps. In this framework, a multiscale and multidirection depth map contour fusion scheme captures and assembles intrinsic geometrical structures through a multiview non‐subsampled contourlet transform manner. This scheme not only isolates the discontinuities of contours but retains the smoothness along the contours. The semantic inference is also utilized to segment and label the depth map into objects/backgrounds‐level which are coplanar. Furthermore, a semantic‐aware label refinement strategy is introduced to correct the rarely inaccurate labels of the label map for upscaling the target pixel with pixels in the same object or background. Experimental results on benchmark depth map dataset demonstrate that the proposed multiscale and multidirection depth map super resolution framework with semantic inference has a significant improvement than the state‐of‐the‐art algorithms both visually and quantitatively.

Image semantic learning method based on social heterogeneous graph networks1

Image semantic learning techniques are crucial for image understanding and classification. In social networks, image data is widely disseminated thanks to convenient acquisition and intuitive expression characteristics. However, due to the freedom of users to publish information, the image has apparent context dependence and semantic fuzziness, which brings difficulties to image representation learning. Fortunately, social attributes such as hashtags carry rich semantic relations, which can be conducive to understanding the meaning of images. Therefore, this paper proposes a new method named Social Heterogeneous Graph Networks (SHGN) for image semantic learning in social networks. First, a heterogeneous graph is built to expand image semantic relations by social attributes. Then the consistent semantic space is reconstructed through cross-media feature alignment. Finally, an image semantic extended learning network is designed to capture and integrate the social semantics and visual feature, which obtains a rich semantic representation of images from a social context. The experiments demonstrate that SHGN can achieve efficient image representation, and favorably against many baseline algorithms.

A multi-view-CNN framework for deep representation learning in image classification

Deep representation learning in image classification is an area in computer vision where deep Convolutional Neural Networks (CNNs) have flourished. Nevertheless, developing an efficient image recognition model for real world applications is a challenging task, since image datasets are characterized by instances with a large amount of noise and redundant information. Thus, it is essential to incorporate an intelligent feature extraction and filtering method in order to create robust and efficient image representations. In this work, we propose a Multi-View-CNN framework which drastically boosts the performance of pre-trained CNN models, such as ResNet and VGG in image classification applications. In this approach different type of views of the same initial image are used in order to extract different types of features utilizing pre-trained CNN models. However, in order to reduce the huge dimensional size of the raw CNN’s output features and create a robust image representation, the Principal Component Analysis (PCA) dimension reduction method is applied. Then, all these extracted feature vectors are concatenated building a final composite feature representation of the initial image dataset. Finally, this augmented feature vector is used for training a linear model (Logistic Regression) in order to perform the final classification tasks. The main findings of this work are summarized as follows. First, the proposed Multi-View-CNN framework managed to drastically increase the performance results of pre-trained CNN models. Second, the incorporation of PCA as a final layer into the main CNN topology, instead of using the classical dimension reduction layer components such as Averaging and Max Pooling operations, managed to significantly improve the performance. The whole implementation code of this framework alongside with the datasets used in our experimental simulations was uploaded to our public GitHub repository to the following link: https://github.com/EmmanuelPintelas/A-Multi-View-CNN-Framework-for-Deep-Representation-Learning-in-Image-Classification.

Image Quality Assessment–driven Reinforcement Learning for Mixed Distorted Image Restoration

Due to the diversity of the degradation process that is difficult to model, the recovery of mixed distorted images is still a challenging problem. The deep learning model trained under certain degradation declines significantly in other degradation situations. In this article, we explore ways to use a combination of tools to deal with the mixed distortion. First, we illustrate the limitations of a single deep network in dealing with multiple distortion types and then introduce a hierarchical toolkit with distinguished powerful tools. Second, we investigate how an efficient representation of images combined with a reinforcement learning (RL) paradigm helps to deal with tool noise in continuous restoration. The proposed method can accurately capture the distortion preferences for selecting the optimal recovery tools by RL agent. Finally, to fully utilize random tools for unknown distortion combinations, we adopt the exploration scheme with various quality evaluation methods to achieve more quality improvements. Experimental results demonstrate that the peak signal-to-noise ratio of the proposed method is 3.30 dB higher than other state-of-the-art RL-based methods on the CSIQ single distortion dataset and 0.95 dB higher on the DIV2K mixed distortion dataset.

Learning Deep Binary Descriptors via Bitwise Interaction Mining.

In this paper, we propose a GraphBit method to learn unsupervised deep binary descriptors for efficient image representation. Conventional binary representation learning methods directly quantize each element according to the threshold without considering the quantization ambiguousness. The elements near the boundary dubbed as "ambiguous bits" fail to collect effective information for reliable binarization and are sensitive to noise that causes reversed bits. We argue that there are implicit inner relationships among bits in binary descriptors called bitwise interaction, where the related bits can provide extra instruction as prior knowledge for ambiguousness reduction. Specifically, we design a deep reinforcement learning model to learn the structure of the graph for bitwise interaction mining, and the uncertainty of binary codes is reduced by maximizing the mutual information with input and related bits. Consequently, the ambiguous bits receive additional instruction from the graph for reliable binarization. Moreover, we further present a differentiable search method (GraphBit+) that mines the bitwise interaction in continuous space, so that the heavy search cost caused by the training difficulties in reinforcement learning is significantly reduced. Since the GraphBit and GraphBit+ methods learn fixed bitwise interaction which is suboptimal for various input, the inaccurate instruction from the fixed bitwise interaction cannot effectively decrease the ambiguousness of binary descriptors. To address this, we further propose the unsupervised binary descriptor learning method via dynamic bitwise interaction mining (D-GraphBit), where a graph convolutional network called GraphMiner reasons the optimal bitwise interaction for each input sample. Extensive experimental results on the CIFAR-10, NUS-WIDE, ImageNet-100, Brown and HPatches datasets demonstrate the efficiency and effectiveness of the proposed GraphBit, GraphBit+ and D-GraphBit.

IoT-Cloud Empowered Aerial Scene Classification for Unmanned Aerial Vehicles

Recent trends in communication technologies and unmanned aerial vehicles (UAVs) find its application in several areas such as healthcare, surveillance, transportation, etc. Besides, the integration of Internet of things (IoT) with cloud computing environment offers several benefits for the UAV communication. At the same time, aerial scene classification is one of the major research areas in UAV-enabled MEC systems. In UAV aerial imagery, efficient image representation is crucial for the purpose of scene classification. The existing scene classification techniques generate mid-level image features with limited representation capabilities that often end up in producing average results. Therefore, the current research work introduces a new DL-enabled aerial scene classification model for UAV-enabled MEC systems. The presented model enables the UAVs to capture aerial images which are then transmitted to MEC for further processing. Next, Capsule Network (CapsNet)-based feature extraction technique is applied to derive a set of useful feature vectors from the aerial image. It is important to have an appropriate hyperparameter tuning strategy, since manual parameter tuning of DL model tend to produce several configuration errors. In order to achieve this and to determine the hyperparameters of CapsNet model, Shuffled Shepherd Optimization (SSO) algorithm is implemented. Finally, Backpropagation Neural Network (BPNN) classification model is applied to determine the appropriate class labels of aerial images. The performance of SSO-CapsNet model was validated against two openly-accessible datasets namely, UC Merced (UCM) Land Use dataset and WHU-RS dataset. The proposed SSO-CapsNet model outperformed the existing state-of-the-art methods and achieved maximum accuracy of 0.983, precision of 0.985, recall of 0.982, and F-score of 0.983.

Image preparations of multi-mode quantum image representation and their application on quantum image reproduction

Efficient and general quantum image representation and preparation are the precondition of quantum image processing. The multi-mode quantum image representation is compatible with many important color modes. Studying image preparation and image processing will help to further improve the universality of quantum image on different application fields. In this paper, we improve the multi-mode quantum image representation to reduce the difficulty of entanglement preparation. We give a detailed image preparation process and algorithm based on this proved representation. Then, to reduce the number of Not-gates in preparation circuits, the optimal preparation is proposed. Furthermore, we give the reproduction circuit based on these preparations. The preparation and reproduction experiments are performed on the IBM Quantum lab platform. The circuits and the Bell-state statistic histograms of the measurement collapse are given. The results show that the preparations and reproduction are correct and efficient.

Improving multi-view facial expression recognition through two novel texture-based feature representations

Although several automatic computer systems have been proposed to address facial expression recognition problems, the majority of them still fail to cope with some requirements of many practical application scenarios. In this paper, one of the most influential and common issues raised in practical application scenarios when applying automatic facial expression recognition system, head pose variation, is comprehensively explored and investigated. In order to do this, two novel texture feature representations are proposed for implementing multi-view facial expression recognition systems in practical environments. These representations combine the block-based techniques with Local Ternary Pattern-based features, providing a more informative and efficient feature representation of the facial images. In addition, an in-house multi-view facial expression database has been designed and collected to allow us to conduct a detailed research study of the effect of out-of-plane pose angles on the performance of a multi-view facial expression recognition system. Along with the proposed in-house dataset, the proposed system is tested on two well-known facial expression databases, CK+ and BU-3DFE datasets. The obtained results shows that the proposed system outperforms current state-of-the-art 2D facial expression systems in the presence of pose variations.

A generalized multi-snapshot model for 3D homing and route following

Inspired by the learning walks of the ant Ocymyrmex robustior, the original multi-snapshot model was introduced, which—in contrast to the classical “single snapshot at the goal” model—collects multiple snapshots in the vicinity of the goal location that subsequently can be used for homing, that is, for guiding the return to the goal. In this study, we show that the multi-snapshot model can be generalized to homing in three dimensions. In addition to capturing snapshots at positions shifted in all three dimensions, we suggest to decouple the home direction from the orientation of snapshots and to associate a home vector with each snapshot. We then propose a modification of the multi-snapshot model for three-dimensional route following and evaluate its performance in an accurate reconstruction of a real environment. As an illumination-invariant alternative to grayscale images, we also examine sky-segmented images. We use spherical harmonics as efficient representation of panoramic images enabling low memory usage and fast similarity estimation of rotated images. The results show that our approach can steer an agent reliably along a route, making it also suitable for robotic applications using on-board computers with limited resources.

A Novel Image Description With the Stretched Natural Vector Method: Application to Face Recognition

Nowadays there is growing research interest in designing high performance algorithms for automatic facial recognition systems, and an efficient computational approach is required. Accurate face recognition, however, is difficult due to facial complexity. In this paper, we propose a novel and efficient facial image representation named the Stretched Natural Vector (SNV) method which is defined on the intensity values in a grayscale image matrix, where each entry in an intensity matrix records the level of gray at a single pixel in a m×n array. We prove that the SNV defined in this context can distinguish photo matrices in strict one-to-one fashion. This is to say it is theoretically possible to fully recover a grayscale image matrix from the corresponding complete SNV. Experiments on a number of datasets demonstrate that our truncated SNV method compares favorably both in recognition accuracy and efficiency (measured in wall-clock time) against “Full-Pixel” algorithm, Principal Component Analysis (PCA) method, and even its widely used variants - two dimensional PCA (2DPCA) method and two dimensional Euler PCA (2D-EPCA) method.

Double quantum color images encryption scheme based on DQRCI

In this paper, a double quantum color images representation model (DQRCI) which can store two color digital images simultaneously into a quantum superposition state is investigated. Based on DQRCI, some simple image processing operations are discussed and the corresponding quantum circuits are designed. The circuits indicate that these operations have very low computational complexity. Moreover, to ensure the security of DQRCI quantum state image, an encryption scheme is proposed and the validity verification of image encryption system in quantum field is given for the first time. Numerical simulation and performance comparison demonstrate that the proposed double quantum color images encryption scheme outperforms the previous pertinent work in terms of security and computational complexity. This work will help the researchers to further investigate more efficient quantum image representation model and corresponding encryption algorithm.

Multi-instance multi-label learning for surgical image annotation.

Various techniques have been proposed in the literature for phase and tool recognition from laparoscopic videos. In comparison, research in multilabel annotation of still frames is limited. We describe a framework for multilabel annotation of images extracted from laparoscopic cholecystectomy (LC) videos based on multi-instance multiple-label learning. The image is considered as a bag of features extracted from local regions after coarse segmentation. A method based on variational Bayesian gaussian mixture models (VBGMM) is proposed for bag representation. Three techniques based on different feature extraction and bag representation models are employed for comparison. Four anatomical structures (abdominal wall, gallbladder, fat, and liver bed) and a tool-like object (specimen bag) were annotated in 482 images. Our method achieved the best performance on single label accuracy: 0.87 (highest) and 0.69 (lowest). Moreover, the performance was >20% higher in terms of four multilabel classification error metrics (one-error, ranking-loss, hamming-loss, and coverage). Our approach provides an accurate and efficient image representation for multilabel classification of still images captured in LC.

Facial expression recognition using digital signature feature descriptor

Facial feature extraction is the most crucial part for efficient representation of facial images. For facial expression recognition, efficiency of recognition mainly depends on discriminative nature of features that describe optical changes of facial expressions. Facial expressions are very dynamic in nature. The dynamics of facial muscle changes is required to be captured and encoded accurately. Facial expressions are contractions and expansions of facial muscles which results in high-frequency edges on different part of facial image. A novel feature extraction framework called digital signature based on high-frequency edges in combination with LBP histogram features is developed, and the proposed methodology is used for facial expression recognition (FER). Digital signature descriptor of facial dynamic is obtained by projecting edge pixels vertically and horizontally. Digital signature uniquely and completely describes the facial expressions. Support vector method classifier is used to classify six basic expressions based on one-against-all strategy of classification. The validity of proposed algorithm is tested on standard widely used Cohn–Kanade Facial Expression Database (CKFED), Taiwanese Facial Expression Database (TFED) and Japanese Female Facial Expression databases (JFED). Experimental results show that the efficiency of expression recognition of proposed novel technique is 96.25% on CKFED which is higher than other existing methods.

Combinatorial space of watershed hierarchies for image characterization

We propose a framework for image characterization using hierarchies of segmentations. For this purpose, we structure the space of hierarchies using the Gromov–Hausdorff distance. We propose different ways of combining hierarchies and study their properties thanks to the GH distance. We then expose how to leverage the combinatorial space of hierarchies to derive efficient image representations. This framework opens a path for a controlled exploration and use of the combinatorial space of hierarchies.

Toward Efficient Image Representation: Sparse Concept Discriminant Matrix Factorization

The key ingredients of matrix factorization lie in basic learning and coefficient representation. To enhance the discriminant ability of the learned basis, discriminant graph embedding is usually introduced in the matrix factorization model. However, the existing matrix factorization methods based on graph embedding generally conduct discriminant analysis via a single type of adjacency graph, either similarity-based graphs (e.g., Laplacian eigenmaps graph) or reconstruction-based graphs (e.g., ${L}_{1}$ -graph), while ignoring the cooperation of the different types of adjacency graphs that can better depict the discriminant structure of original data. To address the above issue, we propose a novel Fisher-like criterion, based on graph embedding, to extract sufficient discriminant information via two different types of adjacency graphs. One graph preserves the reconstruction relationships of neighboring samples in the same category, and the other suppresses the similarity relationships of neighboring samples from different categories. Moreover, we also leverage the sparse coding to promote the sparsity of the coefficients. By virtue of the proposed Fisher-like criterion and sparse coding, a new matrix factorization framework called Sparse concept Discriminant Matrix Factorization (SDMF) is proposed for efficient image representation. Furthermore, we extend the Fisher-like criterion to an unsupervised context, thus yielding an unsupervised version of SDMF. Experimental results on seven benchmark datasets demonstrate the effectiveness and efficiency of the proposed SDMFs on both image classification and clustering tasks.

LAND USE CLASSIFICATION BASED ON MULTI-STRUCTURE CONVOLUTION NEURAL NETWORK FEATURES CASCADING

Abstract. Learning efficient image representations is at the core of the classification task of remote sensing imagery. The existing methods for solving image classification task, based on either feature coding approaches extracted from convolution neural networks(CNNs) or training new CNNs, can only generate image features with limited representative ability, which essentially prevents them from achieving better performance. In this paper, we investigate how to transfer features from these successfully pre-trained CNNs for classification. We propose a scenario for generating image features via cascading features extracted from different CNNs. First, pre-trained CNNs, like CaffeNet, VGG-S and VGG-F, are used as feature extractor since their different structures help extract richer information of images. Then the fully-connected layers of the pre-trained CNNs are fine-tuned with UC Merced land use dataset. Finally, the image features generating from cascading the outputs of three networks above, are fed into multi-class Optimal Margin Distribution Machine (mcODM) to obtain the final classification results. Extensive experiments on public land use classification dataset demonstrates that the image features obtained by the proposed scenario can result in remarkable performance and improve the state-of-the-art by a significant margin. The results reveal that the features from pre-trained CNNs generalize well to land use dataset and are more expressive than features from single CNN.

HWNet v2: an efficient word image representation for handwritten documents

We present a framework for learning an efficient holistic representation for handwritten word images. The proposed method uses a deep convolutional neural network with traditional classification loss. The major strengths of our work lie in: (i) the efficient usage of synthetic data to pre-train a deep network, (ii) an adapted version of the ResNet-34 architecture with the region of interest pooling (referred to as HWNet v2) which learns discriminative features for variable sized word images, and (iii) a realistic augmentation of training data with multiple scales and distortions which mimics the natural process of handwriting. We further investigate the process of transfer learning to reduce the domain gap between synthetic and real domain and also analyze the invariances learned at different layers of the network using visualization techniques proposed in the literature. Our representation leads to a state-of-the-art word spotting performance on standard handwritten datasets and historical manuscripts in different languages with minimal representation size. On the challenging iam dataset, our method is first to report an mAP of around 0.90 for word spotting with a representation size of just 32 dimensions. Furthermore, we also present results on printed document datasets in English and Indic scripts which validates the generic nature of the proposed framework for learning word image representation.