Image Representation Learning Research Articles

Learning Representations of Satellite Imagery by Leveraging Point-of-Interests

Satellite imagery depicts the Earth’s surface remotely and provides comprehensive information for many applications, such as land use monitoring and urban planning. Existing studies on unsupervised representation learning for satellite images only take into account the images’ geographic information, ignoring human activity factors. To bridge this gap, we propose using the Point-of-Interest (POI) data to capture human factors and designing a contrastive learning-based framework to consolidate the representation of satellite imagery with POI information. Besides, we introduce a season-invariant representation learning model on satellite imagery, considering that human factors are mostly unchanging with respect to seasons. An attention model is designed at last to merge the representations from the geographic, seasonal, and POI perspectives adaptively. On the basis of real-world datasets collected from Beijing, 1 we evaluate our method for predicting socioeconomic indicators. The results show that the representation containing POI information outperforms the geographic representation in estimating commercial activity-related indicators. Our proposed attentional framework can estimate the socioeconomic indicators with R 2 of 0.874 and outperforms the baseline methods. Furthermore, we explore the differences in the representations of satellite images with varying socioeconomic statuses. Finally, we investigate the impact of geographic and POI perspective information in the representation learning process, as well as the effect of satellite imagery on various spatial resolutions.

Image semantic learning method based on social heterogeneous graph networks1

Image semantic learning techniques are crucial for image understanding and classification. In social networks, image data is widely disseminated thanks to convenient acquisition and intuitive expression characteristics. However, due to the freedom of users to publish information, the image has apparent context dependence and semantic fuzziness, which brings difficulties to image representation learning. Fortunately, social attributes such as hashtags carry rich semantic relations, which can be conducive to understanding the meaning of images. Therefore, this paper proposes a new method named Social Heterogeneous Graph Networks (SHGN) for image semantic learning in social networks. First, a heterogeneous graph is built to expand image semantic relations by social attributes. Then the consistent semantic space is reconstructed through cross-media feature alignment. Finally, an image semantic extended learning network is designed to capture and integrate the social semantics and visual feature, which obtains a rich semantic representation of images from a social context. The experiments demonstrate that SHGN can achieve efficient image representation, and favorably against many baseline algorithms.

Learning image representations for content-based image retrieval of radiotherapy treatment plans

Objective. In this work, we propose a content-based image retrieval (CBIR) method for retrieving dose distributions of previously planned patients based on anatomical similarity. Retrieved dose distributions from this method can be incorporated into automated treatment planning workflows in order to streamline the iterative planning process. As CBIR has not yet been applied to treatment planning, our work seeks to understand which current machine learning models are most viable in this context. Approach. Our proposed CBIR method trains a representation model that produces latent space embeddings of a patient’s anatomical information. The latent space embeddings of new patients are then compared against those of previous patients in a database for image retrieval of dose distributions. All source code for this project is available on github. Main results. The retrieval performance of various CBIR methods is evaluated on a dataset consisting of both publicly available image sets and clinical image sets from our institution. This study compares various encoding methods, ranging from simple autoencoders to more recent Siamese networks like SimSiam, and the best performance was observed for the multitask Siamese network. Significance. Our current results demonstrate that excellent image retrieval performance can be obtained through slight changes to previously developed Siamese networks. We hope to integrate CBIR into automated planning workflow in future works.

Fake Colorized Image Detection Based on Special Image Representation and Transfer Learning

Nowadays, images have become one of the most popular forms of communication as image editing tools have evolved. Image manipulation, particularly image colorization, has become easier, making it harder to differentiate between fake colorized images and actual images. Furthermore, the RGB space is no longer considered to be the best option for color-based detection techniques due to the high correlation between channels and its blending of luminance and chrominance information. This paper proposes a new approach for fake colorized image detection based on a novel image representation created by combining color information from three separate color spaces (HSV, Lab, and Ycbcr) and selecting the most different channels from each color space to reconstruct the image. Features from the proposed image representation are extracted based on transfer learning using the pre-trained CNNs ResNet50 model. The Support Vector Machine (SVM) approach has been used for classification purposes due to its high ability for generalization. Our experiments indicate that our proposed models outperform other state-of-the-art fake colorized image detection methods in several aspects.

Neural correlates of face perception modeled with a convolutional recurrent neural network

Objective. Event-related potential (ERP) sensitivity to faces is predominantly characterized by an N170 peak that has greater amplitude and shorter latency when elicited by human faces than images of other objects. We aimed to develop a computational model of visual ERP generation to study this phenomenon which consisted of a three-dimensional convolutional neural network (CNN) connected to a recurrent neural network (RNN). Approach. The CNN provided image representation learning, complimenting sequence learning of the RNN for modeling visually-evoked potentials. We used open-access data from ERP Compendium of Open Resources and Experiments (40 subjects) to develop the model, generated synthetic images for simulating experiments with a generative adversarial network, then collected additional data (16 subjects) to validate predictions of these simulations. For modeling, visual stimuli presented during ERP experiments were represented as sequences of images (time x pixels). These were provided as inputs to the model. By filtering and pooling over spatial dimensions, the CNN transformed these inputs into sequences of vectors that were passed to the RNN. The ERP waveforms evoked by visual stimuli were provided to the RNN as labels for supervised learning. The whole model was trained end-to-end using data from the open-access dataset to reproduce ERP waveforms evoked by visual events. Main results. Cross-validation model outputs strongly correlated with open-access (r = 0.98) and validation study data (r = 0.78). Open-access and validation study data correlated similarly (r = 0.81). Some aspects of model behavior were consistent with neural recordings while others were not, suggesting promising albeit limited capacity for modeling the neurophysiology of face-sensitive ERP generation. Significance. The approach developed in this work is potentially of significant value for visual neuroscience research, where it may be adapted for multiple contexts to study computational relationships between visual stimuli and evoked neural activity.

Automatic tumor segmentation and metachronous single-organ metastasis prediction of nasopharyngeal carcinoma patients based on multi-sequence magnetic resonance imaging.

Distant metastases is the main failure mode of nasopharyngeal carcinoma. However, early prediction of distant metastases in NPC is extremely challenging. Deep learning has made great progress in recent years. Relying on the rich data features of radiomics and the advantages of deep learning in image representation and intelligent learning, this study intends to explore and construct the metachronous single-organ metastases (MSOM) based on multimodal magnetic resonance imaging. The magnetic resonance imaging data of 186 patients with nasopharyngeal carcinoma before treatment were collected, and the gross tumor volume (GTV) and metastatic lymph nodes (GTVln) prior to treatment were defined on T1WI, T2WI, and CE-T1WI. After image normalization, the deep learning platform Python (version 3.9.12) was used in Ubuntu 20.04.1 LTS to construct automatic tumor detection and the MSOM prediction model. There were 85 of 186 patients who had MSOM (including 32 liver metastases, 25 lung metastases, and 28 bone metastases). The median time to MSOM was 13 months after treatment (7-36 months). The patients were randomly assigned to the training set (N = 140) and validation set (N = 46). By comparison, we found that the overall performance of the automatic tumor detection model based on CE-T1WI was the best (6). The performance of automatic detection for primary tumor (GTV) and lymph node gross tumor volume (GTVln) based on the CE-T1WI model was better than that of models based on T1WI and T2WI (AP@0.5 is 59.6 and 55.6). The prediction model based on CE-T1WI for MSOM prediction achieved the best overall performance, and it obtained the largest AUC value (AUC = 0.733) in the validation set. The precision, recall, precision, and AUC of the prediction model based on CE-T1WI are 0.727, 0.533, 0.730, and 0.733 (95% CI 0.557-0.909), respectively. When clinical data were added to the deep learning prediction model, a better performance of the model could be obtained; the AUC of the integrated model based on T2WI, T1WI, and CE-T1WI were 0.719, 0.738, and 0.775, respectively. By comparing the 3-year survival of high-risk and low-risk patients based on the fusion model, we found that the 3-year DMFS of low and high MSOM risk patients were 95% and 11.4%, respectively (p < 0.001). The intelligent prediction model based on magnetic resonance imaging alone or combined with clinical data achieves excellent performance in automatic tumor detection and MSOM prediction for NPC patients and is worthy of clinical application.

Robustness of Contrastive Learning on Multilingual Font Style Classification Using Various Contrastive Loss Functions

Font is a crucial design aspect, however, classifying fonts is challenging compared with that of other natural objects, as fonts differ from images. This paper presents the application of contrastive learning in font style classification. We conducted various experiments to demonstrate the robustness of contrastive image representation learning. First, we built a multilingual synthetic dataset for Chinese, English, and Korean fonts. Next, we trained the model using various contrastive loss functions, i.e., normalized temperature scaled cross-entropy loss, triplet loss, and supervised contrastive loss. We made some explicit changes to the approach of applying contrastive learning in the domain of font style classification by not applying any image augmentation. We compared the results with those of a fully supervised approach and achieved comparable results using contrastive learning with fewer annotated images and a smaller number of training epochs. In addition, we also evaluated the effect of applying different contrastive loss functions on training.

Implicit and explicit attention mechanisms for zero-shot learning

Zero-Shot Learning (ZSL) aims to recognise unseen object classes which are not observed during the training phase. Most of the existing methods on ZSL focus on learning a compatibility function between the image representation and class semantic information. Few others concentrate on learning image representation by combining local and global features. However, the existing approaches still fail to address the bias issue towards the seen classes. This paper proposes implicit and explicit attention mechanisms to address the existing bias problem in generalised ZSL models. We formulate the implicit attention mechanism with a self-supervised image angle rotation task, which focuses on specific image features aiding in solving the task. On the other hand, the explicit attention mechanism is composed via the consideration of a multi-headed self-attention mechanism in the Vision Transformer model, which learns to attend important image locations and map global image features to semantic space during the training stage. We have conducted comprehensive experiments on three popular benchmarks: AWA2, CUB and SUN, where the effectiveness of our proposed attention mechanisms is shown in both discriminative and generative settings. Our extensive experiments show that our method has achieved state-of-the-art performance obtaining the highest harmonic mean on all three datasets, which is very encouraging to consider the ViT-based attention mechanisms for ZSL tasks in the future.

Medical Image Classifications for 6G IoT-Enabled Smart Health Systems

As day-to-day-generated data become massive in the 6G-enabled Internet of medical things (IoMT), the process of medical diagnosis becomes critical in the healthcare system. This paper presents a framework incorporated into the 6G-enabled IoMT to improve prediction accuracy and provide a real-time medical diagnosis. The proposed framework integrates deep learning and optimization techniques to render accurate and precise results. The medical computed tomography images are preprocessed and fed into an efficient neural network designed for learning image representations and converting each image to a feature vector. The extracted features from each image are then learned using a MobileNetV3 architecture. Furthermore, we enhanced the performance of the arithmetic optimization algorithm (AOA) based on the hunger games search (HGS). In the developed method, named AOAHG, the operators of the HGS are applied to enhance the AOA's exploitation ability while allocating the feasible region. The developed AOAG selects the most relevant features and ensures the overall model classification improvement. To assess the validity of our framework, we conducted evaluation experiments on four datasets, including ISIC-2016 and PH2 for skin cancer detection, white blood cell (WBC) detection, and optical coherence tomography (OCT) classification, using different evaluation metrics. The framework showed remarkable performance compared to currently existing methods in the literature. In addition, the developed AOAHG provided results better than other FS approaches according to the obtained accuracy, precision, recall, and F1-score as performance measures. For example, AOAHG had 87.30%, 96.40%, 88.60%, and 99.69% for the ISIC, PH2, WBC, and OCT datasets, respectively.

Generative Reasoning Integrated Label Noise Robust Deep Image Representation Learning.

The development of deep learning based image representation learning (IRL) methods has attracted great attention for various image understanding problems. Most of these methods require the availability of a set of high quantity and quality of annotated training images, which can be time-consuming, complex and costly to gather. To reduce labeling costs, crowdsourced data, automatic labeling procedures or citizen science projects can be considered. However, such approaches increase the risk of including label noise in training data. It may result in overfitting on noisy labels when discriminative reasoning is employed as in most of the existing methods. This leads to sub-optimal learning procedures, and thus inaccurate characterization of images. To address this issue, in this paper, we introduce a generative reasoning integrated label noise robust deep representation learning (GRID) approach. The proposed GRID approach aims to model the complementary characteristics of discriminative and generative reasoning for IRL under noisy labels. To this end, we first integrate generative reasoning into discriminative reasoning through a supervised variational autoencoder. This allows the proposed GRID approach to automatically detect training samples with noisy labels. Then, through our label noise robust hybrid representation learning strategy, GRID adjusts the whole learning procedure for IRL of these samples through generative reasoning and that of the other samples through discriminative reasoning. Our approach learns discriminative image representations while preventing interference of noisy labels during training independently from the IRL method being selected. Thus, unlike the existing label noise robust methods, GRID does not depend on the type of annotation, label noise, neural network architecture, loss function or learning task, and thus can be directly utilized for various image understanding problems. Experimental results show the effectiveness of the proposed GRID approach compared to the state-of-the-art methods. The code of the proposed approach is publicly available at https://github.com/gencersumbul/GRID.

Pelphix: Surgical Phase Recognition from X-ray Images in Percutaneous Pelvic Fixation.

Surgical phase recognition (SPR) is a crucial element in the digital transformation of the modern operating theater. While SPR based on video sources is well-established, incorporation of interventional X-ray sequences has not yet been explored. This paper presents Pelphix, a first approach to SPR for X-ray-guided percutaneous pelvic fracture fixation, which models the procedure at four levels of granularity - corridor, activity, view, and frame value - simulating the pelvic fracture fixation workflow as a Markov process to provide fully annotated training data. Using added supervision from detection of bony corridors, tools, and anatomy, we learn image representations that are fed into a transformer model to regress surgical phases at the four granularity levels. Our approach demonstrates the feasibility of X-ray-based SPR, achieving an average accuracy of 99.2% on simulated sequences and 71.7% in cadaver across all granularity levels, with up to 84% accuracy for the target corridor in real data. This work constitutes the first step toward SPR for the X-ray domain, establishing an approach to categorizing phases in X-ray-guided surgery, simulating realistic image sequences to enable machine learning model development, and demonstrating that this approach is feasible for the analysis of real procedures. As X-ray-based SPR continues to mature, it will benefit procedures in orthopedic surgery, angiography, and interventional radiology by equipping intelligent surgical systems with situational awareness in the operating room.

HistoPerm: A permutation-based view generation approach for improving histopathologic feature representation learning

Deep learning has been effective for histology image analysis in digital pathology. However, many current deep learning approaches require large, strongly- or weakly labeled images and regions of interest, which can be time-consuming and resource-intensive to obtain. To address this challenge, we present HistoPerm, a view generation method for representation learning using joint embedding architectures that enhances representation learning for histology images. HistoPerm permutes augmented views of patches extracted from whole-slide histology images to improve classification performance. We evaluated the effectiveness of HistoPerm on 2 histology image datasets for Celiac disease and Renal Cell Carcinoma, using 3 widely used joint embedding architecture-based representation learning methods: BYOL, SimCLR, and VICReg. Our results show that HistoPerm consistently improves patch- and slide-level classification performance in terms of accuracy, F1-score, and AUC. Specifically, for patch-level classification accuracy on the Celiac disease dataset, HistoPerm boosts BYOL and VICReg by 8% and SimCLR by 3%. On the Renal Cell Carcinoma dataset, patch-level classification accuracy is increased by 2% for BYOL and VICReg, and by 1% for SimCLR. In addition, on the Celiac disease dataset, models with HistoPerm outperform the fully supervised baseline model by 6%, 5%, and 2% for BYOL, SimCLR, and VICReg, respectively. For the Renal Cell Carcinoma dataset, HistoPerm lowers the classification accuracy gap for the models up to 10% relative to the fully supervised baseline. These findings suggest that HistoPerm can be a valuable tool for improving representation learning of histopathology features when access to labeled data is limited and can lead to whole-slide classification results that are comparable to or superior to fully supervised methods.

Visual Probing: Cognitive Framework for Explaining Self-Supervised Image Representations

Recently introduced self-supervised methods for image representation learning provide on par or superior results to their fully supervised competitors, yet the corresponding efforts to explain the self-supervised approaches lag behind. Motivated by this observation, we introduce a novel visual probing framework for explaining the self-supervised models by leveraging probing tasks employed previously in natural language processing. The probing tasks require knowledge about semantic relationships between image parts. Hence, we propose a systematic approach to obtain analogs of natural language in vision, such as visual words, context, and taxonomy. Our proposal is grounded in Marr's computational theory of vision and concerns features like textures, shapes, and lines. We show the effectiveness and applicability of those analogs in the context of explaining self-supervised representations. Our key findings emphasize that relations between language and vision can serve as an effective yet intuitive tool for discovering how machine learning models work, independently of data modality. Our work opens a plethora of research pathways towards more explainable and transparent AI.

An inception V3 approach for malware classification using machine learning and transfer learning

Malware instances have been extremely used for illegitimate purposes, and new variants of malware are observed every day. Machine learning in network security is one of the prime areas of research today because of its performance and has shown tremendous growth in the last decade. In this paper, we formulate the malware signature as a 2D image representation and leverage deep learning approaches to characterize the signature of malware contained in BIG15 dataset across nine classes. The current research compares the performance of various machine learning and deep learning technologies towards malware classification such as Logistic Regression (LR), Artificial Neural Network (ANN), Convolutional Neural Network (CNN), transfer learning on CNN and Long Short Term Memory (LSTM). The transfer learning approach using InceptionV3 resulted in a good performance with respect to the compared models like LSTM with a classification accuracy of 98.76% on the test dataset and 99.6% on the train dataset.

Accurate prediction of molecular properties and drug targets using a self-supervised image representation learning framework

The clinical efficacy and safety of a drug is determined by its molecular properties and targets in humans. However, proteome-wide evaluation of all compounds in humans, or even animal models, is challenging. In this study, we present an unsupervised pretraining deep learning framework, named ImageMol, pretrained on 10 million unlabelled drug-like, bioactive molecules, to predict molecular targets of candidate compounds. The ImageMol framework is designed to pretrain chemical representations from unlabelled molecular images on the basis of local and global structural characteristics of molecules from pixels. We demonstrate high performance of ImageMol in evaluation of molecular properties (that is, the drug’s metabolism, brain penetration and toxicity) and molecular target profiles (that is, beta-secretase enzyme and kinases) across 51 benchmark datasets. ImageMol shows high accuracy in identifying anti-SARS-CoV-2 molecules across 13 high-throughput experimental datasets from the National Center for Advancing Translational Sciences. Via ImageMol, we identified candidate clinical 3C-like protease inhibitors for potential treatment of COVID-19.

Deep Learning Extraction of the Temperature-Dependent Parameters of Bulk Defects.

Bulk defects in silicon solar cells are a key contributor to loss of efficiency. To detect and identify those defects, temperature- and injection-dependent lifetime spectroscopy is usually used, and the defect parameters are traditionally extracted using fitting methods to the Shockley-Read-Hall recombination statistics. In this study, we propose a deep learning-based extraction technique that is based on an alternative representation of the lifetime curves: lifetime mapping in the temperature and minority carrier concentration space. The deep learning approach successfully predicts all the defect parameters while addressing one of the main limitations of the traditional approach of locating the defect in the energy spectrum, which usually outputs two possible solutions. Furthermore, the approach is applied to temperature-dependent defect parameters where the traditional approach is not applicable, achieving satisfying levels of prediction of the defect parameters. Image representation and deep learning have the potential to bolster solar cell characterization techniques by extracting more insights from the characterization data.

Evaluation of a Self-Supervised Machine Learning Method for Screening of Particulate Samples: A Case Study in Liquid Formulations

Imaging is commonly used as a characterization method in the pharmaceuticals industry, including for quantifying subvisible particles in solid and liquid formulations. Extracting information beyond particle size, such as classifying morphological subpopulations, requires some type of image analysis method. Suggested methods to classify particles have been based on pre-determined morphological features or use supervised training of convolutional neural networks to learn image representations in relation to ground truth labels. Complications arising from highly complex morphologies, unforeseen classes, and time-consuming preparation of ground truth labels, are some of the challenges faced by these methods. In this work, we evaluate the application of a self-supervised contrastive learning method in studying particle images from therapeutic solutions. Unlike with supervised training, this approach does not require ground truth labels and representations are learned by comparing particle images and their augmentations. This method provides a fast and easily implementable tool of coarse screening for morphological attribute assessment. Furthermore, our analysis shows that in cases with relatively balanced datasets, a small subset of an image dataset is sufficient to train a convolutional neural network encoder capable of extracting useful image representations. It is also demonstrated that particle classes typically observed in protein solutions administered by pre-filled syringes emerge as separated clusters in the encoder's embedding space, facilitating performing tasks such as training weakly-supervised classifiers or identifying the presence of new subpopulations.

S2-Net: Self-Supervision Guided Feature Representation Learning for Cross-Modality Images

Dear Editor, This letter focuses on combining the respective advantages of cross-modality images which can compensate for the lack of information in the single modality. Meanwhile, due to the great appearance differences between cross-modality image pairs, it often fails to make the feature representations of correspondences as close as possible. In this letter, we design a cross-modality feature representation learning network, S2-Net, which is based on the recently successful detect-and-describe pipeline, originally proposed for visible images but adapted to work with cross-modality image pairs. Extensive experiments show that our elegant formulation of combined optimization of supervised and self-supervised learning outperforms state-of-the-arts on three cross-modal datasets.

C3-DINO: Joint Contrastive and Non-Contrastive Self-Supervised Learning for Speaker Verification

Self-supervised learning (SSL) has drawn an increased attention in the field of speech processing. Recent studies have demonstrated that contrastive learning is able to learn discriminative speaker embeddings in a self-supervised manner. However, base contrastive self-supervised learning (CSSL) assumes that the pairs generated from a view of anchor instance and any view of other instances are all negative, which introduces many false negative pairs in constructing the loss function. The problem is referred as $class$-$collision$, which remains as one major issue that impedes the CSSL based speaker verification (SV) systems from achieving better performances. In the meanwhile, studies reveal that negative sample free SSL frameworks perform well in learning speaker or image representations. In this study, we investigate SSL techniques that lead to an improved SV performance. We first analyse the impact of false negative pairs in the CSSL systems. Then, a multi-stage Class-Collision Correction (C3) method is proposed, which leads to the state-of-the-art CSSL based speaker embedding system. On the basis of the pretrained CSSL model, we further propose to employ a negative sample free SSL objective (i.e., DINO) to fine-tune the speaker embedding network. The resulting speaker embedding system (C3-DINO) achieves 2.5% EER with a simple Cosine Distance Scoring method on Voxceleb1 test set, which outperforms the previous SOTA SSL system (4.86%) by a significant +45% relative improvement. With speaker clustering and pseudo labeling on Voxceleb2 training set, a LDA/CDS back-end applying on the C3-DINO speaker embeddings is able to further push the EER to 2.2%. Comprehensive experimental investigations of the Voxceleb benchmarks and our internal dataset demonstrate the effectiveness of our proposed methods, and the performance gap between the SSL SV and the supervised counterpart narrows further.

New Benchmark for Household Garbage Image Recognition

Household garbage images are usually faced with complex backgrounds, variable illuminations, diverse angles, and changeable shapes, which bring a great difficulty in garbage image classification. Due to the ability to discover problem-specific features, deep learning and especially convolutional neural networks (CNNs) have been successfully and widely used for image representation learning. However, available and stable household garbage datasets are insufficient, which seriously limits the development of research and application. Besides, the state-of-the-art in the field of garbage image classification is not entirely clear. To solve this problem, in this study, we built a new open benchmark dataset for household garbage image classification by simulating different lightings, backgrounds, angles, and shapes. This dataset is named 30 classes of household garbage images (HGI-30), which contains 18 000 images of 30 household garbage classes. The publicly available HGI-30 dataset allows researchers to develop accurate and robust methods for household garbage recognition. We also conducted experiments and performance analyses of the state-of-the-art deep CNN methods on HGI-30, which serves as baseline results on this benchmark.