Query Sample Research Articles

Semantic-Driven Interpretable Deep Multi-Modal Hashing for Large-Scale Multimedia Retrieval

Multi-modal hashing focuses on fusing different modalities and exploring the complementarity of heterogeneous multi-modal data for compact hash learning. However, existing multi-modal hashing methods still suffer from several problems, including: 1) Almost all existing methods generate unexplainable hash codes. They roughly assume that the contribution of each hash code bit to the retrieval results is the same, ignoring the discriminative information embedded in hash learning and semantic similarity in hash retrieval. Moreover, the length of hash code is empirically set, which will cause bit redundancy and affect retrieval accuracy. 2) Most existing methods exploit shallow models which fail to fully capture higher-level correlation of multi-modal data. 3) Most existing methods adopt online hashing strategy based on immutable direct projection, which generates query codes for new samples without considering the differences of semantic categories. In this paper, we propose a Semantic-driven Interpretable Deep Multi-modal Hashing (SIDMH) method to generate interpretable hash codes driven by semantic categories within a deep hashing architecture, which can solve all these three problems in an integrated model. The main contributions are: 1) A novel deep multi-modal hashing network is developed to progressively extract hidden representations of heterogeneous modality features and deeply exploit the complementarity of multi-modal data. 2) Learning interpretable hash codes, with discriminant information of different categories distinctively embedded into hash codes and their different impacts on hash retrieval intuitively explained. Besides, the code length depends on the number of categories in the dataset, which can reduce the bit redundancy and improve the retrieval accuracy. 3) The semantic-driven online hashing strategy encodes the significant branches and discards the negligible branches of each query sample according to the semantics contained in it, therefore it could capture different semantics in dynamic queries. Finally, we consider both the nearest neighbor similarity and semantic similarity of hash codes. Experiments on several public multimedia retrieval datasets validate the superiority of the proposed method.

Chemometrics in forensic science

Chemometrics in forensic science

Adversarially learned one-class novelty detection with confidence estimation

Given samples from a particular normal class, image novelty detection is aimed at determining whether a query sample is from the normal class. Images from this particular class are termed inliers or normal images, whereas images not belonging to the class are termed novelties. Most novelty detection approaches use deep neural networks. However, few of them are end-to-end, and state-of-the-art neural networks tend to be overconfident in their predictions. This may result in incorrect predictions and may negatively affect novelty detection. In this paper, we propose a novel model termed adversarially learned one-class novelty detection with confidence estimation for image novelty detection. The proposed model consists of a representation and a detection module, which are adversarially trained to collaboratively learn the inlier distribution. Moreover, the model uses confidence estimation so that the detection module can more effective. The proposed model is end-to-end and does not require additional calculations such as novelty scores after training. We conduct comprehensive experiments on four publicly available datasets that are commonly used for novelty detection, and the model is compared with state-of-the-art methods to demonstrate its performance.

Wise optimisation: deep image embedding by informative pair weighting and ranked list learning

Deep image embedding learns how to map images onto feature vectors. Image retrieval performance is often used to evaluate embedding quality. In this study, the authors proposed a wise deep image embedding optimisation (WDIEO) algorithm based on informative pair weighting and ranked list learning (IPWRLL) for network optimisation of fine-grained image retrieval. First, a hard sample mining method Top-k is proposed to select positive and negative samples. Then, for the selected query sample, a ranking list is obtained by comparing the similarity between samples in the data set and the query sample, and the sample is labelled according to the similarity. Finally, for positive samples, two optimisation rules with different functions are used, while ensuring two key issues of instance weighting and intra-class data distribution. For negative samples, different from the widely adopted methods based on the weight of sample information, the authors’ algorithm's weights are set according to the ranking list, which keeps the inter-class data distribution and the optimisation direction consistent with the loss reduction direction. The WDIEO-IPWRLL model is an end-to-end optimisation that can share parameters in the testing process. Experiments show that their proposed model achieves the state-of-the-art performance on the benchmark data set.

Active learning for segmentation based on Bayesian sample queries

Segmentation of anatomical structures is a fundamental image analysis task for many applications in the medical field. Deep learning methods have been shown to perform well, but for this purpose large numbers of manual annotations are needed in the first place, which necessitate prohibitive levels of resources that are often unavailable. In an active learning framework of selecting informed samples for manual labeling, expert clinician time for manual annotation can be optimally utilized, enabling the establishment of large labeled datasets for machine learning. In this paper, we propose a novel method that combines representativeness with uncertainty in order to estimate ideal samples to be annotated, iteratively from a given dataset. Our novel representativeness metric is based on Bayesian sampling, by using information-maximizing autoencoders. We conduct experiments on a shoulder magnetic resonance imaging (MRI) dataset for the segmentation of four musculoskeletal tissue classes. Quantitative results show that the annotation of representative samples selected by our proposed querying method yields an improved segmentation performance at each active learning iteration, compared to a baseline method that also employs uncertainty and representativeness metrics. For instance, with only 10% of the dataset annotated, our method reaches within 5% of Dice score expected from the upper bound scenario of all the dataset given as annotated (an impractical scenario due to resource constraints), and this gap drops down to a mere 2% when less than a fifth of the dataset samples are annotated. Such active learning approach to selecting samples to annotate enables an optimal use of the expert clinician time, being often the bottleneck in realizing machine learning solutions in medicine.

DLA-MatchNet for Few-Shot Remote Sensing Image Scene Classification

Few-shot scene classification aims to recognize unseen scene concepts from few labeled samples. However, most existing works are generally inclined to learn metalearners or transfer knowledge while ignoring the importance to learn discriminative representations and a proper metric for remote sensing images. To address these challenges, in this article, we propose an end-to-end network for boosting a few-shot remote sensing image scene classification, called discriminative learning of adaptive match network (DLA-MatchNet). Specifically, we first adopt the attention technique to delve into the interchannel and interspatial relationships to automatically discover discriminative regions. Then, the channel attention and spatial attention modules can be incorporated with the feature network by using different feature fusion schemes, achieving “discriminative learning.” Afterward, considering the issues of the large intraclass variances and interclass similarity of remote sensing images, instead of simply computing the distances between the support samples and query samples, we concatenate the support and query discriminative features in depth and utilize a matcher to “adaptively” select the semantically relevant sample pairs to assign similarity scores. Our method leverages an episode-based strategy to train the model. Once trained, our model can predict the category of query image without further fine-tuning. Experimental results on three public remote sensing image data sets demonstrate the effectiveness of our model in the few-shot scene classification task.

Privacy-Preserving Collaborative Analytics on Medical Time Series Data

Medical time series data analytics based on dynamic time warping (DTW) greatly benefits modern medical research. Driven by the distributed nature of medical data, the collaboration of multiple healthcare institutions is usually necessary for a sound medical conclusion. Among others, a typical use case is disease screening for public health, where multiple healthcare institutions wish to collaboratively detect over their joint datasets the patients whose medical records have similar features to the given query samples. However, sharing the medical data faces critical privacy obstacles with the increasingly strict legal regulations on data privacy. In this article, we present the design of a novel system enabling privacy-preserving DTW-based analytics on distributed medical time series datasets. Our system is built from a delicate synergy of techniques from both cryptography and data mining domains, where the key idea is to leverage observations on the advancements in plaintext DTW analytics (e.g., clustering and pruning) to facilitate the scalable computation in the ciphertext domain, through our tailored security design. Extensive experiments over real medical time series datasets demonstrate the promising performance of our system, e.g., our system is able to process a secure DTW query computation over 15K time series sequences in 34 minutes.

Soft Sensing of Nonlinear and Multimode Processes Based on Semi-Supervised Weighted Gaussian Regression

This paper presents a new semi-supervised probabilistic density-based regression approach, called Semi-supervised Weighted Gaussian Regression (SWGR), for the soft sensing of nonlinear and multimode industrial processes given a limited number of labeled data samples. In SWGR, different weights are assigned to each training sample based on their similarities to a query sample. Then a local weighted Gaussian density is built for capturing the joint probability of historical samples around the query sample. The training process of parameters in SWGR incorporates both labeled and unlabeled data samples via a maximum likelihood estimation algorithm. In this way, the soft sensor model is able to approximate the nonlinear mechanics of input and output variables and remedy the insufficiency of labeled samples. At last, the output prediction as well as the uncertainty of prediction can be obtained by the conditional distribution. Two case studies validate that the proposed semi-supervised soft sensing method outperforms some recent methods.

Fast Discrete Collaborative Multi-Modal Hashing for Large-Scale Multimedia Retrieval

Many achievements have been made on learning to hash for uni-modal and cross-modal retrieval. However, it is still an unsolved problem that how to directly and efficiently learn discriminative discrete hash codes for the multimedia retrieval, where both query and database samples are represented with heterogeneous multi-modal features. With this motivation, we propose a Fast Discrete Collaborative Multi-modal Hashing (FDCMH) method in this paper. We first propose an efficient collaborative multi-modal mapping that first transforms heterogeneous multi-modal features into the unified factors to exploit the complementarity of multi-modal features and preserve the semantic correlations in multiple modalities with linear computation and space complexity. Such shared factors also bridge the heterogeneous modality gap and remove the inter-modality redundancy. Further, we develop an asymmetric hashing learning module to simultaneously correlate the learned hash codes with low-level data distribution and high-level semantics. In particular, this design could avoid the challenging symmetric semantic matrix factorization and $O(n^2)$ O ( n 2 ) memory cost ( $n$ n is the number of training samples). It can support both computation and memory efficient discrete hash optimization. Experiments on several public multimedia retrieval datasets demonstrate the superiority of the proposed approach compared with state-of-the-art hashing techniques, in terms of both model learning efficiency and retrieval accuracy.

A new globally adaptive k-nearest neighbor classifier based on local mean optimization

The k-nearest neighbor (KNN) rule is a simple and effective nonparametric classification algorithm in pattern classification. However, it suffers from several problems such as sensitivity to outliers and inaccurate classification decision rule. Thus, a local mean-based k-nearest neighbor classifier (LMKNN) was proposed to address these problems, which assigns the query sample with a class label based on the closest local mean vector among all classes. It is proven that the LMKNN classifier achieves better classification performance and is more robust to outliers than the classical KNN classifier. Nonetheless, the unreliable nearest neighbor selection rule and single local mean vector strategy in LMKNN classifier severely have negative effect on its classification performance. Considering these problems in LMKNN, we propose a globally adaptive k-nearest neighbor classifier based on local mean optimization, which utilizes the globally adaptive nearest neighbor selection strategy and the implementation of local mean optimization to obtain more convincing and reliable local mean vectors. The corresponding experimental results conducted on twenty real-world datasets demonstrated that the proposed classifier achieves better classification performance and is less sensitive to the neighborhood size $$k$$ compared with other improved KNN-based classification methods.

A Just-in-Time Fine-Tuning Framework for Deep Learning of SAE in Adaptive Data-Driven Modeling of Time-Varying Industrial Processes

In modern industrial processes, soft sensors have played increasingly important roles for effective process monitoring, control and optimization. Deep learning has shown excellent ability for hierarchical nonlinear feature representation in soft sensors. However, the existing deep learning based soft sensors are mostly trained offline and applied online without updating mechanism. This may cause their performance degradation in time-varying processes. To deal with this problem, an adaptive updating framework is proposed for deep learning, which is based on just-in-time fine-tuning of stacked autoencoder (JIT-SAE). In JIT-SAE, an offline SAE model is first trained with layer-wise unsupervised pre-training and supervised fine tuning. For online prediction, the network is dynamically fine-tuned upon the query sample. For each query sample, the most relevant labeled samples are selected to form a fine-tuning dataset from the historical labeled database, which is regularly augmented once new labeled samples are available from laboratory analysis. Moreover, each relevant sample is assigned with a weight according to its similarity with the query sample. Then, the deep network is fine-tuned with these relevant labeled samples by designing a weighted loss function. Thus, JIT-SAE is able to track the newest process running state timely and match the data pattern accurately. Case study on an industrial hydrocracking process is provided to demonstrate the effectiveness of the JIT-SAE framework.

A novel dynamic ensemble selection classifier for an imbalanced data set: An application for credit risk assessment

Credit risk assessment is usually regarded as an imbalanced classification task solved by static ensemble classifiers. However, the dynamic ensemble selection (DES) strategy that can select different ensemble classifiers for each query sample is rarely used. Deficiency of the existing DES algorithm in dealing with imbalanced data is the major challenge. In this paper, a novel combined DES model is developed for imbalanced learning problems. To handle the imbalanced data sets, the synthetic minority over-sampling technique is initially used to balance a training set before generating a candidate classifier pool; then, the weighting mechanism of DES-MI (multi-class imbalance) is used to highlight the importance of minority instances when evaluating classifier competences. To further ensure the comprehensive evaluation and right selection of the ensemble classifier, the meta-learning framework of META-DES is used to account for multiple criteria, and the two-step selection strategy of DES-KNN (k-nearest neighbours) is employed to perform a trade-off between the competence and diversity of the classifiers. Our experiments on 15 imbalanced data sets from the KEEL repository show that the proposed model improves the performance of seven known and popular DES algorithms in terms of the area under the curve. Moreover, the type I error rate of the proposed method is lower than that of XGBoost and LightGBM in a real P2P loan data set indicating the efficiency of the proposed method for credit risk assessment.

Contributive Representation-Based Reconstruction for Online 3D Action Recognition

Recent years have seen an increasing trend in developing 3D action recognition methods. However, despite the advances, existing models still suffer from some major drawbacks including the lack of any provision for recognizing action sequences with some missing frames. This significantly hampers the applicability of these methods for online scenarios, where only an initial part of sequences are already provided. In this paper, we introduce a novel sequence-to-sequence representation-based algorithm in which a query sample is characterized using a collaborative frame representation of all the training sequences. This way, an optimal classifier is tailored for the existing frames of each query sample, making the model robust to the effect of missing frames in sequences (e.g. in online scenarios). Moreover, due to the collaborative nature of the representation, it implicitly handles the problem of varying styles during the course of activities. Experimental results on three publicly available databases, UTKinect, TST fall, and UTD-MHAD, respectively, show 95.48%, 90.91%, and 91.67% accuracy when using the beginning 75% portion of query sequences and 84.42%, 60.98%, and 87.27% accuracy for their initial 50%.

EnsPKDE&IncLKDE: a hybrid time series prediction algorithm integrating dynamic ensemble pruning, incremental learning, and kernel density estimation

Ensemble pruning can effectively overcome several shortcomings of the classical ensemble learning paradigm, such as the relatively high time and space complexity. However, each predictor has its own unique ability. One predictor may not perform well on some samples, but it will perform very well on other samples. Blindly underestimating the power of specific predictors is unreasonable. Choosing the best predictor set for each query sample is exactly what dynamic ensemble pruning techniques address. This paper proposes a hybrid Time Series Prediction (TSP) algorithm to implement one-step-ahead prediction task, integrating Dynamic Ensemble Pruning (DEP), Incremental Learning (IL), and Kernel Density Estimation (KDE), abbreviated as the EnsPKDEI 2) Dynamic Ensemble Pruning (DEP), achieved by one subalgorithm called EnsPKDE; 3) Incremental Learning (IL), realized by one subalgorithm termed IncLKDE. Benefited from the advantages of integrating Dynamic Ensemble Pruning scheme, Incremental Learning paradigm and Kernel Density Estimation, in the experimental results, EnsPKDE&IncLKDE demonstrates superior prediction performance to several other state-of-the-art algorithms in fulfilling time series forecasting tasks.

Joint Reflectance Field Estimation and Sparse Representation for Face Image Illumination Preprocessing and Recognition

Illumination preprocessing is an important ingredient for handling lighting variation face recognition challenge. Nonetheless, existing methods are usually designed to be independent of the face recognition methods and the interaction between them is not yet well explored. In this paper, we formulate the face image illumination preprocessing and recognition into a unified sparse representation framework and propose a novel joint reflectance field estimation and sparse representation (JRSR) method for face recognition under extreme lighting conditions. The proposed method separates the identify factor and the interfered illumination of a query sample simultaneously by one nonconvex sparse optimizing model. We also present an efficient approximation algorithm to solve JRSR in this paper. Evaluation on several face databases and the experimental results of face recognition with illumination variation clearly demonstrate the advantages of our proposed JRSR algorithm in illumination preprocessing efficiency and recognition accuracy.

A mutual information-based Variational Autoencoder for robust JIT soft sensing with abnormal observations

Considering industrial process with high-dimensional, intrinsic nonlinearities and possibly abnormal observations, a robust deep learning soft sensor model is developed under the just-in-time learning framework. As an unsupervised deep learning approach, Variational Autoencoder (VAE) has been successfully applied to soft sensing problems owing to its ability to describe the latent representations by probability distributions. In this work, to construct high performance soft sensor model, mutual information (MI) is first introduced for input variable selection. By further incorporating MI as weights on variable of the traditional VAE model, a MI-based output-relevant VAE is developed. For each new sample that arrives, by utilizing Symmetric Kullback-Leibler (SKL) divergence, its relevance with historical samples is determined. Based on the SKL divergence, the input samples that are most relevant to the query sample can be collected. The selected historical input samples and corresponding output samples are employed to build a Gaussian process regression (GPR) local model. Expectation maximation (EM) algorithm is utilized to deal with the nonlinearity and abnormal output observations in GPR local model simultaneously for robustness of the soft sensors. Numerical simulations and a benchmark process are employed to validate the effectiveness of the proposed soft sensor, which demonstrates its superior performance over traditional approaches.

Hyperspectral Leaf Image-Based Cucumber Disease Recognition Using the Extended Collaborative Representation Model.

Collaborative representation (CR)-based classification has been successfully applied to plant disease recognition in cases with sufficient training samples of each disease. However, collecting enough training samples is usually time consuming and labor-intensive. Moreover, influenced by the non-ideal measurement environment, samples may be corrupted by variables introduced by bad illumination and occlusions of adjacent leaves. Consequently, an extended collaborative representation (ECR)-based classification model is presented in this paper. Then, it is applied to cucumber leaf disease recognition, which constructs a pure spectral library consisting of several representative samples for each disease and designs a universal variation spectral library that deals with linear variables superimposed on samples. Thus, each query sample is encoded as a linear combination of atoms from these two spectral libraries and disease identity is determined by the disease of minimal reconstruction residuals. Experiments are conducted on spectral curves extracted from normal leaves and the disease lesions of leaves infected with cucumber anthracnose and brown spot. The diagnostic accuracy is higher than 94.7% and the average online diagnosis time is short, about 1 to 1.3 ms. The results indicate that the ECR-based classification model is feasible in the fast and accurate diagnosis of cucumber leaf diseases.

Output-relevant Variational autoencoder for Just-in-time soft sensor modeling with missing data

Main challenges for developing data-based models lie in the existence of high-dimensional and possibly missing observations that exist in stored data from industry process. Variational autoencoder (VAE) as one of the deep learning methods has been applied for extracting useful information or features from high-dimensional dataset. Considering that existing VAE is unsupervised, an output-relevant VAE is proposed for extracting output-relevant features in this work. By using correlation between process variables, different weight is correspondingly assigned to each input variable. With symmetric Kullback–Leibler (SKL) divergence, the similarity is evaluated between the stored samples and a query sample. According to the values of the SKL divergence, data relevant for modeling are selected. Subsequently, Gaussian process regression (GPR) is utilized to establish a model between the input and the corresponding output at the query sample. In addition, owing to the common existence of missing data in output data set, the parameters and missing data in the GPR are estimated simultaneously. A practical debutanizer industrial process is utilized to illustrate the effectiveness of the proposed method.

Translating Clinical Questions by Physicians Into Searchable Queries: Analytical Survey Study.

BackgroundStaying up to date and answering clinical questions with current best evidence from health research is challenging. Evidence-based clinical texts, databases, and tools can help, but clinicians first need to translate their clinical questions into searchable queries. MacPLUS FS (McMaster Premium LiteratUre Service Federated Search) is an online search engine that allows clinicians to explore multiple resources simultaneously and retrieves one single output that includes the following: (1) evidence from summaries (eg, UpToDate and DynaMed), (2) preappraised research (eg, EvidenceAlerts), and (3) non-preappraised research (eg, PubMed), with and without validated bibliographic search filters. MacPLUS FS can also be used as a laboratory to explore clinical questions and evidence retrieval.ObjectiveOur primary objective was to examine how clinicians formulate their queries on a federated search engine, according to the population, intervention, comparison, and outcome (PICO) framework. Our secondary objective was to assess which resources were accessed by clinicians to answer their questions.MethodsWe performed an analytical survey among 908 clinicians who used MacPLUS FS in the context of a randomized controlled trial on search retrieval. Recording account log-ins and usage, we extracted all 1085 queries performed during a 6-month period and classified each search term according to the PICO framework. We further categorized queries into background (eg, “What is porphyria?”) and foreground questions (eg, “Does treatment A work better than B?”). We then analyzed the type of resources that clinicians accessed.ResultsThere were 695 structured queries, after exclusion of meaningless queries and iterations of similar searches. We classified 56.5% (393/695) of these queries as background questions and 43.5% (302/695) as foreground questions, the majority of which were related to questions about therapy (213/695, 30.6%), followed by diagnosis (48/695, 6.9%), etiology (24/695, 3.5%), and prognosis (17/695, 2.5%). This distribution did not significantly differ between postgraduate residents and medical faculty physicians (P=.51). Queries included a median of 3 search terms (IQR 2-4), most often related to the population and intervention or test, rarely related to the outcome, and never related to the comparator. About half of the resources accessed (314/610, 51.5%) were summaries, 24.4% (149/610) were preappraised research, and 24.1% were (147/610) non-preappraised research.ConclusionsOur results, from a large sample of real-life queries, could guide the development of educational interventions to improve clinicians’ retrieval skills, as well as inform the design of more useful evidence-based resources for clinical practice.Trial RegistrationClinicalTrials.gov NCT02038439; https://www.clinicaltrials.gov/ct2/show/NCT02038439

Double Competitive Constraints-Based Collaborative Representation for Pattern Classification

Representation-based classification (RBC) has attracted much attention in pattern recognition. As a linear representative RBC method, collaborative representation-based classification (CRC) is very promising for classification. Although many extensions of CRC have been developed recently, the discriminative and competitive representations of different classes for favorable classification has not yet been fully explored. To design the discriminative and competitive collaborative representations for enhancing the power of pattern discrimination, we propose a novel double competitive constraints-based collaborative representation for classification (DCCRC). In the proposed DCCRC, one competitive constraint is the l2-norm regularization of residuals between each query sample and the class-specific representations, the other one is the l2-norm regularization of the representations of all the classes excluding any one class. In two competitive constraints, the class discrimination information is employed to generate competitive representations. Moreover, the proposed method integrates both the representation learning and classification into the unified model. We study the effectiveness and robustness of the proposed method by comparing it with the state-of-the-art CRC methods on six face databases and twelve UCI data sets. The experimental results demonstrate the promising classification performance of the proposed method.