Large-scale Training Datasets Research Articles

Multi-Task Driven Feature Models for Thermal Infrared Tracking

Existing deep Thermal InfraRed (TIR) trackers usually use the feature models of RGB trackers for representation. However, these feature models learned on RGB images are neither effective in representing TIR objects nor taking fine-grained TIR information into consideration. To this end, we develop a multi-task framework to learn the TIR-specific discriminative features and fine-grained correlation features for TIR tracking. Specifically, we first use an auxiliary classification network to guide the generation of TIR-specific discriminative features for distinguishing the TIR objects belonging to different classes. Second, we design a fine-grained aware module to capture more subtle information for distinguishing the TIR objects belonging to the same class. These two kinds of features complement each other and recognize TIR objects in the levels of inter-class and intra-class respectively. These two feature models are learned using a multi-task matching framework and are jointly optimized on the TIR tracking task. In addition, we develop a large-scale TIR training dataset to train the network for adapting the model to the TIR domain. Extensive experimental results on three benchmarks show that the proposed algorithm achieves a relative gain of 10% over the baseline and performs favorably against the state-of-the-art methods. Codes and the proposed TIR dataset are available at https://github.com/QiaoLiuHit/MMNet.

FD-GAN: Generative Adversarial Networks with Fusion-Discriminator for Single Image Dehazing

Recently, convolutional neural networks (CNNs) have achieved great improvements in single image dehazing and attained much attention in research. Most existing learning-based dehazing methods are not fully end-to-end, which still follow the traditional dehazing procedure: first estimate the medium transmission and the atmospheric light, then recover the haze-free image based on the atmospheric scattering model. However, in practice, due to lack of priors and constraints, it is hard to precisely estimate these intermediate parameters. Inaccurate estimation further degrades the performance of dehazing, resulting in artifacts, color distortion and insufficient haze removal. To address this, we propose a fully end-to-end Generative Adversarial Networks with Fusion-discriminator (FD-GAN) for image dehazing. With the proposed Fusion-discriminator which takes frequency information as additional priors, our model can generator more natural and realistic dehazed images with less color distortion and fewer artifacts. Moreover, we synthesize a large-scale training dataset including various indoor and outdoor hazy images to boost the performance and we reveal that for learning-based dehazing methods, the performance is strictly influenced by the training data. Experiments have shown that our method reaches state-of-the-art performance on both public synthetic datasets and real-world images with more visually pleasing dehazed results.

Aerial-DEM geolocalization for GPS-denied UAS navigation

Accelerated by the proliferation of small, affordable, and lightweight electronically scanning radar systems as well as advances in Unmanned Aircraft System (UAS) technology, Geo-Registered Radar Returns data are becoming an incredible source for geolocalization in GPS-denied UAS navigation. Most existing approaches match aerial images to pre-stored Digital Elevation Models (DEMs) through 3D terrain reconstruction or GPU-based terrain rendering techniques. However, these reconstruction or rendering processes are themselves error-prone and time-consuming, which further decrease UAS navigation accuracy. In this work, we propose a novel geolocalization approach by directly matching aerial images to DEMs. Inspired by success of deep learning in face recognition/verification, we develop a triplet-ranking network to embed aerial images and DEMs into the same low-dimensional feature space, where matching Aerial-DEM are near one another and mismatched Aerial-DEM are far apart. To create large-scale training dataset, we design an efficient terrain generation approach using per-pixel displacement mapping technique. This approach augments aerial datasets by simulating visual appearances of terrain under different lighting conditions. Experiments are conducted to show the effectiveness of our deep network in finding matches between aerial images and DEMs.

Real-Time Recognition of Facial Expressions Using Facial Electromyograms Recorded Around the Eyes for Social Virtual Reality Applications

Recent developments of social virtual reality (VR) services using avatars have increased the need for facial expression recognition (FER) technology. FER systems are generally implemented using optical cameras; however, the performance of these systems can be limited when users are wearing head-mounted displays (HMDs) as users' faces are largely covered by the HMDs. Facial electromyograms (fEMGs) that can be recorded around users' eyes can be potentially used for implementing FER systems for VR applications. However, this technology lacks practicality owing to the need for large-scale training datasets; furthermore, it is hampered by a relatively low performance. In this study, we proposed an fEMG-based FER system based on the Riemannian manifold-based approach to reduce the number of training datasets needed and enhance FER performance. Our experiments with 42 participants showed an average classification accuracy as high as 85.01% for recognizing 11 facial expressions with only a single training dataset for each expression. We further developed an online FER system that could animate a virtual avatar's expression reflecting a user's facial expression in real time, thus demonstrating that our FER system can be potentially used for practical interactive VR applications, such as social VR networks, smart education, and virtual training.

Challenges in Task Incremental Learning for Assistive Robotics

Recent breakthroughs in computer vision areas, ranging from detection, segmentation, to classification, rely on the availability of large-scale representative training datasets. Yet, robotic vision poses new challenges towards applying visual algorithms developed from these datasets because the latter implicitly assume a fixed set of categories and time-invariant distribution of tasks. In practice, assistive robots should be able to operate in dynamic environments with everyday changes. The variations of four commonly observed factors, including illumination, occlusion, camera-object distance/angles and clutter, could make lifelong/continual learning in computer vision more challenging. Large-scale datasets previously made publicly available were relatively simple, and rarely include such real-world challenges in data collection. Benefited from the recent released OpenLORIS-Object dataset, which explicitly includes these real-world challenges in the lifelong object recognition task, we evaluate three most adopted regularization methods in lifelong/continual learning (Learning without Forgetting, Elastic Weights Consolidation, and Synaptic Intelligence). Their performances were compared with the naive and cumulative training modes as the lower bound and upper bound of performances, respectively. The experiments conducted on the dataset focused on task incremental learning, i.e., incremental difficulty based on the four environment of factors. However, all the three most reported lifelong/continual learning algorithms have failed with the increase in encountered batches across various metrics with indistinguishable performance comparing to the naive training mode. Our results highlight the current challenges in lifelong object recognition for assistive robots to operate in real-world dynamic scene.

Weakly Supervised Deep Learning for Whole Slide Lung Cancer Image Analysis.

Histopathology image analysis serves as the gold standard for cancer diagnosis. Efficient and precise diagnosis is quite critical for the subsequent therapeutic treatment of patients. So far, computer-aided diagnosis has not been widely applied in pathological field yet as currently well-addressed tasks are only the tip of the iceberg. Whole slide image (WSI) classification is a quite challenging problem. First, the scarcity of annotations heavily impedes the pace of developing effective approaches. Pixelwise delineated annotations on WSIs are time consuming and tedious, which poses difficulties in building a large-scale training dataset. In addition, a variety of heterogeneous patterns of tumor existing in high magnification field are actually the major obstacle. Furthermore, a gigapixel scale WSI cannot be directly analyzed due to the immeasurable computational cost. How to design the weakly supervised learning methods to maximize the use of available WSI-level labels that can be readily obtained in clinical practice is quite appealing. To overcome these challenges, we present a weakly supervised approach in this article for fast and effective classification on the whole slide lung cancer images. Our method first takes advantage of a patch-based fully convolutional network (FCN) to retrieve discriminative blocks and provides representative deep features with high efficiency. Then, different context-aware block selection and feature aggregation strategies are explored to generate globally holistic WSI descriptor which is ultimately fed into a random forest (RF) classifier for the image-level prediction. To the best of our knowledge, this is the first study to exploit the potential of image-level labels along with some coarse annotations for weakly supervised learning. A large-scale lung cancer WSI dataset is constructed in this article for evaluation, which validates the effectiveness and feasibility of the proposed method. Extensive experiments demonstrate the superior performance of our method that surpasses the state-of-the-art approaches by a significant margin with an accuracy of 97.3%. In addition, our method also achieves the best performance on the public lung cancer WSIs dataset from The Cancer Genome Atlas (TCGA). We highlight that a small number of coarse annotations can contribute to further accuracy improvement. We believe that weakly supervised learning methods have great potential to assist pathologists in histology image diagnosis in the near future.

SaltSeg: Automatic 3D salt segmentation using a deep convolutional neural network

Salt boundary interpretation is important for the understanding of salt tectonics and velocity model building for seismic migration. Conventional methods consist of computing salt attributes and extracting salt boundaries. We have formulated the problem as 3D image segmentation and evaluated an efficient approach based on deep convolutional neural networks (CNNs) with an encoder-decoder architecture. To train the model, we design a data generator that extracts randomly positioned subvolumes from large-scale 3D training data set followed by data augmentation, then feed a large number of subvolumes into the network while using salt/nonsalt binary labels generated by thresholding the velocity model as ground truth labels. We test the model on validation data sets and compare the blind test predictions with the ground truth. Our results indicate that our method is capable of automatically capturing subtle salt features from the 3D seismic image with less or no need for manual input. We further test the model on a field example to indicate the generalization of this deep CNN method across different data sets.

A Bi-layered Parallel Training Architecture for Large-Scale Convolutional Neural Networks

Benefitting from large-scale training datasets and the complex training network, Convolutional Neural Networks (CNNs) are widely applied in various fields with high accuracy. However, the training process of CNNs is very time-consuming, where large amounts of training samples and iterative operations are required to obtain high-quality weight parameters. In this paper, we focus on the time-consuming training process of large-scale CNNs and propose a Bi-layered Parallel Training (BPT-CNN) architecture in distributed computing environments. BPT-CNN consists of two main components: (a) an outer-layer parallel training for multiple CNN subnetworks on separate data subsets, and (b) an inner-layer parallel training for each subnetwork. In the outer-layer parallelism, we address critical issues of distributed and parallel computing, including data communication, synchronization, and workload balance. A heterogeneous-aware Incremental Data Partitioning and Allocation (IDPA) strategy is proposed, where large-scale training datasets are partitioned and allocated to the computing nodes in batches according to their computing power. To minimize the synchronization waiting during the global weight update process, an Asynchronous Global Weight Update (AGWU) strategy is proposed. In the inner-layer parallelism, we further accelerate the training process for each CNN subnetwork on each computer, where computation steps of convolutional layer and the local weight training are parallelized based on task-parallelism. We introduce task decomposition and scheduling strategies with the objectives of thread-level load balancing and minimum waiting time for critical paths. Extensive experimental results indicate that the proposed BPT-CNN effectively improves the training performance of CNNs while maintaining the accuracy.

Cooperative Hybrid Semi-Supervised Learning for Text Sentiment Classification

A large-scale and high-quality training dataset is an important guarantee to learn an ideal classifier for text sentiment classification. However, manually constructing such a training dataset with sentiment labels is a labor-intensive and time-consuming task. Therefore, based on the idea of effectively utilizing unlabeled samples, a synthetical framework that covers the whole process of semi-supervised learning from seed selection, iterative modification of the training text set, to the co-training strategy of the classifier is proposed in this paper for text sentiment classification. To provide an important basis for selecting the seed texts and modifying the training text set, three kinds of measures—the cluster similarity degree of an unlabeled text, the cluster uncertainty degree of a pseudo-label text to a learner, and the reliability degree of a pseudo-label text to a learner—are defined. With these measures, a seed selection method based on Random Swap clustering, a hybrid modification method of the training text set based on active learning and self-learning, and an alternately co-training strategy of the ensemble classifier of the Maximum Entropy and Support Vector Machine are proposed and combined into our framework. The experimental results on three Chinese datasets (COAE2014, COAE2015, and a Hotel review, respectively) and five English datasets (Books, DVD, Electronics, Kitchen, and MR, respectively) in the real world verify the effectiveness of the proposed framework.

A radial basis probabilistic process neural network model and corresponding classification algorithm

A radial basis probabilistic process neuron (RBPPN) and radial basis probabilistic process neural network (RBPPNN) model are proposed to fuse a priori knowledge for application to time-varying signal pattern classification. RBPPN inputs were multi-channel time-varying signals and a generalized inner product was used to perform spatio-temporal aggregation of input signals in the kernel. Typical signal samples from various pattern subsets in the sample set were used as kernel center functions, which use morphological distribution characteristics and combination relationships to implicitly express prior knowledge for the signal category. The exponential probability function was used as the activation function to achieve kernel transformation and RBPPN probability output. The RBPPNN is composed of process signal input layers, an RBPPN hidden layer, a pattern layer, and a Softmax classifier developed through stacking. Generalized inner product operations were used to conduct probability similarity measurements of distribution characteristics between process signals. The pattern layer selectivity summed inputs from the RBPPN hidden layer to the pattern layer according to the category of the kernel center function. Its outputs were then used as inputs in the Softmax classifier. The proposed RBPPNN information processing mechanism was extended to the time domain, and through learning time-varying signal training samples, achieved extraction, expression, and information association of time-varying signal characteristics, as well as direct classification. It can improve the deficiencies of existing neural networks, such as a complete large-scale training dataset is needed, and the information processing flow is complex. In this paper, the properties of the RBPPNN are analyzed and a specific learning algorithm is presented which synthesizes dynamic time warping, dynamic C-means clustering, and the mean square error algorithm. A series of 12-lead electrocardiogram (ECG) signals were used for classification testing of heart disease diagnosis results. The ECG classification accuracy across ten disease types was 75.52% and sinus arrhythmia was identified with an accuracy of 86.75%, verifying the effectiveness of the model and algorithm.

Blind quality assessment for screen content images by combining local and global features

Recently, several no-reference image quality assessment (NR-IQA) metrics have been developed for the quality evaluation of screen content images (SCIs). While, most of them are opinion-aware methods, which are limited by the subjective opinion scores of training data. Hence, in this paper, we propose a novel opinion-unaware method to predict the quality of SCIs without any prior information. Firstly, an union feature is proposed by considering the local and global visual characteristics of human visual system simultaneously. Specifically, a local structural feature is extracted from the rough and smooth regions of SCIs by leveraging a sparse representation model. As a supplement, a global feature is obtained by combining the luminance statistical feature and local binary pattern (LBP) feature of entire SCIs. Secondly, to get rid of the limitation of subjective opinion scores, a new large-scale training dataset contained 80,000 distorted SCIs is constructed, and the quality labels of those distorted SCIs are derived by an advanced full-reference IQA metric. Thirdly, a regression model between image features and image quality labels is learned from the training dataset by employing a learning-based framework. And then, the quality scores of test SCIs can be predicted by the pre-trained regression model. The experimental results on two largest SCI-oriented databases show that the proposed method is superior to the state-of-the-art NR-IQA metrics.

FgER: Fine-Grained Entity Recognition

Fine-grained Entity Recognition (FgER) is the task of detecting and classifying entity mentions into more than 100 types. The type set can span various domains including biomedical (e.g., disease, gene), sport (e.g., sports event, sports player), religion and mythology (e.g., religion, god) and entertainment (e.g., movies, music). Most of the existing literature for Entity Recognition (ER) focuses on coarse-grained entity recognition (CgER), i.e., recognition of entities belonging to few types such as person, location and organization. In the past two decades, several manually annotated datasets spanning different genre of texts were created to facilitate the development and evaluation of CgER systems (Nadeau and Sekine 2007). The state-of-the-art CgER systems use supervised statistical learning models trained on manually annotated datasets (Ma and Hovy 2016). In contrast, FgER systems are yet to match the performance level of CgER systems. There are two major challenges associated with failure of FgER systems. First, manually annotating a large-scale multi-genre training data for FgER task is expensive, time-consuming and error-prone. Note that, a human-annotator will have to choose a subset of types from a large set of types and types for the same entity might differ in sentences based on the contextual information. Second, supervised statistical learning models when trained on automatically generated noisy training data fits to noise, impacting the model’s performance. The objective of my thesis is to create a FgER system by exploring an off the beaten path which can eliminate the need for manually annotating large-scale multi-genre training dataset. The path includes: (1) automatically generating a large-scale single-genre training dataset, (2) noise-aware learning models that learn better in noisy datasets, and (3) use of knowledge transfer approaches to adapt FgER system to different genres of text.

Adversarial Discriminative Heterogeneous Face Recognition

The gap between sensing patterns of different face modalities remains a challenging problem in heterogeneous face recognition (HFR). This paper proposes an adversarial discriminative feature learning framework to close the sensing gap via adversarial learning on both raw-pixel space and compact feature space. This framework integrates cross-spectral face hallucination and discriminative feature learning into an end-to-end adversarial network. In the pixel space, we make use of generative adversarial networks to perform cross-spectral face hallucination. An elaborate two-path model is introduced to alleviate the lack of paired images, which gives consideration to both global structures and local textures. In the feature space, an adversarial loss and a high-order variance discrepancy loss are employed to measure the global and local discrepancy between two heterogeneous distributions respectively. These two losses enhance domain-invariant feature learning and modality independent noise removing. Experimental results on three NIR-VIS databases show that our proposed approach outperforms state-of-the-art HFR methods, without requiring of complex network or large-scale training dataset.

Selective convolutional neural networks and cascade classifiers for remote sensing image classification

ABSTRACTTraining convolutional neural network (CNN) architecture fully, using pretrained CNNs as feature extractors, and fine-tuning pretrained CNNs on target datasets are three popular strategies used in state-of-the-art methods for remote sensing image classification. The full training strategy requires large-scale training dataset, whereas the fine-tuning strategy requires a pretrained model to resume network learning. In this study, we propose a new strategy based on selective CNNs and cascade classifiers to improve the classification accuracy of remote sensing images relative to single CNN. First, we conduct a comparative study of existing pretrained CNNs in terms of data augmentation and the use of fully connected layers. Second, selective CNNs, which based on class separability criterion, are presented to obtain an optimal combination from multiple pretrained models. Finally, classification accuracy is improved by introducing two-stage cascade linear classifiers, the prediction probability of which in the first stage is used as input for the second stage. Experiments on three public remote sensing datasets demonstrate the effectiveness of the proposed method in comparison with state-of-the-art methods.

Ordinal Constrained Binary Code Learning for Nearest Neighbor Search

Recent years have witnessed extensive attention in binary code learning, a.k.a. hashing, for nearest neighbor search problems. It has been seen that high-dimensional data points can quantize into binary codes to give an efficient similarity approximation via Hamming distance. Among the existing schemes, ranking-based hashing is recent promising that targets at preserving ordinal relations of ranking in the Hamming space to minimize retrieval loss. However, the size of the ranking tuples that show the ordinal relations, is quadratic or cubic to the size of training samples. It is so very expensive to embed such ranking tuples in binary code learning, especially given a large-scale training data set. Besides, it remains difficult to build ranking tuples efficiently for most ranking-preserving hashing, which are deployed over an ordinal graph-based setting. To handle these problems, we propose a novel ranking-preserving hashing method, dubbed Ordinal Constraint Hashing (OCH), which efficiently learns the optimal hashing functions with a graph-based approximation to embed the ordinal relations. The core idea is to reduce the size of ordinal graph with ordinal constraint projection, which preserves the ordinal relations through a small data set (such as clusters or random samples). In particular, to learn such hash functions effectively, we further relax the discrete constraints and design a specific stochastic gradient decent algorithm for optimization. Experimental results on three large-scale visual search benchmark datasets, i.e. LabelMe, Tiny100K and GIST1M, show that the proposed OCH method can achieve superior performance over the state-of-the-arts approaches.

Crowdsourcing Human Annotation on Web Page Structure

Parsing the semantic structure of a web page is a key component of web information extraction. Successful extraction algorithms usually require large-scale training and evaluation datasets, which are difficult to acquire. Recently, crowdsourcing has proven to be an effective method of collecting large-scale training data in domains that do not require much domain knowledge. For more complex domains, researchers have proposed sophisticated quality control mechanisms to replicate tasks in parallel or sequential ways and then aggregate responses from multiple workers. Conventional annotation integration methods often put more trust in the workers with high historical performance; thus, they are called performance-based methods. Recently, Rzeszotarski and Kittur have demonstrated that behavioral features are also highly correlated with annotation quality in several crowdsourcing applications. In this article, we present a new crowdsourcing system, called Wernicke, to provide annotations for web information extraction. Wernicke collects a wide set of behavioral features and, based on these features, predicts annotation quality for a challenging task domain: annotating web page structure. We evaluate the effectiveness of quality control using behavioral features through a case study where 32 workers annotate 200 Q&A web pages from five popular websites. In doing so, we discover several things: (1) Many behavioral features are significant predictors for crowdsourcing quality. (2) The behavioral-feature-based method outperforms performance-based methods in recall prediction, while performing equally with precision prediction. In addition, using behavioral features is less vulnerable to the cold-start problem, and the corresponding prediction model is more generalizable for predicting recall than precision for cross-website quality analysis. (3) One can effectively combine workers’ behavioral information and historical performance information to further reduce prediction errors.

A Novel Local Structure Descriptor for Color Image Retrieval

A novel local structure descriptor (LSD) for color image retrieval is proposed in this paper. Local structures are defined based on a similarity of edge orientation, and LSD is constructed using the underlying colors in local structures with similar edge direction. LSD can effectively combine color, texture and shape as a whole for image retrieval. LSH integrates the advantages of both statistical and structural texture description methods, and it possesses high indexing capability and low dimensionality. In addition, the proposed feature extraction algorithm does not need to train on a large scale training datasets, and it can extract local structure histogram based on LSD. The experimental results on the Corel image databases show that the descriptor has a better image retrieval performance than other descriptors.

Hybrid CNN and Dictionary-Based Models for Scene Recognition and Domain Adaptation

Convolutional neural network (CNN) has achieved state-of-the-art performance in many different visual tasks. Learned from a large-scale training dataset, CNN features are much more discriminative and accurate than the hand-crafted features. Moreover, CNN features are also transferable among different domains. On the other hand, traditional dictionarybased features (such as BoW and SPM) contain much more local discriminative and structural information, which is implicitly embedded in the images. To further improve the performance, in this paper, we propose to combine CNN with dictionarybased models for scene recognition and visual domain adaptation. Specifically, based on the well-tuned CNN models (e.g., AlexNet and VGG Net), two dictionary-based representations are further constructed, namely mid-level local representation (MLR) and convolutional Fisher vector representation (CFV). In MLR, an efficient two-stage clustering method, i.e., weighted spatial and feature space spectral clustering on the parts of a single image followed by clustering all representative parts of all images, is used to generate a class-mixture or a classspecific part dictionary. After that, the part dictionary is used to operate with the multi-scale image inputs for generating midlevel representation. In CFV, a multi-scale and scale-proportional GMM training strategy is utilized to generate Fisher vectors based on the last convolutional layer of CNN. By integrating the complementary information of MLR, CFV and the CNN features of the fully connected layer, the state-of-the-art performance can be achieved on scene recognition and domain adaptation problems. An interested finding is that our proposed hybrid representation (from VGG net trained on ImageNet) is also complementary with GoogLeNet and/or VGG-11 (trained on Place205) greatly.

A-ELM⁎: Adaptive Distributed Extreme Learning Machine with MapReduce

Due to the outstanding advantage, such as generalization performance and fast convergence, Extreme Learning Machine (ELM) and its variants have been widely used for many applications. The distributed ELM with MapReduce could handle large-scale training dataset efficiently, but how to cope with its updated hidden nodes number which aims to get the higher accuracy is still a challenging task. In this paper, we propose a novel Adaptive Distributed Extreme Learning Machine with MapReduce (A-ELM⁎). It could overcome the weakness of ELM⁎ in learning massive training dataset for updating hidden nodes number. Firstly, we found that through partial adjustment of incremental hidden nodes and decremental hidden nodes, matrix multiplication (the most computation-expensive part in A-ELM⁎) can be calculated. Next, A-ELM⁎ based on MapReduce framework is proposed. A-ELM⁎ first calculates the intermediate matrix multiplications of the updated hidden nodes subset, and then update the matrix multiplications by modifying the old matrix multiplications with the intermediate ones. Then, based on the updated matrix multiplications, there could obtain the corresponding new output weight vector with centralized computing. Therefore, it is effective for learning large scale training dataset, in which the hidden nodes update rapidly. Finally, we verify the effectiveness and efficiency of our proposed A-ELM⁎, using synthetic data with extensive experiments, in learning updated hidden nodes.

Learning kernel subspace for face recognition

Kernel subspace methods are effective for face recognition. However, their final solutions need to take advantage of all training examples, such that their runs in real-world applications with large scale training data sets are time-consuming. This paper proposes to apply radial basis function neural network (RBFNN) to learn the feature extraction process of kernel subspace methods, specifically, Kernel Principle Component Analysis (KPCA) and two-phase Kernel Linear Discriminant Analysis (KLDA or KPCA+LDA), in order to improve the running efficiency of testing phase of kernel-based face recognition system. Experimental results based on three small face benchmark data sets including ORL, YALE and UMIST as well as two larger benchmark data sets including AR and FERET, show that the proposed method can reach approximately the recognition accuracy of the original KPCA or the two-phase KLDA by using less vectors in its final solution than those of the two original kernel methods. The proposed method can be applied to real-time or online face recognition systems.