Inter-class Samples Research Articles

Deep Ring-Block-Wise Network for Hyperspectral Image Classification.

Deep learning has achieved many successes in the field of the hyperspectral image (HSI) classification. Most of existing deep learning-based methods have no consideration of feature distribution, which may yield lowly separable and discriminative features. From the perspective of spatial geometry, one excellent feature distribution form requires to satisfy both properties, i.e., block and ring. The block means that in a feature space, the distance of intraclass samples is close and the one of interclass samples is far. The ring represents that all class samples are overall distributed in a ring topology. Accordingly, in this article, we propose a novel deep ring-block-wise network (DRN) for the HSI classification, which takes full consideration of feature distribution. To obtain the good distribution used for high classification performance, in this DRN, a ring-block perception (RBP) layer is built by integrating the self-representation and ring loss into a perception model. By such way, the exported features are imposed to follow the requirements of both block and ring, so as to be more separably and discriminatively distributed compared with traditional deep networks. Besides, we also design an optimization strategy with alternating update to obtain the solution of this RBP layer model. Extensive results on the Salinas, Pavia Centre, Indian Pines, and Houston datasets have demonstrated that the proposed DRN method achieves the better classification performance in contrast to the state-of-the-art approaches.

Relevance Pooling Guidance and Class-Balanced Feature Enhancement for Fine-Grained Oriented Object Detection in Remote Sensing Images

Fine-grained object detection in remote sensing images is highly challenging due to class imbalance and high inter-class indistinguishability. The strategies employed by most existing methods to resolve these two challenges are relatively rudimentary, resulting in suboptimal model performance. To address these issues, we propose a fine-grained object-oriented detection method based on relevance pooling guidance and class balance feature enhancement. Firstly, we propose a global attention mechanism that dynamically retains spatial features pertinent to objects through relevance pooling during down-sampling, thereby enabling the model to acquire more discriminative features. Next, a class balance correction module is proposed to alleviate the class imbalance problem. This module employs feature translation and a learnable reinforcement coefficient to highlight the boundaries of tail class features while maintaining their distinctiveness. Furthermore, we present an enhanced contrastive learning strategy. By dynamically adjusting the contribution of inter-class samples and intra-class similarity measures, this strategy not only constrains inter-class feature distances but also facilitates tighter intra-class clustering, making it more suitable for imbalanced datasets. Evaluation on the FAIR1M and MAR20 datasets demonstrates that our method is superior compared to other methods in object detection and achieves 46.44 and 85.05% mean average precision, respectively.

Supervised spectral feature selection with neighborhood rough set

Spectral feature selection, an excellent dimensionality reduction method, is extensively used in knowledge mining and protein sequence analysis. However, the graph representation derived from data with potential noises significantly impacts feature selection performance. Similarly, its performance deteriorates when label information is ignored, as the selected features have poor discrimination ability. To overcome these drawbacks, this article optimizes neighborhood structure and further constructs the conditional entropy based on neighborhood purity to generate a precise graph representation incorporating label information to enhance the compactness of intra-class samples and increase the dispersion between inter-class samples. Then this article proposes a novel spectral feature selection method. Specifically, the method dynamically learns the precise graph representation from low-dimension space, which can effectively preserve the local structure of data, and the global structure is preserved by introducing a constraint term that maintains the graph representation before and after dimension reduction. Additionally, ℓ2,1-norm is introduced to ensure row sparsity and select the valuable features. The alternative optimization algorithm is employed to solve the optimization problem and its convergence is theoretically analyzed. Finally, to assess the performance of the proposed method, a series of comprehensive experiments are performed on several real-world datasets.

Residual Shrinkage ViT with Discriminative Rebalancing Strategy for Small and Imbalanced Fault Diagnosis

In response to the challenge of small and imbalanced Datasets, where the total Sample size is limited and healthy Samples significantly outweigh faulty ones, we propose a diagnostic framework designed to tackle Class imbalance, denoted as the Dual-Stream Adaptive Deep Residual Shrinkage Vision Transformer with Interclass–Intraclass Rebalancing Loss (DSADRSViT-IIRL). Firstly, to address the issue of limited Sample quantity, we incorporated the Dual-Stream Adaptive Deep Residual Shrinkage Block (DSA-DRSB) into the Vision Transformer (ViT) architecture, creating a DSA-DRSB that adaptively removes redundant signal information based on the input data characteristics. This enhancement enables the model to focus on the Global receptive field while capturing crucial local fault discrimination features from the extremely limited Samples. Furthermore, to tackle the problem of a significant Class imbalance in long-tailed Datasets, we designed an Interclass–Intraclass Rebalancing Loss (IIRL), which decouples the contributions of the Intraclass and Interclass Samples during training, thus promoting the stable convergence of the model. Finally, we conducted experiments on the Laboratory and CWRU bearing Datasets, validating the superiority of the DSADRSViT-IIRL algorithm in handling Class imbalance within mixed-load Datasets.

Feature-level SMOTE: Augmenting fault samples in learnable feature space for imbalanced fault diagnosis of gas turbines

A challenge in gas turbine fault diagnosis is that labeled fault samples are relatively rare and much fewer than normal samples. Conventional data augmentation techniques generate fault samples in original data spaces, resulting in the issue that synthetic fault samples highly overlap with normal samples. Aiming at the issue, a feature-level data augmentation method, namely feature-level SMOTE, is developed by integrating deep Siamese multi-head self-attention network (DSMHSA) with synthetic minority over-sampling technique (SMOTE) to reduce inter-class imbalance and overlap simultaneously. First, the DSMHSA maps original data into a feature space with better inter-class separability, in which inter-class samples stay far away from one another. Second, the SMOTE generates synthetic fault samples in the well-separable space, in order to balance the data set. Finally, the effectiveness of the developed feature-level SMOTE in imbalanced fault diagnosis has been evaluated through two case studies including the real gas turbine fault dataset and the public robot execution failures dataset. To be specific, its average balanced accuracy is 90.38% on the gas turbine dataset, yielding 9.67%, 13.94%, and 12.39% improvements compared to the OUPS, A-SUWO, and NRAS, respectively.

Dual Transformer With Multi-Grained Assembly for Fine-Grained Visual Classification

Fine-grained visual classification requires distinguishing sub-categories within the same super-category, which suffers from small inter-class and large intra-class variances. This paper aims to improve the FGVC task towards better performance, for which we deliver a novel dual Transformer framework (coined Dual-TR) with multi-grained assembly. The Dual-TR is well-designed to encode fine-grained objects by two parallel hierarchies, which is amenable to capturing the subtle yet discriminative cues via the self-attention mechanism in ViT. Specifically, we perform orthogonal multi-grained assembly within the Transformer structure for a more robust representation, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e</i> ., intra-layer and inter-layer assembly. The former aims to explore the informative feature in various self-attention heads within the Transformer layer. The latter pays attention to the token assembly across Transformer layers. Meanwhile, we introduce the constraint of center loss to pull intra-class samples’ compactness and push that of inter-class samples. Extensive experiments show that Dual-TR performs on par with the state-of-the-art methods on four public benchmarks, including CUB-200-2011, NABirds, iNaturalist2017, and Stanford Dogs. The comprehensive ablation studies further demonstrate the effectiveness of architectural design choices.

Contactless Palm Vein Recognition Based on Attention-Gated Residual U-Net and ECA-ResNet

Palm vein recognition has received some considerable attention regarding its use in biometric identification. Palm vein characteristics offer a superior level of security and reliability in personal identification compared to extrinsic methods such as fingerprint, face, and palm print recognition, as vein patterns are difficult to duplicate and do not change throughout one’s lifetime. This study proposes both segmentation and recognition methods to enhance the authentication performance and achieve correct identification using palm vein features. First, we propose a segmentation model based on the U-Net model, enhanced with an attention gate, to effectively segment palm vein patterns. The incorporation of both the attention gate and residual block allows the segmentation model for the learning of the essential features required for specific segmentation tasks. The Hessian-based Jerman filtering method is used for ground-truth labeling. The segmentation model extracts the palm vein patterns and filters out the irrelevant and noisy pixels for the purpose of recognition. The efficient channel attention residual network is trained to learn discriminative features for personal identification using combined margin-based loss functions for palm vein recognition. The channel attention module enhances the useful information and suppresses irrelevant features in the feature maps, which overcomes the problem of rotation, position translation, and scale transformation, as well as improves the recognition rate. The combined loss function used in this study increases the similarity between the intra-class samples and the diversity between inter-class samples. The proposed recognition model achieved 100% accuracy for palm vein recognition and an equal error rate of 0.018 for palm vein verification.

ViCTer: A semi-supervised video character tracker

Video character tracking problem refers to tracking certain characters of interest in the video and returning the appearing time slots for those characters. Solutions to this problem can be applied in various video-analysis-related areas, such as movie analysis and automatic video clipping. However, there are very few researches investigating this problem and there are no existing relevant benchmark datasets available. In this paper, we design a novel model11The code for the project is available on https://github.com/zilinghan/victer. to solve this problem by combining a semi-supervised face recognition network and a multi-human tracker. For the face recognition network, we propose a semi-supervised learning method to fully leverage the unlabeled images in the video, thus reducing the required number of labeled face images. Triplet loss is also used during the training to better distinguish among inter-class samples. However, a single face recognition network is insufficient for video character tracking since people do not always show their frontal faces, or sometimes their faces are blocked by some obstacles. Therefore, a multi-human tracker is integrated into the model to address those problems. Additionally, we collect a dataset for the video character tracking problem, Character Face in Video, which can support various experiments for evaluating video character tracker performance. Experiments show that the proposed semi-supervised face recognition model can achieve more than 98.5% recognition accuracy, and our video character tracker can track in near-real-time and achieve 70% ∼ 80% average intersection-over-union tracking accuracy on the dataset.

Locality preserving triplet discriminative projections for dimensionality reduction

The graph embedding framework is widely used for developing dimensionality-reduction algorithms. Many such supervised algorithms construct an intrinsic graph to compact intra-class samples or describe their local structures and build a penalty graph to increase the separability of inter-class samples. However, in the available intrinsic graph construction, manually selecting a set of appropriate neighbors associated with each sample is challenging. In addition, the construction of a penalty graph may seriously impair the intrinsic structures of the samples. Thus, in an attempt to address these issues, this study proposes an algorithm referred to as locality-preserving triplet discriminative projections. The proposed algorithm comprises locality-preserving and discriminative graphs. First, a weighted least-square function is used to calculate the edge weights of the locality-preserving graph. An improvement of the locality-preserving graph constructed in this study is that suitable neighbors are soft-selected. Meanwhile, the triplets of the samples are exploited for discriminative graph construction. Through the separation of the marginal samples and using less focus on samples with good discriminability, this graph enhances the discriminability with less damage to the intrinsic structure. In addition, to further improve the performance of the discriminative graph, a class-relevant margin is added to separate similar classes. The experimental results on four public datasets show that the proposed algorithm outperforms several other dimensionality reduction methods.

Mix-ViT: Mixing attentive vision transformer for ultra-fine-grained visual categorization

Ultra-fine-grained visual categorization (ultra-FGVC) moves down the taxonomy level to classify sub-granularity categories of fine-grained objects. This inevitably poses a challenge, i.e., classifying highly similar objects with limited samples, which impedes the performance of recent advanced vision transformer methods. To that end, this paper introduces Mix-ViT, a novel mixing attentive vision transformer to address the above challenge towards improved ultra-FGVC. The core design is a self-supervised module that mixes the high-level sample tokens and learns to predict whether a token has been substituted after attentively substituting tokens. This drives the model to understand the contextual discriminative details among inter-class samples. Via incorporating such a self-supervised module, the network gains more knowledge from the intrinsic structure of input data and thus improves generalization capability with limited training sample. The proposed Mix-ViT achieves competitive performance on seven publicly available datasets, demonstrating the potential of vision transformer compared to CNN for the first time in addressing the challenging ultra-FGVC tasks. The code is available at https://github.com/Markin-Wang/MixViT

A self-adaptive DRSN-GPReLU for bearing fault diagnosis under variable working conditions

Recently, deep learning has been widely used for intelligent fault diagnosis of rolling bearings due to its no-mankind feature extraction capability. The majority of intelligent diagnosis methods are based on the assumption that the data collected is from constant working conditions. However, rolling bearings often operate under variable working conditions in the real diagnosis scenario, which reduces the generalization capability of the diagnosis model. To solve this problem, a self-adaptive deep residual shrinkage network with a global parametric rectifier linear unit (DRSN-GPReLU) is proposed in this paper. First, the DRSN is used as the basic architecture to improve the anti-noise ability of the proposed method. Then, a novel activation function—the GPReLU—is developed, which can achieve better intra-class compactness for vibration signals, and the inter-class samples are better mapped into remote areas. Finally, a sub-network based on the attention mechanism is designed to automatically infer the slope of the GPReLU. Various experimental results demonstrate that the DRSN-GPReLU can realize better performance compared with traditional methods under variable working conditions, and has better robustness under noise interference.

Benign-malignant classification of pulmonary nodule with deep feature optimization framework

Convolutional neural network (CNN) has been widely utilized for benign-malignant classification of pulmonary nodules in Computed Tomography images. For traditional CNN models, single-input strategy limits the ability of feature extraction, while multi-input CNN models usually achieve better performance by exploring comprehensive information with pulmonary nodules. However, the concatenation layer in multi-input CNN methods generates high-dimensional deep features, which can easily bring about the curse of dimensionality. To tackle these issues, a manifold-based deep learning model termed deep feature optimization framework (DFOF) is proposed to perk up the performance. In feature extraction stage, a two-stream network is adopted for extracting perinodular and intranodular features from CT images, which forms high-dimensional features. In feature optimization stage, a manifold optimization process is proposed to compact intraclass neighbors while separating interclass samples in low-dimensional embedding space. After that, the optimization features are classified by classifiers, such as nearest neighbor, support vector machine, and random forest. Experiments were conducted using two datasets with 5-cross-validation. The accuracy, area under curve, precision, recall, and F-score reach 92.13%, 95.54%, 94.16%, 87.16%, and 89.93% on the LIDC-IDRI dataset, 90.03%, 94.06%, 96.95%, 89.91%, and 93.38% on the external validation dataset. The results indicate that DFOF has a remarkably better benign-malignant classification performance than several state-of-the-art methods.

A SAR-GMTI Approach Aided by Online Knowledge With an Airborne Multichannel Quad-Pol Radar System

In the complicated geographical environment, there will be a seriously deleterious effect to the performance of synthetic aperture radar (SAR)-ground moving target indication (SAR-GMTI) system, because it is difficult to obtain the homogeneous training samples to accurately estimate the clutter covariance matrix (CCM) without prior information of the observed scene. To this end, this paper proposes a SAR-GMTI approach aided by online knowledge with an airborne multichannel quadrature-polarimetric (quad-pol) radar system. Generally, this paper can be divided into two parts: online knowledge acquisition and polarization knowledge-aided (Pol-KA) SAR-GMTI processing. Firstly, based on the similarity of pixels from the multichannel and multi-polarization information, a weighed estimation method of polarimetric coherency matrix is proposed, which can overcome the over-smoothing problem and increase the estimation accuracy of coherency matrix. Further, a hybrid weighted local K-means based on geodesic distance (GD-HWLKM) clustering algorithm is proposed to achieve the aim of unsupervised classification. Here, geodesic distance (GD) is exploited to measure the distance between multi-feature region covariance matrixes (MFRCMs) and a hybrid weight from different scales (including local cover class distribution, region and pixel) is calculated to automatically update the cluster centroid, which can make full use of the local spatial information by taking the interclass samples' similarity and the diversity of different classes into consideration. Secondly, with the assistance of the previous polarization SAR (PolSAR) image classification result, a Pol-KA SAR-GMTI method is developed. For each ground cover category, an accurate CCM can be estimated with the independent and identically distributed (IID) training samples. Then, the multichannel clutter suppression and preliminary constant false alarm rate (CFAR) detection are performed. Finally, with an airborne multichannel quad-pol radar system, the experimental results on real measured data demonstrate that the proposed method can efficiently improve the clutter suppression preformation and moving-target detection preformation.

Homo–Heterogenous Transformer Learning Framework for RS Scene Classification

Remote sensing (RS) scene classification plays an essential role in the RS community and has attracted increasing attention due to its wide applications. Recently, benefiting from the powerful feature learning capabilities of convolutional neural networks (CNNs), the accuracy of the RS scene classification has significantly been improved. Although the existing CNN-based methods achieve excellent results, there is still room for improvement. First, the CNN-based methods are adept at capturing the global information from RS scenes. Still, the context relationships hidden in RS scenes cannot be thoroughly mined. Second, due to the specific structure, it is easy for normal CNNs to exploit the heterogenous information from RS scenes. Nevertheless, the homogenous information, which is also crucial to comprehensively understand complex contents within RS scenes, does not get the attention it deserves. Third, most CNNs focus on establishing the relationships between RS scenes and semantic labels. However, the similarities between them are not considered deeply, which are helpful to distinguish the intra-/interclass samples. To overcome the limitations mentioned previously, we propose a homo–heterogenous transformer learning (HHTL) framework for the RS scene classification in this article. First, a patch generation module is designed to generate homogenous and heterogenous patches. Then, a dual-branch feature learning module (FLM) is proposed to mine homogenous and heterogenous information within RS scenes simultaneously. In the FLM, based on vision transformer, not only the global information but also the local areas and their context information can be captured. Finally, we design a classification module, which consists of a fusion submodule and a metric-learning module. It can integrate homo–heterogenous information and compact/separate samples from the same/different RS scene categories. Extensive experiments are conducted on four public RS scene datasets. The encouraging results demonstrate that our HHTL framework can outperform many state-of-the-art methods. Our source codes are available at the below website.

Weighted Correlation Embedding Learning for Domain Adaptation.

Domain adaptation leverages rich knowledge from a related source domain so that it can be used to perform tasks in a target domain. For more knowledge to be obtained under relaxed conditions, domain adaptation methods have been widely used in pattern recognition and image classification. However, most of the existing domain adaptation methods only consider how to minimize different distributions of the source and target domains, which neglects what should be transferred for a specific task and suffers negative transfer by distribution outliers. To address these problems, in this paper, we propose a novel domain adaptation method called weighted correlation embedding learning (WCEL) for image classification. In the WCEL approach, we seamlessly integrated correlation learning, graph embedding, and sample reweighting into a unified learning model. Specifically, we extracted the maximum correlated features from the source and target domains for image classification tasks. In addition, two graphs were designed to preserve the discriminant information from interclass samples and neighborhood relations in intraclass samples. Furthermore, to prevent the negative transfer problem, we developed an efficient sample reweighting strategy to predict the target with different confidence levels. To verify the performance of the proposed method in image classification, extensive experiments were conducted with several benchmark databases, verifying the superiority of the WCEL method over other state-of-the-art domain adaptation algorithms.

Building Damage Assessment From Post-Hurricane Imageries Using Unsupervised Domain Adaptation With Enhanced Feature Discrimination

Rapid damage assessment after hurricane disasters is crucial for initiating effective emergency response actions. Unsupervised domain adaptation (UDA), which improves the classification accuracy of the target domain with leveraging abundant labeled data of the source domain, exhibits the potential to solve the issue of lacking labeled data in the task of damage building assessment. However, the application of UDA to building damage detection remains a challenge due to the complexity of post-hurricane imageries. The image characteristics of the source domain differ from those of the target domain, and the similarity of interclass samples is high, thereby degrading the transfer performance of UDA. We propose Duplex Alignment Networks with enhanced feature discrimination, which consists of a pair of generative adversarial networks (GANs) and a dedicated classifier, to address this issue. Our approach can be highlighted in two aspects: 1) a pair of GANs are used to eliminate the discrepancy between the source and the target domains by the alignments at the feature and pixel levels and 2) a dedicated classifier is integrated into the second discriminator to improve the discrimination of the extracted features of the target domain, which can alleviate performance deterioration caused by the high similarity of interclass samples. The proposed approach is validated by two challenging transfer tasks by using the data sets of Hurricanes Sandy, Maria, and Irma. Average accuracy rates of 85.1% (Hurricane Sandy <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\to $ </tex-math></inline-formula> Maria) and 98.3% (Hurricane Sandy <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\to $ </tex-math></inline-formula> Irma) are achieved, thereby leading to superior performance compared with the state-of-the-art methods.

Large Margin Graph Embedding-Based Discriminant Dimensionality Reduction

Discriminant graph embedding-based dimensionality reduction methods have attracted more and more attention over the past few decades. These methods construct an intrinsic graph and penalty graph to preserve the intrinsic geometry structures of intraclass samples and separate the interclass samples. However, the marginal samples cannot be accurately characterized only by penalty graphs since they treat every sample equally. In practice, these marginal samples often influence the classification performance, which needs to be specially tackled. In this study, the near neighbors’ hypothesis margin of marginal samples has been further maximized to separate the interclass samples and improve the discriminant ability by integrating intrinsic graph and penalty graph. A novel discriminant dimensionality reduction named LMGE-DDR has been proposed. Several experiments on public datasets have been conducted to verify the effectiveness of the proposed LMGE-DDR such as ORL, Yale, UMIST, FERET, CMIU-PIE09, and AR. LMGE-DDR performs better than other compared methods, and the corresponding standard deviation of LMGE-DDR is smaller than others. This demonstrates that the evaluation method verifies the effectiveness of the introduced method.

Spectral-Locational-Spatial Manifold Learning for Hyperspectral Images Dimensionality Reduction

Dimensionality reduction (DR) plays an important role in hyperspectral image (HSI) classification. Unsupervised DR (uDR) is more practical due to the difficulty of obtaining class labels and their scarcity for HSIs. However, many existing uDR algorithms lack the comprehensive exploration of spectral-locational-spatial (SLS) information, which is of great significance for uDR in view of the complex intrinsic structure in HSIs. To address this issue, two uDR methods called SLS structure preserving projection (SLSSPP) and SLS reconstruction preserving embedding (SLSRPE) are proposed. Firstly, to facilitate the extraction of SLS information, a weighted spectral-locational (wSL) datum is generated to break the locality of spatial information extraction. Then, a new SLS distance (SLSD) excavating the SLS relationships among samples is designed to select effective SLS neighbors. In SLSSPP, a new uDR model that includes a SLS adjacency graph based on SLSD and a cluster centroid adjacency graph based on wSL data is proposed, which compresses intraclass samples and approximately separates interclass samples in an unsupervised manner. Meanwhile, in SLSRPE, for preserving the SLS relationship among target pixels and their nearest neighbors, a new SLS reconstruction weight was defined to obtain the more discriminative projection. Experimental results on the Indian Pines, Pavia University and Salinas datasets demonstrate that, through KNN and SVM classifiers with different classification conditions, the classification accuracies of SLSSPP and SLSRPE are approximately 4.88%, 4.15%, 2.51%, and 2.30%, 5.31%, 2.41% higher than that of the state-of-the-art DR algorithms.

Metric learning for image-based flower cultivars identification

BackgroundThe study of plant phenotype by deep learning has received increased interest in recent years, which impressive progress has been made in the fields of plant breeding. Deep learning extremely relies on a large amount of training data to extract and recognize target features in the field of plant phenotype classification and recognition tasks. However, for some flower cultivars identification tasks with a huge number of cultivars, it is difficult for traditional deep learning methods to achieve better recognition results with limited sample data. Thus, a method based on metric learning for flower cultivars identification is proposed to solve this problem.ResultsWe added center loss to the classification network to make inter-class samples disperse and intra-class samples compact, the script of ResNet18, ResNet50, and DenseNet121 were used for feature extraction. To evaluate the effectiveness of the proposed method, a public dataset Oxford 102 Flowers dataset and two novel datasets constructed by us are chosen. For the method of joint supervision of center loss and L2-softmax loss, the test accuracy rate is 91.88%, 97.34%, and 99.82% across three datasets, respectively. Feature distribution observed by T-distributed stochastic neighbor embedding (T-SNE) verifies the effectiveness of the method presented above.ConclusionsAn efficient metric learning method has been described for flower cultivars identification task, which not only provides high recognition rates but also makes the feature extracted from the recognition network interpretable. This study demonstrated that the proposed method provides new ideas for the application of a small amount of data in the field of identification, and has important reference significance for the flower cultivars identification research.

Towards self-similarity consistency and feature discrimination for unsupervised domain adaptation

Recent advances in unsupervised domain adaptation mainly focus on learning shared representations by global statistics alignment, such as the Maximum Mean Discrepancy (MMD) which matches the Mean statistics across domains. The lack of class information, however, may lead to partial alignment (or even misalignment) and poor generalization performance. For robust domain alignment, we argue that the similarities across different features in the source domain should be consistent with that in the target domain. Based on this assumption, we propose a new domain discrepancy metric, i.e., Self-similarity Consistency (SSC), to enforce the pairwise relationship between different features being consistent across domains. The Gram matrix matching and Correlation Alignment is proven to be a special case, and a sub-optimal measure of our proposed SSC. Furthermore, we also propose to mitigate the side effect of the partial alignment and misalignment by incorporating the discriminative information of the deep representations. Specifically, a simple yet effective feature norm constraint is exploited to enlarge the discrepancy of inter-class samples. It relieves the requirements of strict alignment when performing adaptation, therefore improving the adaptation performance significantly. Extensive experiments on visual domain adaptation tasks demonstrate the effectiveness of our proposed SSC metric and feature discrimination approach.