Orthogonal Regularization Research Articles

Self-Supervised, Multi-View, Semantics-Aware Anchor Clustering

Data-driven artificial intelligence systems effectively enhance accuracy and robustness by utilizing multi-view learning to aggregate consistent and complementary information from multi-source data. As one of the most important branches of multi-view learning, multi-view anchor clustering greatly reduces the time complexity via learning similarity graphs between anchors and data instead of data-to-data similarities, which has gained widespread attention in data-driven artificial intelligence domains. However, two issues still exist in current methods: (1) They commonly utilize orthogonal regularization to enhance anchor diversity, which may lead to a distorted anchor distribution, e.g., some clusters might have few or even no corresponding anchors. (2) They only utilize view-sharing anchors to aggregate complementary and consistent information between views, which may fail to ensure anchor robustness due to the heterogeneity gap between views. To this end, self-supervised, multi-view, semantics-aware anchor clustering (SMA2C) is proposed, containing multi-view representation alignment, adaptive anchor selection, and global spectral optimization. Specifically, SMA2C devises dual-level contrastive learning on representations and clustering partitioning between views within a deep encoding–decoding architecture to achieve multi-granularity alignment of views with the heterogeneity gap. Meanwhile, SMA2C introduces adaptive anchor selection via filtering outliers and refining clusters to enhance correlations between clusters and anchors, which ensures the diversity and discriminability of anchors. Finally, extensive evaluations across four real-world datasets confirm that SMA2C establishes a new benchmark for multi-view anchor clustering. In particular, SMA2C achieves a 6.75% improvement in accuracy over the second-best result on the HW dataset.

Enhancing Major Depressive Disorder Diagnosis With Dynamic-Static Fusion Graph Neural Networks.

Major Depressive Disorder (MDD) is a debilitating, complex mental condition with unclear mechanisms hindering diagnostic progress. Research links MDD to abnormal brain connectivity using functional magnetic resonance imaging (fMRI). Yet, existing fMRI-based MDD models suffer from limitations, including neglecting dynamic network traits, lacking interpretability, and struggling with small datasets. We present DSFGNN, a novel graph neural network framework addressing these issues for improved MDD diagnosis. DSFGNN employs a graph isomorphism encoder to model static and dynamic brain networks, achieving effective fusion of temporal and spatial information through a spatiotemporal attention mechanism, thereby enhancing interpretability. Furthermore, we incorporate a causal disentangling module and orthogonal regularization module to augment the model's expressiveness. We evaluate DSFGNN on the Rest-meta-MDD dataset, yielding superior results compared to the best baseline. Besides, extensive ablation studies and interpretability analysis confirm DSFGNN's effectiveness and potential for biomarker discovery.

DMTFS-FO: Dynamic multi-task feature selection based on flexible loss and orthogonal constraint

Multi-task feature selection (MTFS) has been proven effective for reducing the curse of dimensionality in large-scale classification. Many existing MTFS methods assume that all tasks learn concurrently in a static environment without considering the dynamism of tasks in real-world scenarios. However, new tasks emerge dynamically in practical applications, meaning that the aforementioned static assumption is insufficient. In this paper, we construct a dynamic multi-task feature selection framework to achieve feature reduction for constantly arriving new tasks. First, we modify the traditional mapping by shifting from hard labels to soft labels. Unlike the conventional rigid mapping, the new flexible loss function changes the direct mapping strategy to an indirect one. Second, we use the orthogonal regularization term to constrain the independent relationship between new and old tasks. This ensures that the selected relevant features for new tasks differ from prior tasks. Finally, we integrate the flexible loss and the orthogonal regularization term in the dynamic multi-task feature selection framework. Our method outperforms nine other advanced feature selection methods in terms of effectiveness and efficiency across six datasets. For example, the ACC value of our method is almost 1% higher than the next-best method on the large-scale SUN dataset.

Incorporating syntax and semantics with dual graph neural networks for aspect-level sentiment analysis

Aspect-level sentiment analysis is a more fine-grained task that aims to determine the sentiment polarity of specific aspects. Recent studies have employed graph attention networks and graph convolutional networks to model dependency trees, effectively establishing explicit associations between aspects and opinions, yielding promising performance. However, these methods have limitations in capturing complex linguistic features and the intricate dependencies between aspects and their contexts, resulting in suboptimal performance. In this paper, we propose a dual graph neural network that incorporates syntax and semantics, called IDGNN. Specifically, we utilize the relational graph attention network (RGAT) to encode the syntactic dependency tree and obtain syntactic information, while incorporating dependency labels to enhance aspect representation. Additionally, the semantic graph convolutional network (SemGCN) is employed to encode the self-attention matrix and capture semantic information, with the inclusion of orthogonal regularization to enhance semantic association. Furthermore, we introduce two fusion strategies based on gate mechanisms: the syntax fusion module (SYF) and the semantic fusion module (SEF). SYF combines contextual and syntactic representations to obtain global syntactic features, while SEF fuses semantic information with global syntactic features to obtain the final feature representation. Experimental results demonstrate that our proposed model achieves state-of-the-art performance on several benchmark datasets.

A Large-Class Few-Shot Learning Method Based on High-Dimensional Features

Large-class few-shot learning has a wide range of applications in many fields, such as the medical, power, security, and remote sensing fields. At present, many few-shot learning methods for fewer-class scenarios have been proposed, but little research has been performed for large-class scenarios. In this paper, we propose a large-class few-shot learning method called HF-FSL, which is based on high-dimensional features. Recent theoretical research shows that if the distribution of samples in a high-dimensional feature space meets the conditions of compactness within the class and the dispersion between classes, the large-class few-shot learning method has a better generalization ability. Inspired by this theory, the basic idea is use a deep neural network to extract high-dimensional features and unitize them to project the samples onto a hypersphere. The global orthogonal regularization strategy can then be used to make samples of different classes on the hypersphere that are as orthogonal as possible, so as to achieve the goal of sample compactness within the class and the dispersion between classes in high-dimensional feature space. Experiments on Omniglot, Fungi, and ImageNet demonstrate that the proposed method can effectively improve the recognition accuracy in a large-class FSL problem.

Improving real-time driver distraction detection via constrained attention mechanism

Real-time driving distraction detection has garnered significant attention due to its potential to build various driving safety protections such as distraction warnings and driver assistance systems. Recent studies have focused on the development of neural networks for vision-based detection, though achieving a balance between performance and efficiency has proven challenging. In this paper, we propose a novel constrained attention (CA) mechanism for real-time driver distraction detection, which aims to achieve better performance meanwhile ensuring the computation efficiency. Specifically, we conduct some case studies by generating class activation maps to check the model attention, and three potential factors affecting performance are mined, which are ambiguous attention signal, excessive attention region, and similar attention between different classes. Two regularization terms are designed to optimize the three obstacles. Firstly, a concentrative regularization is introduced to limit the size of the attention region, meanwhile, pixels in the region have clear attention degrees. Secondly, an orthogonal regularization is proposed to optimize the attention of different classes to be discriminative. To further inspiring the model, we design an intersample constraint, which optimizes the attention of images with the same ground truth to be similar. Experiments are conducted on two driver distraction detection datasets, and the experimental results showed that our CA mechanism can bring significant performance improvement. More importantly, there will be no additional computational burden when the trained model is deployed in actual scenarios. Codes are released at https://github.com/gaohangcodes/CAN4DDD.

Improving the adversarial robustness of quantized neural networks via exploiting the feature diversity

Quantized neural networks (QNNs) have become one of the most prevalent approaches in deep learning model compression due to their computational and storage efficiency. However, there is a lack of research specialized in the adversarial robustness of QNNs, which is important for applications in security-critical domains. Existing defenses focus on conventional full-precision networks, which can result in behavioral disparities and degrade the expected performance when directly transferred to QNNs. A novel defensive strategy promotes feature diversity through an orthogonal constraint, which can synergize well with quantization. Inspired by this intuition, we propose an orthogonal regularization with quantization to improve the adversarial robustness of QNNs in this paper. Moreover, we observe that quantization serves as an implicit regularization and is able to alleviate orthogonal degeneration. The proposed orthogonal regularization with quantization is validated on several typical network architectures and benchmark datasets. The results demonstrate that the proposed method can notably enhance adversarial robustness against both white-box and black-box attacks.

Hybrid Regularizations for Multi-Aspect Category Sentiment Analysis

Aspect level sentiment classification aims to identify the sentiment polarity towards a particular aspect in a sentence. Previous attention-based methods generate an aspect-specific representation for each aspect and employ it to classify the sentiment polarity. However, normalized attention scores scatter over every word in the sentence, resulting in two issues. First, the attention may inherently introduce noise and downgrade the performance. Second, the opinion words may be “diluted” by other words, while the opinion feature should dominate for sentiment analysis. The issues become more severe in multi-aspect sentences. In this paper, we address the above two issues via hybrid regularizations, i.e., <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">aspect-level</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">task-level regularizations</i> . Concretely, the aspect-level regularizations constrain the attention weights to alleviate noise. Among them, orthogonal regularization is designed for multi-aspect sentences and sparse regularization is for single-aspect sentences. To extract sentiment-dominant features, task-level regularization is proposed by introducing an orthogonal auxiliary task, i.e., aspect category detection. This regularization can allocate task-oriented context information for specific downstream tasks. Extensive experimental results on three public datasets demonstrate the effectiveness of the proposed approach in both single-task and multi-task scenarios.

Fuzzy clustering method with approximate orthogonal regularization

As a widely used unsupervised learning method, clustering model plays an indispensable role in exploring data structures. Spectral analysis is a useful technique for clustering problems. The spectral analysis could yield good results when used in conjunction with fuzzy clustering models. However, the existing methods are limited by factors such as the definition of the similarity graph and predetermined number of clusters. In this paper, we propose a new fuzzy clustering model with the approximate orthogonal regularization. To obtain the relatively stable results, we reconstruct the Laplacian matrix with geometrically-nearest-neighbor similarity measurement. This new Laplacian matrix can represent the properties of datasets. For this constrained optimization problem, by applying the indicator function to simplify it as an unconstrained optimization problem. Then we propose an algorithm based on Alternating Direction Method of Multipliers (ADMM) and theoretically prove its convergence. We make some empirical experiments on both synthetic and real datasets. The experimental results demonstrate the model’s effectiveness and showcase the desirable properties of the resulting solutions.

RDLNet: A Regularized Descriptor Learning Network.

Local image descriptor learning has been instrumental in various computer vision tasks. Recent innovations lie with similarity measurement of descriptor vectors with metric learning for randomly selected Siamese or triplet patches. Local image descriptor learning focuses more on hard samples since easy samples do not contribute much to optimization. However, few studies focus on hard samples of image patches from the perspective of loss functions and design appropriate learning algorithms to obtain a more compact descriptor representation. This article proposes a regularized descriptor learning network (RDLNet) that makes the network focus on the learning of hard samples and compact descriptor with triplet networks. A novel hard sample mining strategy is designed to select the hardest negative samples in mini-batch. Then batch margin loss concerned with hard samples is adopted to optimize the distance of extreme cases. Finally, for a more stable network and preventing network collapsing, orthogonal regularization is designed to constrain convolutional kernels and obtain rich deep features. RDLNet provides a compact discriminative low-dimensional representation and can be embedded in other pipelines easily. This article gives extensive experimental results for large benchmarks in multiple scenarios and generalization in matching applications with significant improvements.

Hybrid partial-constrained learning with orthogonality regularization for unsupervised person re-identification

Person re-identification (re-ID) aims at determining whether there is a specific person in image sets or videos via computer vision technology. State-of-the-art unsupervised re-ID methods extract image features through CNNs-based networks and store these extracted features in memory for identity matching. However, extracted global features of these methods ignore the problem of information redundancy and the influence of the constraints between the internal features. To overcome these problems, a Hybrid Partial-constrained Learning (HPcL) network with orthogonality regularization is proposed to learn a discriminative visual representation by generating hybrid features. Specifically, the hybrid features are generated by our designed Dynamic Fusion Module (DFM) to initialize the memory dictionary and match the identity, which can constrain each part of the features extracted by our proposed Multi-Scale (M-S) module and learn robust visual representations. In addition, a new orthogonal regularization method is introduced to constrain orthogonality of the kernel weights and features, which reduces the correlations among features. Extensive experimental results on Market-1501, DukeMTMC-reID, PersonX, and MSMT17 datasets demonstrate that our method is effective and superior to the state-of-the-art methods.

Generated Image Editing Method Based on Global-Local Jacobi Disentanglement for Machine Learning.

Accurate semantic editing of the generated images is extremely important for machine learning and sample enhancement of big data. Aiming at the problem of semantic entanglement in generated image latent space of the StyleGAN2 network, we proposed a generated image editing method based on global-local Jacobi disentanglement. In terms of global disentanglement, we extract the weight matrix of the style layer in the pre-trained StyleGAN2 network; obtain the semantic attribute direction vector by using the weight matrix eigen decomposition method; finally, utilize this direction vector as the initialization vector for the Jacobi orthogonal regularization search algorithm. Our method improves the speed of the Jacobi orthogonal regularization search algorithm with the proportion of effective semantic attribute editing directions. In terms of local disentanglement, we design a local contrast regularized loss function to relax the semantic association local area and non-local area and utilize the Jacobi orthogonal regularization search algorithm to obtain a more accurate semantic attribute editing direction based on the local area prior MASK. The experimental results show that the proposed method achieves SOTA in semantic attribute disentangled metrics and can discover more accurate editing directions compared with the mainstream unsupervised generated image editing methods.

Controllable facial attribute editing via Gaussian mixture model disentanglement

Generative adversarial networks (GANs) have made much progress in the field of high-quality and realistic facial image synthesis in recent years. However, compared with their powerful generation ability, it is difficult for users to modify the desired attributes of the resulting image while keeping the others. How to disentangle the latent space of pre-trained GANs is essential and critical for controllable image synthesis. In this paper, a novel controllable facial attribute editing algorithm based on the Gaussian mixture model (GMM) representation is proposed. First, we assume that the latent variables with respect to each facial attribute lie in a subspace of the whole latent manifold composed of a fixed number of learned features, and each attribute subspace can be modeled by a GMM. Then, to avoid unintended changes during attribute editing, a coordinate accumulation strategy with orthogonal regularization is introduced to enhance the independence of distinct attribute subspaces which helps improving the controllability of attribute editing. In addition, a resampling strategy is utilized to improve the stability of the model. Through qualitative and quantitative experimental results, the proposed method achieves the state-of-the-art performance on facial attribute editing, and improves the controllability of desired attribute editing.

A majorization–minimization based solution to penalized nonnegative matrix factorization with orthogonal regularization

Nonnegative matrix factorization (NMF) is a dimension reduction and clustering technique for data analysis which has been widely used in image processing, text analysis and hyperspectral decomposition because of its stronger practical significance and better interpretability. Approximate matrix factorization techniques with both nonnegativity and orthogonality constraints, referred to as orthogonal NMF (ONMF), have been shown to work remarkably better for clustering tasks than NMF. At present, a large number of algorithms have been used to solve the ONMF problems, but these methods usually cannot take into account the classification accuracy and calculation speed. In this paper, we propose a new form of penalized NMF with orthogonal regularization that combines the decomposition residual minimization based on the Euclidean distance and the orthogonality maximization based on the Kullback–Leibler divergence. This paper uses Majorization–Minimization (MM) method by minimizing a majorization function of the original problem and obtains a new iterative scheme (MM-ONMF). Comparing with several traditional ONMF methods on eight datasets, experimental results show that the proposed method has better clustering results and less computing time.

Robust local-coordinate non-negative matrix factorization with adaptive graph for robust clustering

With its unique geometric properties, non-negative matrix factorization (NMF) has become one of the widely used clustering methods in the field of data mining. Regrettably, most existing NMF methods are sensitive to super-noise (super-outliers). This paper proposes a novel robust clustering method to address this issue. Based on the Hx loss function, this method establishes a novel robust adaptive local structure learning strategy, reducing the interference of noise (outliers) on data reconstruction and space exploration. In addition, a new orthogonal regularization term is incorporated into the model, ensuring the orthogonality of the factor matrix and enhancing the discriminant ability. Finally, we develop an efficient algorithm to solve the resultant model and analyze its convergence from theoretical and experimental aspects. Experimental results on random synthetic data sets and benchmark databases demonstrate that the proposed method outperforms the existing robust NMF methods in terms of spatial structure learning, discriminant power, and robustness.

Towards an Effective Orthogonal Dictionary Convolution Strategy

Orthogonality regularization has proven effective in improving the precision, convergence speed and the training stability of CNNs. Here, we propose a novel Orthogonal Dictionary Convolution Strategy (ODCS) on CNNs to improve orthogonality effect by optimizing the network architecture and changing the regularized object. Specifically, we remove the nonlinear layer in typical convolution block “Conv(BN) + Nonlinear + Pointwise Conv(BN)”, and only impose orthogonal regularization on the front Conv. The structure, “Conv(BN) + Pointwise Conv(BN)”, is then equivalent to a pair of dictionary and encoding, defined in sparse dictionary learning. Thanks to the exact and efficient representation of signal with dictionaries in low-dimensional projections, our strategy could reduce the superfluous information in dictionary Conv kernels. Meanwhile, the proposed strategy relieves the too strict orthogonality regularization in training, which makes hyper-parameters tuning of model to be more flexible. In addition, our ODCS can modify the state-of-the-art models easily without any extra consumption in inference phase. We evaluate it on a variety of CNNs in small-scale (CIFAR), large-scale (ImageNet) and fine-grained (CUB-200-2011) image classification tasks, respectively. The experimental results show that our method achieve a stable and superior improvement.

Orthogonal Graph Neural Networks

Graph neural networks (GNNs) have received tremendous attention due to their superiority in learning node representations. These models rely on message passing and feature transformation functions to encode the structural and feature information from neighbors. However, stacking more convolutional layers significantly decreases the performance of GNNs. Most recent studies attribute this limitation to the over-smoothing issue, where node embeddings converge to indistinguishable vectors. Through a number of experimental observations, we argue that the main factor degrading the performance is the unstable forward normalization and backward gradient resulted from the improper design of the feature transformation, especially for shallow GNNs where the over-smoothing has not happened. Therefore, we propose a novel orthogonal feature transformation, named Ortho-GConv, which could generally augment the existing GNN backbones to stabilize the model training and improve the model's generalization performance. Specifically, we maintain the orthogonality of the feature transformation comprehensively from three perspectives, namely hybrid weight initialization, orthogonal transformation, and orthogonal regularization. By equipping the existing GNNs (e.g. GCN, JKNet, GCNII) with Ortho-GConv, we demonstrate the generality of the orthogonal feature transformation to enable stable training, and show its effectiveness for node and graph classification tasks.

EXTRACTION OF ORTHOGONAL BUILDING BOUNDARY FROM AIRBORNE LIDAR DATA BASED ON FEATURE DIMENSION REDUCTION

Abstract. Building boundary extraction is an active research topic in the field of feature extraction from airborne LiDAR point cloud data. Owing to the high complexity of most building extraction algorithms based on point clouds, multiple feature parameters must often be combined with iterative operations, particularly in the process of mitigating the sawtooth phenomenon using the sleeve algorithm and its improved versions. To improve the degree of automation and ensure accuracy, this study proposes a fast corner point detection method based on a dimensionality reduction technique, which utilizes reduced data mapping from 3D to 2D. We converted the boundaries extracted by the alpha shape algorithm to a 2D image and applied recursive Gaussian filtering with a relatively high level of automation to smoothen the image edges and mitigate the sawtooth phenomenon, thereby improving upon the sleeve algorithm, which requires a large number of iterations. Subsequently, the Douglas Peucker algorithm is used to retrieve the contour key points after extracting the contour lines and obtaining the regularized building contours using the grouped orthogonal regularization method. To verify the accuracy of the algorithm, it was compared with a cluster and adjustment (CAA)method based on the sleeve algorithm using three major evaluation metrics with respect to four representative building instances in two experimental datasets of urban areas. The value of the RMSE was reduced by an average of 43.79%. In addition, the time complexity decreased from O(n2) to O(n). These results demonstrate that the proposed method improves not only the accuracy of boundary extraction, but also the efficiency of data processing.

A network-based matrix factorization framework for ceRNA co-modules recognition of cancer genomic data.

With the development of high-throughput technologies, the accumulation of large amounts of multidimensional genomic data provides an excellent opportunity to study the multilevel biological regulatory relationships in cancer. Based on the hypothesis of competitive endogenous ribonucleic acid (RNA) (ceRNA) network, lncRNAs can eliminate the inhibition of microRNAs (miRNAs) on their target genes by binding to intracellular miRNA sites so as to improve the expression level of these target genes. However, previous studies on cancer expression mechanism are mostly based on individual or two-dimensional data, and lack of integration and analysis of various RNA-seq data, making it difficult to verify the complex biological relationships involved. To explore RNA expression patterns and potential molecular mechanisms of cancer, a network-regularized sparse orthogonal-regularized joint non-negative matrix factorization (NSOJNMF) algorithm is proposed, which combines the interaction relations among RNA-seq data in the way of network regularization and effectively prevents multicollinearity through sparse constraints and orthogonal regularization constraints to generate good modular sparse solutions. NSOJNMF algorithm is performed on the datasets of liver cancer and colon cancer, then ceRNA co-modules of them are recognized. The enrichment analysis of these modules shows that >90% of them are closely related to the occurrence and development of cancer. In addition, the ceRNA networks constructed by the ceRNA co-modules not only accurately mine the known correlations of the three RNA molecules but also further discover their potential biological associations, which may contribute to the exploration of the competitive relationships among multiple RNAs and the molecular mechanisms affecting tumor development.

Towards Disentangling Latent Space for Unsupervised Semantic Face Editing.

Facial attributes in StyleGAN generated images are entangled in the latent space which makes it very difficult to independently control a specific attribute without affecting the others. Supervised attribute editing requires annotated training data which is difficult to obtain and limits the editable attributes to those with labels. Therefore, unsupervised attribute editing in an disentangled latent space is key to performing neat and versatile semantic face editing. In this paper, we present a new technique termed Structure-Texture Independent Architecture with Weight Decomposition and Orthogonal Regularization (STIA-WO) to disentangle the latent space for unsupervised semantic face editing. By applying STIA-WO to GAN, we have developed a StyleGAN termed STGAN-WO which performs weight decomposition through utilizing the style vector to construct a fully controllable weight matrix to regulate image synthesis, and employs orthogonal regularization to ensure each entry of the style vector only controls one independent feature matrix. To further disentangle the facial attributes, STGAN-WO introduces a structure-texture independent architecture which utilizes two independently and identically distributed (i.i.d.) latent vectors to control the synthesis of the texture and structure components in a disentangled way. Unsupervised semantic editing is achieved by moving the latent code in the coarse layers along its orthogonal directions to change texture related attributes or changing the latent code in the fine layers to manipulate structure related ones. We present experimental results which show that our new STGAN-WO can achieve better attribute editing than state of the art methods.