Loss Of Orthogonality Research Articles

Improving Systematic Generalization of Linear Transformer Using Normalization Layers and Orthogonality Loss Function

A Linear Transformer linearizes the attention mechanism of the vanilla Transformer architecture, significantly improving efficiency and achieving linear theoretical complexity with respect to sequence length. However, few studies have explored the capabilities of the Linear Transformer beyond its efficiency. In this work, we investigate the systematic generalization capability of the Linear Transformer, a crucial property for strong generalization to unseen data. Through preliminary experiments, we identify two major issues contributing to its unstable systematic generalization performance: (i) unconstrained norms of Queries and Keys, and (ii) high correlation among Values across the sequence. To address these issues, we propose two simple yet effective methods: normalization layers for Queries and Keys, and an orthogonality loss function applied to Values during training. In experiments, we demonstrate that applying these methods to the Linear Transformer significantly improves its stability and systematic generalization performance across several well-known tasks. Furthermore, our proposed methods outperform the vanilla Transformer on specific systematic generalization tasks, such as the sort-of-CLEVR and SCAN tasks.

Fault domain separation networks: A cross-domain diagnostic method for variable operating conditions in the elevator guidance system

High-speed elevators play a critical role in vertical transportation. However, they may exhibit different working conditions under varying operating conditions, posing challenges for fault diagnosis. We propose a model named fault domain separation networks (FDSN) to tackle the cross-domain diagnostic problem of guidance system by leveraging transfer learning techniques. FDSN comprises three distinct modules that extract invariant fault features and variable operating condition features, and use orthogonal loss function for feature decoupling, achieving effective feature allocation, and mitigating the impact of variable operating conditions on fault diagnosis. To evaluate and demonstrate the efficacy of FDSN, a high-speed elevator platform is constructed, and fault vibration data under various working conditions are simulated to form a comprehensive dataset. The proposed FDSN achieved an average diagnostic accuracy of 92.7% in cross-domain diagnostic tasks, outperforming other comparative methods and demonstrating the remarkable superiority.

Learning adaptive shift and task decoupling for discriminative one-step person search

Mainstream person search models aim to jointly optimize person detection and re-identification (ReID) in a one-step manner. Despite notable progress, existing one-step person search models still face three major challenges in extracting discriminative features: 1) incomplete feature extraction and fusion hinder the effective utilization of multiscale information, 2) the models struggle to capture critical features in complex occlusion scenarios, and 3) the optimization objectives of person detection and ReID are in conflict in the shared feature space. To address these issues, this study proposes a novel adaptive shift and task decoupling (ASTD) method that aims to enhance the accuracy and robustness of extracting discriminative features within the region of interest. In particular, we introduce a scale-aware transformer to handle scale/pose variations and occlusions. This transformer incorporates scale-aware modulation to enhance the utilization of multiscale information and adaptive shift augmentation to learn adaptation to occlusions dynamically. In addition, we design a task decoupling mechanism to hierarchically learn independent task representations using orthogonal loss to decouple two subtasks during training. Experimental results show that ASTD achieves state-of-the-art performance on the CUHK-SYSU and PRW datasets. Our code is accessible at https://github.com/zqx951102/ASTD.

VPDETR: End-to-End Vanishing Point DEtection TRansformers

In the field of vanishing point detection, previous works commonly relied on extracting and clustering straight lines or classifying candidate points as vanishing points. This paper proposes a novel end-to-end framework, called VPDETR (Vanishing Point DEtection TRansformer), that views vanishing point detection as a set prediction problem, applicable to both Manhattan and non-Manhattan world datasets. By using the positional embedding of anchor points as queries in Transformer decoders and dynamically updating them layer by layer, our method is able to directly input images and output their vanishing points without the need for explicit straight line extraction and candidate points sampling. Additionally, we introduce an orthogonal loss and a cross-prediction loss to improve accuracy on the Manhattan world datasets. Experimental results demonstrate that VPDETR achieves competitive performance compared to state-of-the-art methods, without requiring post-processing.

WGDPool: A broad scope extraction for weighted graph data

Graph pooling is a commonly used operation in graph neural networks to reduce the size of graph representation. To extract key information, pooling and representation need to be coupled. We propose a graph pooling method for weighted graphs called WGDPool (Weighted Graph Dual Pooling). Unlike traditional graph representation learning methods, the weight information of edges is also fed into convolutional graph neural networks (ConvGNN) to obtain graph representations. Dual branch convolutional graph neural networks is designed to learn the nodes’ and edges’ embeddings independently, and they are fused into a comprehensive representation of graph data. Pooling, as a tool of feature extraction and scale reduction of graph representation, adopts a differentiable version of k-means clustering and a multi-item parameterized loss function. Cut loss, orthogonality loss, clustering loss, and reconstruction loss are simultaneously considered. By parameterization, WGDPool is competent for diverse graph tasks. WGDPool outperformed other graph pooling methods in such common supervised and unsupervised tasks as biological or chemical classification, bibliography clustering and integrated circuit partition, demonstrating the effectiveness of our proposed pooling method.

High spectral efficiency modulation scheme based on joint interaction of orthogonal compressed chirp division multiplexing and power superimposed code.

In this paper, we propose a high spectral efficiency modulation scheme based on joint interaction of orthogonal compressed chirp division multiplexing (OCCDM) and power superimposed code (PSC) under the intensity modulation and direct detection (IM/DD) system. OCCDM is a novel orthogonal chirp division multiplexing technology featuring spectral compression through the implementation of processing similar to a discrete Fourier transform, enhancing the spectral efficiency (SE) through bandwidth savings without loss of orthogonality of each chirp. Meanwhile, PSC technology enables multiple code words being transmitted superimposed on the same chirp. This technique involves allocating varying power levels to different users, thereby distinguishing them, increasing the transmission's net bit rate and substantially boosting the SE. The transmission has been performed experimentally using a 2 km 7-core fiber span. The impact of the above-mentioned technologies on the bit error rate (BER) performance is assessed in the power, frequency, and joint domain. The BER and enhancements in the SE can be balanced when the spectral bandwidth compression factor (α) and power distribution ratio are equal to 0.9 and 4, respectively. The observed outcome leads to the transmission's SE increase to more than double the baseline value, at 2.22 times. Based on the above analysis, we believe this structure is expected to become a potential for developing next-generation PON.

Inverse-Like Antagonistic Scene Text Spotting via Reading-Order Estimation and Dynamic Sampling.

Scene text spotting is a challenging task, especially for inverse-like scene text, which has complex layouts, e.g., mirrored, symmetrical, or retro-flexed. In this paper, we propose a unified end-to-end trainable inverse-like antagonistic text spotting framework dubbed IATS, which can effectively spot inverse-like scene texts without sacrificing general ones. Specifically, we propose an innovative reading-order estimation module (REM) that extracts reading-order information from the initial text boundary generated by an initial boundary module (IBM). To optimize and train REM, we propose a joint reading-order estimation loss ( LRE ) consisting of a classification loss, an orthogonality loss, and a distribution loss. With the help of IBM, we can divide the initial text boundary into two symmetric control points and iteratively refine the new text boundary using a lightweight boundary refinement module (BRM) for adapting to various shapes and scales. To alleviate the incompatibility between text detection and recognition, we propose a dynamic sampling module (DSM) with a thin-plate spline that can dynamically sample appropriate features for recognition in the detected text region. Without extra supervision, the DSM can proactively learn to sample appropriate features for text recognition through the gradient returned by the recognition module. Extensive experiments on both challenging scene text and inverse-like scene text datasets demonstrate that our method achieves superior performance both on irregular and inverse-like text spotting.

Orthogonal Space-Time Block Coding for Double Scattering V2V Links with LOS and Ground Reflections

This work presents the performance analysis of space-time block codes (STBCs) for vehicle-to-vehicle (V2V) fast-fading channels in scenarios with modified line-of-sight (LOS). The objective is to investigate how the V2V MIMO (multiple-input multiple-output) system performance is influenced by two important impairments: deterministic ground reflections and an increased Doppler frequency (time-variant channels). STBCs of various coding rates (using an approximation model) are evaluated by assuming antenna elements distributed over the surface of two contiguous vehicles. A multi-ray model is used to study the multiple constructive/destructive interference patterns of the transmitted/received signals by all pairs of Tx–Rx antenna links considering ground reflections. A double scattering model is used to include the effects of stochastic channel components that depend on the Doppler frequency. The results show that STBCs are capable of counteracting fades produced by destructive self-interference components across a range of inter-vehicle distances and for a range of Doppler frequency values. Notably, the effectiveness of STBCs in deep fades is shown to outperform schemes with exclusive receive diversity, despite the interference created by the loss of orthogonality in time-varying channels with a moderate increase of Doppler frequency (mainly due to higher vehicle speeds, higher frequency or shorter time slots). Higher-order STBCs with rate losses are also evaluated using an approximation model, showing interesting gains even for low coding rate performance, particularly when accompanied by a multiple antenna receiver. Overall, these results can shed light on how to exploit transmit diversity in time-varying vehicular channels with modified LOS.

Self-supervised Graph-level Representation Learning with Adversarial Contrastive Learning

The recently developed unsupervised graph representation learning approaches apply contrastive learning into graph-structured data and achieve promising performance. However, these methods mainly focus on graph augmentation for positive samples, while the negative mining strategies for graph contrastive learning are less explored, leading to sub-optimal performance. To tackle this issue, we propose a Graph Adversarial Contrastive Learning (GraphACL) scheme that learns a bank of negative samples for effective self-supervised whole-graph representation learning. Our GraphACL consists of (i) a graph encoding branch that generates the representations of positive samples and (ii) an adversarial generation branch that produces a bank of negative samples. To generate more powerful hard negative samples, our method minimizes the contrastive loss during encoding updating while maximizing the contrastive loss adversarially over the negative samples for providing the challenging contrastive task. Moreover, the quality of representations produced by the adversarial generation branch is enhanced through the regularization of carefully designed bank divergence loss and bank orthogonality loss. We optimize the parameters of the graph encoding branch and adversarial generation branch alternately. Extensive experiments on 14 real-world benchmarks on both graph classification and transfer learning tasks demonstrate the effectiveness of the proposed approach over existing graph self-supervised representation learning methods.

Automatic Loss Function Search for Adversarial Unsupervised Domain Adaptation

Unsupervised domain adaption (UDA) aims to reduce the domain gap between labeled source and unlabeled target domains. Many prior works exploit adversarial learning that leverages pre-designed discriminators to drive the network for aligning distributions between domains. However, most of them do not consider the degeneration of the domain discriminators caused by the gradually dominating gradients of aligned target samples during training, and they still suffer from the cross-domain semantic mismatch problem in the learned feature space. Hence, this paper attempts to understand and solve both issues from the lens of optimization loss and propose an automatic loss function search for adversarial domain adaptation (ALSDA). First, we extend the common adversarial loss by adding an adjustable hyper-parameter that can re-weight the gradients assigned to target samples, so that the domain discriminator can impose consecutive and influential driving forces for domain alignment. Meanwhile, we upgrade the traditional orthogonality loss with class-wisely adjustable hyper-parameters that can strengthen the cross-domain feature separation. Since manually determining the optimal loss functions requires expensive expert efforts, we leverage the popular AutoML to automatically search for the optimal loss functions from a pre-defined novel and unique search space for UDA. Further, to enable the loss function search when the target domain is unlabeled, we introduce a simple-but-effective entropy-guided search strategy with the aid of REINFORCE learning. Extensive experiments on various typical baselines and benchmark datasets such as Office-Home, Office-31, and Birds-31 have been conducted, and the results validate the generalization and superiority of the proposed ALSDA.

ISFB-GAN: Interpretable semantic face beautification with generative adversarial network

Face beautification is an emerging application widely used in the entertainment industry. Previous studies have proved that Generative Adversarial Networks (GANs) can successfully build the mapping from a general face to a target face. However, their facial style code in the latent space is taken as a black box with a lack of semantics interpretation. They usually fail to consider the geometric semantic information, and cannot focus on the facial beauty-related regions. Moreover, most of them change the background information. To overcome these problems, we propose a novel Interpretable Semantic Generative Adversarial Network (ISFB-GAN) for face beautification. First, we design a new dual-branch facial style encoder to factorize the abundant semantic information of the reference face into the facial geometry style codes and the appearance style codes in the latent space. Then, a novelty facial attention generator is proposed to preserve more background information in the generated images. In the attention generator, the facial style attention is designed to focus on the most discriminative facial region; the background attention is designed to retain the background information. Finally, we propose a hybrid loss function including geometric loss, orthogonality loss, segmentation loss, style reconstruction loss, perceptual loss, etc., to enhance the learning capacity and improve the visual quality. The combination of the geometric loss and the orthogonality loss is deployed to guarantee the disentanglement between the semantic information. Extensive experiments conducted on the CelebAMask-HQ dataset and the SCUT-FBP5500 dataset, demonstrate that ISFB-GAN performs better than the state-of-the-art methods in facial attractiveness enhancement.

Deep multi-view spectral clustering via ensemble

Graph-based methods have achieved great success in multi-view clustering. However, existing graph-based models generally utilize shallow and linear embedding functions to obtain the common spectral embedding for clustering assignments. In addition, the fusion similarity graphs from multiple views are generally obtained by a simple weighted-sum rule. To this end, we propose a novel deep multi-view spectral clustering via ensemble model (DMCE), which applies ensemble clustering to fuse the similarity graphs from different views. On this basis, we employ the graph auto-encoder to learn the common spectral embedding, which can be regarded as the indicator matrix directly. Moreover, a unified optimization framework is designed to update the variables in the proposed DMCE, which consists of graph reconstruction loss, orthogonal loss, and graph contrastive learning loss. Extensive experiments on six real-world benchmark datasets have demonstrated the effectiveness of our model compared with the state-of-the-art multi-view clustering methods.

DADRnet: Cross-domain image dehazing via domain adaptation and disentangled representation

Recent years have witnessed remarkable progress of learning-based methods in single image dehazing. Among them, dehazing methods trained on the synthetic images cannot adapt to real hazy ones due to the domain gap, and domain adaptation methods only concentrate on creating a mapping or extracting shared features regardless of representations of deep features. In this paper, we propose an innovative cross-domain dehazing architecture that integrates domain adaptation and disentangled representation. Specifically, we first construct a shared encoder to map synthetic and real hazy images to a latent space and narrow their domain gap at the feature level. Then we utilize a separator to separate the hazy image features into haze-free representations and haze ones. We introduce feature consistency loss and orthogonal loss to further guide the disentanglement process. And then, we utilize a decoder to produce clean images from haze-free features and introduce both supervised and unsupervised losses to guide the training process. Moreover, we also propose to reconstruct the images from both domains by recombining the separated features, which guarantees information completeness. Experiments demonstrate that our method is on par with state-of-the-art methods. Codes are available at https://github.com/lixiaopeng123456/DADRnet.

Using Mixed Precision in Low‐Synchronization Reorthogonalized Block Classical Gram‐Schmidt

AbstractMixed precision hardware has recently become commercially available, and more than 25% of the supercomputers in the TOP500 list now have mixed precision capabilities. Using lower precision in algorithms can be beneficial in terms of reducing both computation and communication costs. There are many current efforts towards developing mixed precision numerical linear algebra algorithms, which can lead to speedups in real applications. Motivated by this, we aim to further the state‐of‐the‐art in developing and analyzing mixed precision variants of iterative methods. In iterative methods based on Krylov subspaces, the orthogonal basis is generated by Arnoldi or Lanczos methods or their variants. In long recurrence methods such as GMRES, one needs to use an explicit orthogonalization scheme such as Gram‐Schmidt to orthonormalize the vectors generated.Krylov subspace methods, such as GMRES, are typically communication‐bound on modern machines; the runtime is usually dominated by the cost of global synchronizations which are required for the necessary orthogonalization. This has motivated the development of various algorithmic variants which attempt to reduce the communication cost while maintaining a stable algorithm. Recent work has focused on the development of low‐synchronization block variants of Gram‐Schmidt orthogonalization, which, when used within communication‐avoiding (s‐step) or block variants of GMRES, can reduce the number of global synchronizations to one per block.In this work, we focus on the block variant of low‐synchronization classical Gram‐Schmidt with reorthogonalization, which we call BCGSI+LS. We demonstrate that the loss of orthogonality produced by this orthogonalization scheme can exceed O(u)κ(𝒳), where u is the unit roundoff and κ(𝒳) is the condition number of the matrix to be orthogonalized, and thus we can not in general expect this to result in a backward stable block GMRES implementation. We then develop a mixed precision variant of this algorithm, called BCGSI+LS‐MP, which uses higher precision in certain parts of the computation. We demonstrate experimentally that for a number of challenging test problems, our mixed precision variant successfully maintains a loss of orthogonality below O(u)κ(𝒳). This indicates that we can achieve a backward stable block GMRES algorithm that requires only one synchronization per iteration.

Multi-level Attention-based Domain Disentanglement for BCDR

Cross-domain recommendation aims to exploit heterogeneous information from a data-sufficient domain (source domain) to transfer knowledge to a data-scarce domain (target domain). A majority of existing methods focus on unidirectional transfer that leverages the domain-shared information to facilitate the recommendation of the target domain. Nevertheless, it is more beneficial to improve the recommendation performance of both domains simultaneously via a dual transfer learning schema, which is known as bidirectional cross-domain recommendation (BCDR). Existing BCDR methods have their limitations, since they only perform bidirectional transfer learning based on domain-shared representations while neglecting rich information that is private to each domain. In this article, we argue that users may have domain-biased preferences due to the characteristics of that domain. Namely, the domain-specific preference information also plays a critical role in the recommendation. To effectively leverage the domain-specific information, we propose a M ulti-level A ttention-based D omain D isentanglement framework dubbed MADD for BCDR, which explicitly leverages the attention mechanism to construct personalized preference with both domain-invariant and domain-specific features obtained by disentangling raw user embeddings. Specifically, the domain-invariant feature is exploited by domain-adversarial learning while the domain-specific feature is learned by imposing an orthogonal loss. We then conduct a reconstruction process on disentangled features to ensure semantic-sufficiency. After that, we devise a multi-level attention mechanism for these disentangled features, which determines their contributions to the final personalized user preference embedding by dynamically learning the attention scores of individual features. We train the model in a multi-task learning fashion to benefit both domains. Extensive experiments on real-world datasets demonstrate that our model significantly outperforms state-of-the-art cross-domain recommendation approaches.

The infinite Lanczos method for symmetric nonlinear eigenvalue problems

A new iterative method for solving large scale symmetric nonlinear eigenvalue problems is presented. We firstly derive an infinite dimensional symmetric linearization of the nonlinear eigenvalue problem, then we apply the indefinite Lanczos method to this specific linearization, resulting in a short-term recurrence. We show how, under specific assumption on the starting vector, this method can be carried out in finite arithmetic and how the exploitation of the problem structure leads to improvements in terms of computation time. The eigenpair approximations are extracted with the nonlinear Rayleigh-Ritz procedure combined with a specific choice of the projection space. We illustrate how this extraction technique resolves the instability issues that may occur due to the loss of orthogonality in many standard Lanczos-type methods.

An enhanced text classification model by the inverted attention orthogonal projection module

ABSTRACT The orthogonal projection method has made significant progress in text classification, especially in generating discriminative features. This method obtains more pure and suitable for classification features by projecting text features onto the orthogonal direction of common features (which are not helpful for classification and actually confuse performance). However, this approach requires an additional branch network to generate these common features, which reduces the flexibility of this method compared to representation optimisation methods such as self-attention mechanisms, as it requires significant modification of the base network structure to use. To address this issue, this paper proposes the Inversed Attention Orthogonal Projection Module (IAOPM). IAOPM uses inversed attention (IA) to iteratively reverse the attention map on text features, encouraging the network to remove discriminating features from the text features and obtain potential common features. Unlike the original orthogonal projection method, IAOPM can extract common features within a single network without any branch networks, increasing the flexibility of the orthogonal projection method. We also use an orthogonal loss to ensure the quality of the common features during training, so IAOPM also has better purity performance than the original method. Experiments show that text classification models based on IAOPM outperform the baseline models, self-attention mechanisms, and the original orthogonal projection method on multiple text classification datasets with an average accuracy increase of 1.02%, 0.44%, and 0.52%, respectively.

A flexible block classical Gram–Schmidt skeleton with reorthogonalization

AbstractWe investigate a variant of the reorthogonalized block classical Gram–Schmidt method for computing the QR factorization of a full column rank matrix. Our aim is to bound the loss of orthogonality even when the first local QR algorithm is only conditionally stable. In particular, this allows the use of modified Gram–Schmidt instead of Householder transformations as the first local QR algorithm. Numerical experiments confirm the stable behavior of the new variant. We also examine the use of non‐QR local factorization and show by example that the resulting variants, although less stable, may also be applied to ill‐conditioned problems.

Self-supervised monocular depth estimation in fog

Self-supervised depth estimation has achieved remarkable results in sunny weather. However, in the foggy scenes, their performance is limited because of the low contrast and limited visibility caused by the fog. To address this problem, an end-to-end feature separation network for self-supervised depth estimation of fog images is proposed. We take paired clear and synthetic foggy images as input, separate the image information into interference information (illumination, fog, etc.) and invariant information (structure, texture, etc.) by a feature extractor with orthogonality loss. The invariant information is used to estimate depth. Meanwhile, similarity loss is introduced to constrain the fog image depth using the depth of the clear image as a pseudo-label, and an attention module and reconstruction loss are added to refine the output depth, so that better depth maps can be obtained. Then, real-world fog images are used for fine-tuning, which effectively reduces the domain gap between synthetic data and real data. Experiments show that our approach produces advanced results on both synthetic datasets and Cityscape datasets, demonstrating the superiority of our approach.

Rethinking adversarial domain adaptation: Orthogonal decomposition for unsupervised domain adaptation in medical image segmentation.

Medical image segmentation methods based on deep learning have made remarkable progress. However, such existing methods are sensitive to data distribution. Therefore, slight domain shifts will cause a decline of performance in practical applications. To relieve this problem, many domain adaptation methods learn domain-invariant representations by alignment or adversarial training whereas ignoring domain-specific representations. In response to this issue, this paper rethinks the traditional domain adaptation framework and proposes a novel orthogonal decomposition adversarial domain adaptation (ODADA) architecture for medical image segmentation. The main idea behind our proposed ODADA model is to decompose the input features into domain-invariant and domain-specific representations and then use the newly designed orthogonal loss function to encourage their independence. Furthermore, we propose a two-step optimization strategy to extract domain-invariant representations by separating domain-specific representations, fighting the performance degradation caused by domain shifts. Encouragingly, the proposed ODADA framework is plug-and-play and can replace the traditional adversarial domain adaptation module. The proposed method has consistently demonstrated effectiveness through comprehensive experiments on three publicly available datasets, including cross-site prostate segmentation dataset, cross-site COVID-19 lesion segmentation dataset, and cross-modality cardiac segmentation dataset. The source code is available at https://github.com/YonghengSun1997/ODADA.