Invariant Representation Research Articles

Scaling the HTM Spatial Pooler

The Hierarchical Temporal Memory Cortical Learning Algorithm (HTM CLA) is a theory and machine learning technology that aims to capture cortical algorithm of the neocortex. Inspired by the biological functioning of the neocortex, it provides a theoretical framework, which helps to better understand how the cortical algorithm inside of the brain might work. It organizes populations of neurons in column-like units, crossing several layers such that the units are connected into structures called regions (areas). Areas and columns are hierarchically organized and can further be connected into more complex networks, which implement higher cognitive capabilities like invariant representations. Columns inside of layers are specialized on learning of spatial patterns and sequences. This work targets specifically spatial pattern learning algorithm called Spatial Pooler. A complex topology and high number of neurons used in this algorithm, require more computing power than even a single machine with multiple cores or a GPUs could provide. This work aims to improve the HTM CLA Spatial Pooler by enabling it to run in the distributed environment on multiple physical machines by using the Actor Programming Model. The proposed model is based on a mathematical theory and computation model, which targets massive concurrency. Using this model drives different reasoning about concurrent execution and enables flexible distribution of parallel cortical computation logic across multiple physical nodes. This work is the first one about the parallel HTM Spatial Pooler on multiple physical nodes with named computational model. With the increasing popularity of cloud computing and server less architectures, it is the first step towards proposing interconnected independent HTM CLA units in an elastic cognitive network. Thereby it can provide an alternative to deep neuronal networks, with theoretically unlimited scale in a distributed cloud environment.

Charge instabilities of the extended attractive Hubbard model on the cubic lattice

The paramagnetic phase of the extended attractive Hubbard model on the cubic lattice is studied within the spin rotation invariant Kotliar-Ruckenstein slave boson representation at zero temperature. It is obtained that the quasiparticle residue of the Fermi liquid phase vanishes for all densities at an interaction strength slightly smaller than [Formula: see text] that signals the Brinkman–Rice transition, and that it weakly depends on density. While for vanishing non-local interaction parameters, homogeneous static charge instabilities are found in a rather narrow window centered around quarter filling and [Formula: see text], increasing them to [Formula: see text] results into a severe narrowing of this window. On the contrary, when all interaction parameters are attractive, for example for [Formula: see text], a large parameter range in which homogeneous static charge instabilities is found. Yet, this systematically happens inside the Fermi liquid phase.

Stable and dynamic representations of value in the prefrontal cortex.

Optimal decision-making requires that stimulus-value associations are kept up to date by constantly comparing the expected value of a stimulus with its experienced outcome. To do this, value information must be held in mind when a stimulus and outcome are separated in time. However, little is known about the neural mechanisms of working memory (WM) for value. Contradicting theories have suggested WM requires either persistent or transient neuronal activity, with stable or dynamic representations, respectively. To test these hypotheses, we recorded neuronal activity in the orbitofrontal and anterior cingulate cortex of two monkeys performing a valuation task. We found that features of all hypotheses were simultaneously present in prefrontal activity, and no single hypothesis was exclusively supported. Instead, mixed dynamics supported robust, time invariant value representations while also encoding the information in a temporally specific manner. We suggest that this hybrid coding is a critical mechanism supporting flexible cognitive abilities.

The White Matter Module-Hub Network of Semantics Revealed by Semantic Dementia.

Cognitive neuroscience exploring the architecture of semantics has shown that coherent supramodal concepts are computed in the anterior temporal lobes (ATL), but it is unknown how/where modular information implemented by posterior cortices (word/object/face forms) is conveyed to the ATL hub. We investigated the semantic module-hub network in healthy adults (n = 19) and in semantic dementia patients (n = 28) by combining semantic assessments of verbal and nonverbal stimuli and MRI-based fiber tracking using seeds in three module-related cortices implementing (i) written word forms (visual word form area), (ii) abstract lexical representations (posterior-superior temporal cortices), and (iii) face/object representations (face form area). Fiber tracking revealed three key tracts linking the ATL with the three module-related cortices. Correlation analyses between tract parameters and semantic scores indicated that the three tracts subserve semantics, transferring modular verbal or nonverbal object/face information to the left and right ATL, respectively. The module-hub tracts were functionally and microstructurally damaged in semantic dementia, whereas damage to non-module-specific ATL tracts (inferior longitudinal fasciculus, uncinate fasciculus) had more limited impact on semantic failure. These findings identify major components of the white matter module-hub network of semantics, and they corroborate/materialize claims of cognitive models positing direct links between modular and semantic representations. In combination with modular accounts of cognition, they also suggest that the currently prevailing "hub-and-spokes" model of semantics could be extended by incorporating an intermediate module level containing invariant representations, in addition to "spokes," which subserve the processing of a near-unlimited number of sensorimotor and speech-sound features.

Toward Accurate Sensing with Knitted Fabric: Applications and Technical Considerations

Fabric sensors have been introduced to enable flexible touch-based interaction. We advance the technical capabilities of a scalable and low-profile knitted capacitive touch sensing system by introducing methods to improve its touch localization accuracy. The sensor hardware design tends toward minimalism by using a single conductive yarn and two external connections located at each endpoint. Fewer connectors simplify the textile system integration, but this comes at the expense of reduced signal information output from the system. The electrical continuity of the sensing element, essential to the process of knitting, also increases the uncertainty of localizing touch. We propose using Bode analysis to measure changes in signal due to capacitive touch, as well as design a new algorithm, MSD, which retains the most significant aspects of the signal in terms of touch location identification. We do not classify location of touch, but focus on an invariant signal representation. To evaluate our methods, we introduce ELD, a distance metric to compute the similarity of pairs of key-presses, generalizable to computing distances of tensors of varying lengths. Our experiments show that the proposed sensing method results in high-fidelity signals. Furthermore, the sparse representation of key-presses produced by MSD significantly increases separability between different touch locations. Possible applications based on these sensors are also illustrated through prototypes and use case descriptions.

Representation learning via serial robust autoencoder for domain adaptation

Domain adaptation aims to apply knowledge obtained from a labeled source domain to an unseen target domain from a different distribution. Recently, domain adaptation approaches based on autoencoder have achieved promising performances. However, almost of these approaches ignore the potential relationships of intra-domain features, which can be used to further reduce the distribution discrepancy between source and target domains. Furthermore, almost of them depend on the single autoencoder model, which brings the challenge of extracting multiple characteristics of data. To address these issues, in this paper, we propose a new representation learning method based on serial robust autoencoder for domain adaptation, named SERA. SERA first enriches intra-domain knowledge by mining the potential relationships of features in the source domain and target domain, respectively. Then, SERA learns domain invariant representations by serially connecting two new proposed autoencoder models, including marginalized denoising autoencoder via adaptation regularization (AMDA) and robust autoencoder via graph regularization (GRA). Extensive experiments on four public datasets demonstrate the effectiveness of the proposed method.

Complete set of translation invariant measurements with Lipschitz bounds

In image and audio signal classification, a major problem is to build stable representations that are invariant under rigid motions and, more generally, to small diffeomorphisms. Translation invariant representations of signals in C n are of particular importance. The existence of such representations is intimately related to classical invariant theory, inverse problems in compressed sensing and deep learning. Despite an impressive body of literature on the subject, most representations available are either not stable due to the presence of high frequencies or non discriminative. In the present paper, we construct low dimensional representations of signals in C n that are invariant under finite unitary group actions, as a special case we establish the existence of low-dimensional and complete Z m -invariant representations for any m ∈ N . Our construction yields a stable, discriminative transform with semi-explicit Lipschitz bounds on the dimension; this is particularly relevant for applications. Using some tools from Algebraic Geometry, we define a high dimensional homogeneous function that is injective. We then exploit the projective character of this embedding and see that the target space can be reduced significantly by using a generic linear transformation. Finally, we introduce the notion of non-parallel map, which is enjoyed by our function and employ this to construct a Lipschitz modification of it.

A Multilayer Fusion Network With Rotation- Invariant and Dynamic Feature Representation for Multiview Low-Altitude Image Registration

Due to human and natural factors, when the small unmanned aerial vehicles (UAVs) are monitoring the ground, multiview transformation problems such as image distortion and low overlap will occur, which will inhibit the accuracy of low-altitude image registration and limit the subsequent application. In this letter, we propose a mismatch removal method based on the Siamese architecture to solve the issues of multiview images. A dynamic neighbor-guided patch representation is designed to enhance the representation of each feature point. Meanwhile, a multilayer fusion is used to obtain more comprehensive information on feature points, and whether a pair of points correspond depends on the similarity of its descriptors. The network is trained by adding a rotation-invariant layer to solve the inevitable rotation and image distortion in multiview scenarios. The experimental results prove that our method can deal with the scenarios of the horizontal rotation, vertical rotation, mixture, scaling, and extreme, and is better than the other five state-of-the-art methods in most scenarios.

Multi-branch guided attention network for irregular text recognition

Reading irregular text of arbitrary shape or low quality in natural scene images is a challenging task. Existing irregular scene text recognition methods mainly focus on irregular text with arbitrary shape, but rarely focus on irregular text of low quality. In this work, we propose a simple but effective method for recognizing irregular texts with arbitrary shape and low quality simultaneously. The proposed Multi-Branch guided Attention Network (MBAN) makes mutual guidance among multi-branch data in training, so as to learn invariant semantic representation between regular text images and the corresponding irregular images. Compared with the standard attention framework for text recognition, MBAN can significantly improve the performance of irregular text recognition while preserving similar performance for regular text recognition. In addition, regarding the attention drift problem encountered in standard attention network, MBAN can significantly improve the accuracy of alignment factors at each time step. We verify the effectiveness of MBAN in irregular text recognition and attention drift problem through extensive experiments. The performance of MBAN is shown to be comparable on regular datasets and superior on some irregular datasets with state-of-the-art methods.

Deep Network with Pixel-Level Rectification and Robust Training for Handwriting Recognition

Offline handwriting recognition is a well-known challenging task in the optical character recognition field due to the difficulty caused by various unconstrained handwriting styles and limited training data. In order to learn invariant feature representations for handwriting, we propose a novel method to incorporate pixel-level rectification into a CNN- and RNN-based recognizer. We also propose an adjacent output mixup method for RNN layer’s training to improve the generalization ability of the recognizer, i.e., the previous output of an RNN layer is added to the current output with random weights. We additionally adopt a series of techniques including pre-training, data augmentation and language model to significantly expand the training data scale, and further analyze their contributions to the improvement in the model performance. The proposed method performs well on four public offline handwriting benchmarks, including the IAM, Rimes, IFN/ENIT and CASIA-HWDB datasets.

/dˤ/-Variation in Saudi Newscasting and Phonological Theory

In his analysis of /dˤ/-variation in Saudi Arabian newscasting, Al-Tamimi (2020) finds unpredicatble variability between the standard variant [dˤ] and the non-standard variant [ðˤ] in different in-words positions, in different phonetic environments, and in semantically ‘content’ and suprasegmentally ‘stressed’ lexical itmes assumed to favor the standard variant. He even finds in many of these lexical items an unusual realizational flucatuation between the two variants. The present exploratory and ‘theory-testing’ study aims to find a reasonable account for these findings through examining the explanatory adequacy of a number of available phonological theories, notions, models and proposals that have made different attempts to accommodate variation, and this includes Coexistent Phonemic Systems, Standard Generative Phonology, Lexical Diffusion, Variable Rules, Poly-Lectal Grammar, Articulatory Phonology, different versions of the Optimality Theory, in addition to the Multiple-Trace-Model, as represented by Al-Tamimi’s (2005) Multiple-Trace-Based Proposal. The study reveals the strengths and weaknesses of these theories in embracing the variability in the data, and concludes that the Multiple-Trace-Based Proposal can relatively offer the best insight as its allows variation to be directly encoded in the underlying representations of lexical items, a status strictly prohibited by the rest of the theories that adopt invariant lexical representations in consonance with the ‘Homogeneity Doctrine’.

Scanner invariant representations for diffusion MRI harmonization.

PurposeIn the present work, we describe the correction of diffusion‐weighted MRI for site and scanner biases using a novel method based on invariant representation.Theory and MethodsPooled imaging data from multiple sources are subject to variation between the sources. Correcting for these biases has become very important as imaging studies increase in size and multi‐site cases become more common. We propose learning an intermediate representation invariant to site/protocol variables, a technique adapted from information theory‐based algorithmic fairness; by leveraging the data processing inequality, such a representation can then be used to create an image reconstruction that is uninformative of its original source, yet still faithful to underlying structures. To implement this, we use a deep learning method based on variational auto‐encoders (VAE) to construct scanner invariant encodings of the imaging data.ResultsTo evaluate our method, we use training data from the 2018 MICCAI Computational Diffusion MRI (CDMRI) Challenge Harmonization dataset. Our proposed method shows improvements on independent test data relative to a recently published baseline method on each subtask, mapping data from three different scanning contexts to and from one separate target scanning context.ConclusionsAs imaging studies continue to grow, the use of pooled multi‐site imaging will similarly increase. Invariant representation presents a strong candidate for the harmonization of these data.

Invariant Representations through Adversarial Forgetting

We propose a novel approach to achieving invariance for deep neural networks in the form of inducing amnesia to unwanted factors of data through a new adversarial forgetting mechanism. We show that the forgetting mechanism serves as an information-bottleneck, which is manipulated by the adversarial training to learn invariance to unwanted factors. Empirical results show that the proposed framework achieves state-of-the-art performance at learning invariance in both nuisance and bias settings on a diverse collection of datasets and tasks.

Exploiting Spatial Invariance for Scalable Unsupervised Object Tracking

The ability to detect and track objects in the visual world is a crucial skill for any intelligent agent, as it is a necessary precursor to any object-level reasoning process. Moreover, it is important that agents learn to track objects without supervision (i.e. without access to annotated training videos) since this will allow agents to begin operating in new environments with minimal human assistance. The task of learning to discover and track objects in videos, which we call unsupervised object tracking, has grown in prominence in recent years; however, most architectures that address it still struggle to deal with large scenes containing many objects. In the current work, we propose an architecture that scales well to the large-scene, many-object setting by employing spatially invariant computations (convolutions and spatial attention) and representations (a spatially local object specification scheme). In a series of experiments, we demonstrate a number of attractive features of our architecture; most notably, that it outperforms competing methods at tracking objects in cluttered scenes with many objects, and that it can generalize well to videos that are larger and/or contain more objects than videos encountered during training.

Deep Attentive Ranking Networks for Learning to Order Sentences

We present an attention-based ranking framework for learning to order sentences given a paragraph. Our framework is built on a bidirectional sentence encoder and a self-attention based transformer network to obtain an input order invariant representation of paragraphs. Moreover, it allows seamless training using a variety of ranking based loss functions, such as pointwise, pairwise, and listwise ranking. We apply our framework on two tasks: Sentence Ordering and Order Discrimination. Our framework outperforms various state-of-the-art methods on these tasks on a variety of evaluation metrics. We also show that it achieves better results when using pairwise and listwise ranking losses, rather than the pointwise ranking loss, which suggests that incorporating relative positions of two or more sentences in the loss function contributes to better learning.

Illumination and Rotation Invariant Texture Representation for Face Recognition

This article presents a novel approach for illumination and rotation invariant texture representation for face recognition. A gradient transformation is used as illumination invariance property and a Galois Field for the rotation invariance property. The normalized cumulative histogram bin values of the Gradient Galois Field transformed image represent the illumination and rotation invariant texture features. These features are further used as face descriptors. Experimentations are performed on FERET and extended Cohn Kanade databases. The results show that the proposed method is better as compared to Rotation Invariant Local Binary Pattern, Log-polar transform and Sorted Local Gradient Pattern and is illumination and rotation invariant.

Learning in the machine: To share or not to share?

Weight-sharing is one of the pillars behind Convolutional Neural Networks and their successes. However, in physical neural systems such as the brain, weight-sharing is implausible. This discrepancy raises the fundamental question of whether weight-sharing is necessary. If so, to which degree of precision? If not, what are the alternatives? The goal of this study is to investigate these questions, primarily through simulations where the weight-sharing assumption is relaxed. Taking inspiration from neural circuitry, we explore the use of Free Convolutional Networks and neurons with variable connection patterns. Using Free Convolutional Networks, we show that while weight-sharing is a pragmatic optimization approach, it is not a necessity in computer vision applications. Furthermore, Free Convolutional Networks match the performance observed in standard architectures when trained using properly translated data (akin to video). Under the assumption of translationally augmented data, Free Convolutional Networks learn translationally invariant representations that yield an approximate form of weight-sharing.

One-shot learning of view-invariant object representations in newborn chicks

Can newborn brains perform one-shot learning? To address this question, we reared newborn chicks in strictly controlled environments containing a single view of a single object, then tested their object recognition performance across 24 uniformly-spaced viewpoints. We found that chicks can build view-invariant object representations from a single view of an object: a case of one-shot learning in newborn brains. Chicks can also build the same view-invariant object representation from different views of an object, showing that newborn brains converge on common object representations from different sets of sensory inputs. Finally, by rearing chicks with larger numbers of object views, we found that chicks develop enhanced recognition for familiar views. These results illuminate the earliest stages of object recognition, revealing (1) powerful one-shot learning that builds invariant object representations from the first views of an object and (2) view-based learning that enriches object representations, producing enhanced recognition for familiar views.

Ensemble deep contractive auto-encoders for intelligent fault diagnosis of machines under noisy environment

Intelligent fault diagnosis methods based on deep auto-encoder have achieved great success in the past several years. However, these methods cannot effectively handle the data collected under noisy environment. Therefore, this paper proposes a new ensemble deep contractive auto-encoder (EDCAE) to address the problem. First, we design fifteen deep contractive auto-encoders (DCAE) to learn invariant feature representation automatically. Due to the Jacobian penalty term in DCAE and different characteristics, these models can deal with various noisy data effectively. Second, fisher discriminant analysis is applied to select low-dimensional features with the maximum class separability. Softmax classifier is adopted to identify the selected features and produce fifteen classification results. Finally, a new combination strategy is developed to combine these individual results. Benefitting from the combination strategy, it can produce accurate diagnosis results even under strong background noise. Additionally, to prove the effectiveness of EDCAE, theory analysis about error bound is conducted. The proposed method is verified on three case studies including bearing, gear box and self-priming centrifugal pump. Experiments are conducted under seven different signal-to-noise-ratios. Results show that EDCAE is better than other intelligent diagnosis methods, including individual DCAE, deep auto-encoder, sparse deep auto-encoder, deep denoising auto-encoder and several ensemble methods.

Heterogeneous Daily Living Activity Learning Through Domain Invariant Feature Subspace

In the practical applications of supervised learning methods, the high cost of obtaining labeled data for learning tasks is a critical problem. One promising research area for solving the problem is transfer learning, which aims to learn a task in target domain by utilizing the training data in a different but related source domain. In this article, we propose a novel heterogeneous transfer learning algorithm called Heterogeneous Daily Living Activity Learning (HDLAL) which derives domain invariant feature representation space from cross-domain data distributions by projecting both domain data into the derived space in the close proximity of each other using Maximum Mean Discrepancy. Within the new feature space, we utilize ensemble classification algorithm to train multi-label classifier using the projected data to predict the labels in the target domain. We show the effectiveness of our approach by experimenting on real-world smart home datasets. The results shows that our HDLAL algorithm outperforms the common direct learning approaches in the context of predicting labels of activities of daily living (ADL).