Hierarchical Learning Structure Research Articles

Predictive coding with spiking neurons and feedforward gist signaling.

Predictive coding (PC) is an influential theory in neuroscience, which suggests the existence of a cortical architecture that is constantly generating and updating predictive representations of sensory inputs. Owing to its hierarchical and generative nature, PC has inspired many computational models of perception in the literature. However, the biological plausibility of existing models has not been sufficiently explored due to their use of artificial neurons that approximate neural activity with firing rates in the continuous time domain and propagate signals synchronously. Therefore, we developed a spiking neural network for predictive coding (SNN-PC), in which neurons communicate using event-driven and asynchronous spikes. Adopting the hierarchical structure and Hebbian learning algorithms from previous PC neural network models, SNN-PC introduces two novel features: (1) a fast feedforward sweep from the input to higher areas, which generates a spatially reduced and abstract representation of input (i.e., a neural code for the gist of a scene) and provides a neurobiological alternative to an arbitrary choice of priors; and (2) a separation of positive and negative error-computing neurons, which counters the biological implausibility of a bi-directional error neuron with a very high baseline firing rate. After training with the MNIST handwritten digit dataset, SNN-PC developed hierarchical internal representations and was able to reconstruct samples it had not seen during training. SNN-PC suggests biologically plausible mechanisms by which the brain may perform perceptual inference and learning in an unsupervised manner. In addition, it may be used in neuromorphic applications that can utilize its energy-efficient, event-driven, local learning, and parallel information processing nature.

Learning multi-granular worker intentions from incomplete visual observations for worker-robot collaboration in construction

Construction is often considered a dangerous and labor-intensive industry, with safety and workforce shortages plaguing the industry globally. Owing to its flexibility, Worker-Robot Collaboration (WRC) could combine the strengths of humans and robots, alleviating human workers from dangerous and tedious tasks and boosting productivity. WRC envisions robots assisting workers appropriately, which necessitates the ability to understand workers' intentions. However, challenges arise when robots recognize workers' intentions in construction. First, the highly unstructured and dynamic construction workplace prevents robots from capturing consistent long-term worker activities, which is detrimental to the information accumulation and disambiguation processes in intention recognition. Second, the long-horizon tasks with implicit steps make it difficult for robots to find a suitable granularity to capture workers' intentions. Considering these challenges, this study seeks to (1) explore whether it is feasible to recognize worker intentions with temporally incomplete observations and (2) investigate which granularity is best suited for recognizing worker intentions. To tackle the first challenge, a data-and-knowledge fusion method is proposed, where the contextual information from visual data is combined with task knowledge to mitigate the information loss due to incomplete observations. To address the second challenge, the best-suited granularity is determined by comparing the recognition accuracy and intention modeling ability of different granularities after defining and recognizing multi-granular worker intentions using a hierarchical task structure and Multi-Task Learning (MTL), respectively. Results show that the proposed method can recognize multi-granular worker intentions with macro-F1 scores higher than 0.85, and that the intermediate activity is the best-suited granularity as it strikes a good balance between intention recognition accuracy and intention modeling capability. This study contributes to the existing body of knowledge by recognizing multi-granular worker intentions and determining the appropriate granularity under temporally incomplete observations. The findings will help facilitate seamless WRC in construction.

Multi-view graph network learning framework for identification of major depressive disorder

Functional connectivity (FC) derived from resting-state functional magnetic resonance imaging (rs-fMRI) exhibits non-Euclidean topological structures, which have pathological foundations and serve as ideal objective data for intelligent diagnosis of major depressive disorder (MDD) patients. Additionally, the fully connected FC demonstrates uniform spatial structures. To learn and integrate information from these two structural forms for a more comprehensive identification of MDD patients, we propose a novel hierarchical learning structure called Multi-View Graph Neural Network (MV-GNN). In MV-GNN, the collaborative FC of subjects is filtered and reconstructed from topological view to obtain the reconstructed FC, incorporating various threshold values to calculate the topological attributes of brain regions. ROC analysis is performed on the average scores of these attributes for MDD and healthy control (HC) groups to determine an efficient threshold. Group differences analysis is conducted on the efficient topological attributes of brain regions, followed by their selection. These efficient attributes, along with the reconstructed FC, are combined to construct a graph view using self-attention graph pooling and graph convolutional neural networks, enabling efficient embedding. To extract efficient FC pattern difference information from spatial view, a dual leave-one-out cross-feature selection method is proposed. It selects and extracts relevant information from uniformly sized FC structures’ high-dimensional spatial features, constructing a relationship view between brain regions. This approach incorporates both the whole graph topological view and spatial relationship view in a multi-layered structure, fusing them using gating mechanisms. By incorporating multiple views, it enhances the inference of whether subjects suffer from MDD and reveals differential information between MDD and HC groups across different perspectives. The proposed model structure is evaluated through leave-one-site cross-validation and achieves an average accuracy of 65.61% in identifying MDD patients at a single-center site, surpassing state-of-the-art methods in MDD recognition. The model provides valuable discriminatory information for objective diagnosis of MDD and serves as a reference for pathological foundations.

A Long Skip Connection for Enhanced Color Selectivity in CNN Architectures.

Some recent studies show that filters in convolutional neural networks (CNNs) have low color selectivity in datasets of natural scenes such as Imagenet. CNNs, bio-inspired by the visual cortex, are characterized by their hierarchical learning structure which appears to gradually transform the representation space. Inspired by the direct connection between the LGN and V4, which allows V4 to handle low-level information closer to the trichromatic input in addition to processed information that comes from V2/V3, we propose the addition of a long skip connection (LSC) between the first and last blocks of the feature extraction stage to allow deeper parts of the network to receive information from shallower layers. This type of connection improves classification accuracy by combining simple-visual and complex-abstract features to create more color-selective ones. We have applied this strategy to classic CNN architectures and quantitatively and qualitatively analyzed the improvement in accuracy while focusing on color selectivity. The results show that, in general, skip connections improve accuracy, but LSC improves it even more and enhances the color selectivity of the original CNN architectures. As a side result, we propose a new color representation procedure for organizing and filtering feature maps, making their visualization more manageable for qualitative color selectivity analysis.

Working memory is supported by learning to represent items as actions.

Working memory is typically described as a set of processes that allow for the maintenance and manipulation of information for proximal actions, yet the "action" portion of this construct is commonly overlooked. In contrast, neuroscience-informed theories of working memory have emphasized the hierarchical nature of memory representations, including both goals and sensory representations. These two representational domains are combined for the service of actions. Here, we tested whether, as it is commonly measured (i.e., with computer-based stimuli and button-based responses), working memory involved the planning of motor actions (i.e., specific button presses). Next, we examined the role of motor plan learning in successful working memory performance. Results showed that visual working memory performance was disrupted by unpredictable motor mappings, indicating a role for motor planning in working memory. Further, predictable motor mappings were in fact learned over the course of the experiment, thereby causing the measure of working memory to be partially a measure of participants' ability to learn arbitrary associations between visual stimuli and motor responses. Such learning was not highly specific to certain mappings; in sequences of short tasks, participants improved in their abilities to learn to represent items as actions in working memory. We discuss implications for working memory theories in light of hierarchical structure learning and ecological validity.

Energy Management Strategy for Fuel Cell/Battery/Ultracapacitor Hybrid Electric Vehicles Using Deep Reinforcement Learning With Action Trimming

As for fuel cell hybrid electric vehicle equipped with battery (BAT) and ultracapacitor (UC), its dynamic topology structure is complex and different characteristics of three power sources induce challenges in energy management for fuel economy, power sources lifespan, and dynamic performance of the vehicle. In this paper, an energy management strategy (EMS) based on a hierarchical power splitting structure and deep reinforcement learning (DRL) is proposed. In the higher layer strategy of the proposed EMS, the UC is employed to supply peak power and recover braking energy through the adaptive filter based on fuzzy control. Then, the integrated DRL and equivalent consumption minimization strategy framework is proposed to optimize the power allocation of fuel cell (FC) and BAT in the lower layer, to ensure the highly efficient operation of FC and reduce hydrogen consumption. And the action trimming based on heuristic technique is proposed to further restrain the adverse effect of sudden peak power on FC lifespan. The simulation results show the proposed EMS can make the output of FC smoother, improve its working efficiency to alleviate the stress of BAT, and increase by 14.8% compared with the Q-learning strategy in fuel economy under WLTP driving cycle. Meanwhile, the obtained results under UDDSHDV show fuel economy of the proposed EMS can reach dynamic programming (DP) benchmark level of 89.7 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\%$</tex-math></inline-formula> .

Edge-Assisted Democratized Learning Toward Federated Analytics

A recent take toward federated analytics (FA), which allows analytical insights of distributed data sets, reuses the federated learning (FL) infrastructure to evaluate the summary of model performances across the training devices. However, the current realization of FL adopts single server-multiple client architecture with limited scope for FA, which often results in learning models with poor generalization, i.e., an ability to handle new/unseen data, for real-world applications. Moreover, a hierarchical FL structure with distributed computing platforms demonstrates incoherent model performances at different aggregation levels. Therefore, we need to design a robust learning mechanism than the FL that 1) unleashes a viable infrastructure for FA and 2) trains learning models with better generalization capability. In this work, we adopt the novel democratized learning (Dem-AI) principles and designs to meet these objectives. First, we show the hierarchical learning structure of the proposed edge-assisted Dem-AI mechanism, namely <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Edge-DemLearn</i> , as a practical framework to empower generalization capability in support of FA. Second, we validate Edge-DemLearn as a flexible model training mechanism to build a distributed control and aggregation methodology in regions by leveraging the distributed computing infrastructure. The distributed edge computing servers construct regional models, minimize the communication loads, and ensure distributed data analytic application’s scalability. To that end, we adhere to a near-optimal two-sided many-to-one matching approach to handle the combinatorial constraints in Edge-DemLearn and solve it for fast knowledge acquisition with optimization of resource allocation and associations between multiple servers and devices. Extensive simulation results on real data sets demonstrate the effectiveness of the proposed methods.

Multilayer Batch Learning Growing Neural Gas for Learning Multiscale Topologies

Hierarchical topological structure learning methods are expected to be developed in the field of data mining for extracting multiscale topological structures from an unknown dataset. However, most methods require user-defined parameters, and it is difficult for users to determine these parameters and effectively utilize the method. In this paper, we propose a new parameter-less hierarchical topological structure learning method based on growing neural gas (GNG). First, we propose batch learning GNG (BL-GNG) to improve the learning convergence and reduce the user-designed parameters in GNG. BL-GNG uses an objective function based on fuzzy C-means to improve the learning convergence. Next, we propose multilayer BL-GNG (MBL-GNG), which is a parameter-less unsupervised learning algorithm based on hierarchical topological structure learning. In MBL-GNG, the input data of each layer uses parent nodes to learn more abstract topological structures from the dataset. Furthermore, MBL-GNG can automatically determine the number of nodes and layers according to the data distribution. Finally, we conducted several experiments to evaluate our proposed method by comparing it with other hierarchical approaches and discuss the effectiveness of our proposed method.

Residue-Residue Interaction Prediction via Stacked Meta-Learning.

Protein–protein interactions (PPIs) are the basis of most biological functions determined by residue–residue interactions (RRIs). Predicting residue pairs responsible for the interaction is crucial for understanding the cause of a disease and drug design. Computational approaches that considered inexpensive and faster solutions for RRI prediction have been widely used to predict protein interfaces for further analysis. This study presents RRI-Meta, an ensemble meta-learning-based method for RRI prediction. Its hierarchical learning structure comprises four base classifiers and one meta-classifier to integrate predictive strengths from different classifiers. It considers multiple feature types, including sequence-, structure-, and neighbor-based features, for characterizing other properties of a residue interaction environment to better distinguish between noninteracting and interacting residues. We conducted the same experiments using the same data as previously reported in the literature to demonstrate RRI-Meta’s performance. Experimental results show that RRI-Meta is superior to several current prediction tools. Additionally, to analyze the factors that affect the performance of RRI-Meta, we conducted a comparative case study using different protein complexes.

Cognitive structure learning model for hierarchical multi-label text classification

The human mind grows in learning new knowledge, which finally organizes and develops a basic mental pattern called cognitive structure. Hierarchical multi-label text classification (HMLTC), a fundamental but challenging task in many real-world applications, aims to classify the documents with hierarchical labels to form a resembling cognitive structure learning process. Existing approaches for HMLTC mainly focus on partial new knowledge learning or the global cognitive-structure-like label structure utilization in a cognitive view. However, the complete cognitive structure learning model is a unity that is indispensably constructed by the global label structure utilization and partial knowledge learning, which is ignored among those HMLTC approaches. To address this problem, we will imitate the cognitive structure learning process into the HMLTC learning and propose a unified framework called Hierarchical Cognitive Structure Learning Model (HCSM) in this paper. HCSM is composed of the Attentional Ordered Recurrent Neural Network (AORNN) submodule and Hierarchical Bi-Directional Capsule (HBiCaps) submodule. Both submodules utilize the partial new knowledge and global hierarchical label structure comprehensively for the HMLTC task. On the one hand, AORNN extracts the semantic vector as partial new knowledge from the original text by the word-level and hierarchy-level embedding granularities. On the other hand, AORNN builds the hierarchical text representation learning corresponding to the global label structure by the document-level neurons ordering. HBiCaps employs an iteration to form a unified label categorization process similar to cognitive-structure learning: firstly, using the probability computation of local hierarchical relationships to maintain partial knowledge learning; secondly, modifying the global hierarchical label structure based on the dynamic routing mechanism between capsules. Moreover, the experimental results on four benchmark datasets demonstrate that HCSM outperforms or matches state-of-the-art text classification methods.

Voxel-Based Scene Representation for Camera Pose Estimation of a Single RGB Image

Deep learning has been utilized in end-to-end camera pose estimation. To improve the performance, we introduce a camera pose estimation method based on a 2D-3D matching scheme with two convolutional neural networks (CNNs). The scene is divided into voxels, whose size and number are computed according to the scene volume and the number of 3D points. We extract inlier points from the 3D point set in a voxel using random sample consensus (RANSAC)-based plane fitting to obtain a set of interest points consisting of a major plane. These points are subsequently reprojected onto the image using the ground truth camera pose, following which a polygonal region is identified in each voxel using the convex hull. We designed a training dataset for 2D–3D matching, consisting of inlier 3D points, correspondence across image pairs, and the voxel regions in the image. We trained the hierarchical learning structure with two CNNs on the dataset architecture to detect the voxel regions and obtain the location/description of the interest points. Following successful 2D–3D matching, the camera pose was estimated using n-point pose solver in RANSAC. The experiment results show that our method can estimate the camera pose more precisely than previous end-to-end estimators.

An End-to-End Framework With Multisource Monitoring Data for Bridge Health Anomaly Identification

Bridge health assessment has been a challenging problem due to great evaluation errors caused by heterogeneous and different dimensional bridge factors. Most approaches employ monitoring data of one bridge factor to simplify the evaluation problem resulting in poor assessment performance. To address this issue, this article proposes an end-to-end framework to evaluate the health of bridges by exploring objective features and correlations of multiple monitoring factors. This model aims to learn representative features from raw monitoring data of bridge factors (i.e., strain, temperature, traffic flow, and heavy vehicle number) and classify the comprehensive features into different health degrees in a single framework. Especially, in terms of characteristics of bridge factors (e.g., time sequence and heterogeneity), a hierarchical learning structure with multiple convolutional, pooling, and dense layers is presented to learn the representations of the whole bridge monitoring data. The structure contributes to capturing rich information from different observed factors and improving feature learning ability through designing particular neural networks for each bridge factor in accordance with their corresponding data structure. In addition, a classification scheme with multiple fully connection layers and support vector machines in the framework is designed to achieve higher evaluation accuracy. Experiments on real-world monitoring data of a specific bridge validate the superiority of the proposed model.

Dissociable Neural Systems Support the Learning and Transfer of Hierarchical Control Structure.

Humans can draw insight from previous experiences to quickly adapt to novel environments that share a common underlying structure. Here we combine functional imaging and computational modeling to identify the neural systems that support the discovery and transfer of hierarchical task structure. Human subjects (male and female) completed multiple blocks of a reinforcement learning task that contained a global hierarchical structure governing stimulus-response action mapping. First, behavioral and computational evidence showed that humans successfully discover and transfer the hierarchical rule structure embedded within the task. Next, analysis of fMRI BOLD data revealed activity across a frontoparietal network that was specifically associated with the discovery of this embedded structure. Finally, activity throughout a cingulo-opercular network supported the transfer and implementation of this discovered structure. Together, these results reveal a division of labor in which dissociable neural systems support the learning and transfer of abstract control structures.SIGNIFICANCE STATEMENT A fundamental and defining feature of human behavior is the ability to generalize knowledge from the past to support future action. Although the neural circuits underlying more direct forms of learning have been well established over the last century, we still lack a solid framework from which to investigate more abstract, higher-order human learning and knowledge generalization. We designed a novel behavioral paradigm to specifically isolate a learning process in which previous knowledge, rather than directly indicating the correct action, instead guides the search for the correct action. Moreover, we identify that this learning process is achieved via the coordinated and temporally specific activity of two prominent cognitive control brain networks.

Rocorl: Transferable Reinforcement Learning-Based Robust Control for Cyber-Physical Systems With Limited Data Updates

Autonomous control systems are increasingly using machine learning technologies to process sensor data, making timely and informed decisions about performing control functions based on the data processing results. Among such machine learning technologies, reinforcement learning (RL) with deep neural networks has been recently recognized as one of the feasible solutions, since it enables learning by interaction with environments of control systems. In this paper, we consider RL-based control models and address the problem of temporally outdated observations often incurred in dynamic cyber-physical environments. The problem can hinder broad adoptions of RL methods for autonomous control systems. Specifically, we present an RL-based robust control model, namely rocorl , that exploits a hierarchical learning structure in which a set of low-level policy variants are trained for stale observations and then their learned knowledge can be transferred to a target environment limited in timely data updates. In doing so, we employ an autoencoder-based observation transfer scheme for systematically training a set of transferable control policies and an aggregated model-based learning scheme for data-efficiently training a high-level orchestrator in a hierarchy. Our experiments show that rocorl is robust against various conditions of distributed sensor data updates, compared with several other models including a state-of-the-art POMDP method.

Deep Cascade Model based Face Recognition: When Deep-layered Learning Meets Small Data.

Sparse representation based classification (SRC), nuclear-norm matrix regression (NMR), and deep learning (DL) have achieved a great success in face recognition (FR). However, there still exist some intrinsic limitations among them. SRC and NMR based coding methods belong to one-step model, such that the latent discriminative information of the coding error vector cannot be fully exploited. DL, as a multi-step model, can learn powerful representation, but relies on large-scale data and computation resources for numerous parameters training with complicated back-propagation. Straightforward training of deep neural networks from scratch on small-scale data is almost infeasible. Therefore, in order to develop efficient algorithms that are specifically adapted for small-scale data, we propose to derive the deep models of SRC and NMR. Specifically, in this paper, we propose an end-to-end deep cascade model (DCM) based on SRC and NMR with hierarchical learning, nonlinear transformation and multi-layer structure for corrupted face recognition. The contributions include four aspects. First, an end-to-end deep cascade model for small-scale data without back-propagation is proposed. Second, a multi-level pyramid structure is integrated for local feature representation. Third, for introducing nonlinear transformation in layer-wise learning, softmax vector coding of the errors with class discrimination is proposed. Fourth, the existing representation methods can be easily integrated into our DCM framework. Experiments on a number of small-scale benchmark FR datasets demonstrate the superiority of the proposed model over state-of-the-art counterparts. Additionally, a perspective that deep-layered learning does not have to be convolutional neural network with back-propagation optimization is consolidated. The demo code is available in https://github.com/liuji93/DCM.

&lt;b&gt;Hierarchical Structure Learning in a Bayesian Network for the Analysis of Purchasing Behavior&lt;/b&gt;

&lt;b&gt;Hierarchical Structure Learning in a Bayesian Network for the Analysis of Purchasing Behavior&lt;/b&gt;

Land Cover Classification from fused DSM and UAV Images Using Convolutional Neural Networks

In recent years, remote sensing researchers have investigated the use of different modalities (or combinations of modalities) for classification tasks. Such modalities can be extracted via a diverse range of sensors and images. Currently, there are no (or only a few) studies that have been done to increase the land cover classification accuracy via unmanned aerial vehicle (UAV)–digital surface model (DSM) fused datasets. Therefore, this study looks at improving the accuracy of these datasets by exploiting convolutional neural networks (CNNs). In this work, we focus on the fusion of DSM and UAV images for land use/land cover mapping via classification into seven classes: bare land, buildings, dense vegetation/trees, grassland, paved roads, shadows, and water bodies. Specifically, we investigated the effectiveness of the two datasets with the aim of inspecting whether the fused DSM yields remarkable outcomes for land cover classification. The datasets were: (i) only orthomosaic image data (Red, Green and Blue channel data), and (ii) a fusion of the orthomosaic image and DSM data, where the final classification was performed using a CNN. CNN, as a classification method, is promising due to hierarchical learning structure, regulating and weight sharing with respect to training data, generalization, optimization and parameters reduction, automatic feature extraction and robust discrimination ability with high performance. The experimental results show that a CNN trained on the fused dataset obtains better results with Kappa index of ~0.98, an average accuracy of 0.97 and final overall accuracy of 0.98. Comparing accuracies between the CNN with DSM result and the CNN without DSM result for the overall accuracy, average accuracy and Kappa index revealed an improvement of 1.2%, 1.8% and 1.5%, respectively. Accordingly, adding the heights of features such as buildings and trees improved the differentiation between vegetation specifically where plants were dense.

Multiscale Convolutional Neural Networks for Fault Diagnosis of Wind Turbine Gearbox

This paper proposes a novel intelligent fault diagnosis method to automatically identify different health conditions of wind turbine (WT) gearbox. Unlike traditional approaches, where feature extraction and classification are separately designed and performed, this paper aims to automatically learn effective fault features directly from raw vibration signals while classify the type of faults in a single framework, thus providing an end-to-end learning-based fault diagnosis system for WT gearbox without additional signal processing and diagnostic expertise. Considering the multiscale characteristics inherent in vibration signals of a gearbox, a new multiscale convolutional neural network (MSCNN) architecture is proposed to perform multiscale feature extraction and classification simultaneously. The proposed MSCNN incorporates multiscale learning into the traditional CNN architecture, which has two merits: 1) high-level fault features can be effectively learned by the hierarchical learning structure with multiple pairs of convolutional and pooling layers; and 2) multiscale learning scheme can capture complementary and rich diagnosis information at different scales. This greatly improves the feature learning ability and enables better diagnosis performance. The proposed MSCNN approach is evaluated through experiments on a WT gearbox test rig. Experimental results and comprehensive comparison analysis with respect to the traditional CNN and traditional multiscale feature extractors have demonstrated the superiority of the proposed method.

A Multiscale Feature Learning Scheme Based on Deep Learning for Industrial Process Monitoring and Fault Diagnosis

Considering that industrial data exhibit nonlinearity, high dimensionality and inherent multiscale characteristics, this paper proposes an intelligent industrial process monitoring and fault diagnosis method based on the discrete wavelet transform and deep learning. First, the discrete wavelet transform is used to present the multiscale representation of the raw data. Second, a multiscale convolutional neural network is used to extract the features at each scale, and then the extracted multiscale features are fused by the long short-term memory network to further reduce useless information and retain useful information. Finally, softmax classification is performed. The proposed method has two advantages: 1) the hierarchical learning structure with multiple pairs of convolutional and pooling layers can effectively learn nonlinear, high-dimensional fault features; and 2) the multiscale feature learning scheme can capture complementary diagnosis information at different scales. Detailed comparative studies between the proposed method and conventional methods have been carried out through the Tennessee Eastman benchmark process and the p-xylene oxidation reaction process.

Residual Recurrent Highway Networks for Learning Deep Sequence Prediction Models

A contemporary approach for acquiring the computational gains of depth in recurrent neural networks (RNNs) is to hierarchically stack multiple recurrent layers. However, such performance gains come with the cost of challenging optimization of hierarchal RNNs (HRNNs) which are deep both hierarchically and temporally. The researchers have exclusively highlighted the significance of using shortcuts for learning deep hierarchical representations and deep temporal dependencies. However, no significant efforts are made to unify these finding into a single framework for learning deep HRNNs. We propose residual recurrent highway network (R2HN) that contains highways within temporal structure of the network for unimpeded information propagation, thus alleviating gradient vanishing problem. The hierarchical structure learning is posed as residual learning framework to prevent performance degradation problem. The proposed R2HN contain significantly reduced data-dependent parameters as compared to related methods. The experiments on language modeling (LM) tasks have demonstrated that the proposed architecture leads to design effective models. On LM experiments with Penn TreeBank, the model achieved 60.3 perplexity and outperformed baseline and related models that we tested.