Layerwise Representation Research Articles

Deep Hierarchical Multimodal Metric Learning.

Multimodal metric learning aims to transform heterogeneous data into a common subspace where cross-modal similarity computing can be directly performed and has received much attention in recent years. Typically, the existing methods are designed for nonhierarchical labeled data. Such methods fail to exploit the intercategory correlations in the label hierarchy and, therefore, cannot achieve optimal performance on hierarchical labeled data. To address this problem, we propose a novel metric learning method for hierarchical labeled multimodal data, named deep hierarchical multimodal metric learning (DHMML). It learns the multilayer representations for each modality by establishing a layer-specific network corresponding to each layer in the label hierarchy. In particular, a multilayer classification mechanism is introduced to enable the layerwise representations to not only preserve the semantic similarities within each layer, but also retain the intercategory correlations across different layers. In addition, an adversarial learning mechanism is proposed to bridge the cross-modality gap by producing indistinguishable features for different modalities. Through integration of the multilayer classification and adversarial learning mechanisms, DHMML can obtain hierarchical discriminative modality-invariant representations for multimodal data. Experiments on two benchmark datasets are used to demonstrate the superiority of the proposed DHMML method over several state-of-the-art methods.

What Do Self-Supervised Speech Models Know About Words?

Abstract Many self-supervised speech models (S3Ms) have been introduced over the last few years, improving performance and data efficiency on various speech tasks. However, these empirical successes alone do not give a complete picture of what is learned during pre-training. Recent work has begun analyzing how S3Ms encode certain properties, such as phonetic and speaker information, but we still lack a proper understanding of knowledge encoded at the word level and beyond. In this work, we use lightweight analysis methods to study segment-level linguistic properties—word identity, boundaries, pronunciation, syntactic features, and semantic features—encoded in S3Ms. We present a comparative study of layer-wise representations from ten S3Ms and find that (i) the frame-level representations within each word segment are not all equally informative, and (ii) the pre-training objective and model size heavily influence the accessibility and distribution of linguistic information across layers. We also find that on several tasks—word discrimination, word segmentation, and semantic sentence similarity—S3Ms trained with visual grounding outperform their speech-only counterparts. Finally, our task-based analyses demonstrate improved performance on word segmentation and acoustic word discrimination while using simpler methods than prior work.1

Layer-Wise Representation Fusion for Compositional Generalization

Existing neural models are demonstrated to struggle with compositional generalization (CG), i.e., the ability to systematically generalize to unseen compositions of seen components. A key reason for failure on CG is that the syntactic and semantic representations of sequences in both the uppermost layer of the encoder and decoder are entangled. However, previous work concentrates on separating the learning of syntax and semantics instead of exploring the reasons behind the representation entanglement (RE) problem to solve it. We explain why it exists by analyzing the representation evolving mechanism from the bottom to the top of the Transformer layers. We find that the ``shallow'' residual connections within each layer fail to fuse previous layers' information effectively, leading to information forgetting between layers and further the RE problems. Inspired by this, we propose LRF, a novel Layer-wise Representation Fusion framework for CG, which learns to fuse previous layers' information back into the encoding and decoding process effectively through introducing a fuse-attention module at each encoder and decoder layer. LRF achieves promising results on two realistic benchmarks, empirically demonstrating the effectiveness of our proposal. Codes are available at https://github.com/thinkaboutzero/LRF.

How deep convolutional neural networks lose spatial information with training

A central question of machine learning is how deep nets manage to learn tasks in high dimensions. An appealing hypothesis is that they achieve this feat by building a representation of the data where information irrelevant to the task is lost. For image datasets, this view is supported by the observation that after (and not before) training, the neural representation becomes less and less sensitive to diffeomorphisms acting on images as the signal propagates through the network. This loss of sensitivity correlates with performance and surprisingly correlates with a gain of sensitivity to white noise acquired during training. Which are the mechanisms learned by convolutional neural networks (CNNs) responsible for the these phenomena? In particular, why is the sensitivity to noise heightened with training? Our approach consists of two steps. (1) Analyzing the layer-wise representations of trained CNNs, we disentangle the role of spatial pooling in contrast to channel pooling in decreasing their sensitivity to image diffeomorphisms while increasing their sensitivity to noise. (2) We introduce model scale-detection tasks, which qualitatively reproduce the phenomena reported in our empirical analysis. In these models we can assess quantitatively how spatial pooling affects these sensitivities. We find that the increased sensitivity to noise observed in deep ReLU networks is a mechanistic consequence of the perturbing noise piling up during spatial pooling, after being rectified by ReLU units. Using odd activation functions like tanh drastically reduces the CNNs’ sensitivity to noise.

On Robustness and Sensitivity of a Neural Language Model: A Case Study on Italian L1 Learner Errors

In this paper, we propose a comprehensive linguistic study aimed at assessing the implicit behavior of one of the most prominent Neural Language Models (NLM) based on Transformer architectures, BERT Devlin et al., when dealing with a particular source of noisy data, namely essays written by L1 Italian learners containing a variety of errors targeting grammar, orthography and lexicon. Differently from previous works, we focus on the pre-training stage and we devise two complementary evaluation tasks aimed at assessing the impact of errors on sentence-level inner representations in terms of semantic robustness and linguistic sensitivity. While the first evaluation perspective is meant to probe the model's ability to encode the semantic similarity between sentences also in the presence of errors, the second type of probing task evaluates the influence of errors on BERT's implicit knowledge of a set of raw and morpho-syntactic properties of a sentence. Our experiments show that BERT's ability to compute sentence similarity and to correctly encode multi-leveled linguistic information of a sentence are differently modulated by the category of errors and that the error hierarchies in terms of robustness and sensitivity change across layer-wise representations.

Locality preserving dense graph convolutional networks with graph context-aware node representations

Graph convolutional networks (GCNs) have been widely used for representation learning on graph data, which can capture structural patterns on a graph via specifically designed convolution and readout operations. In many graph classification applications, GCN-based approaches have outperformed traditional methods. However, most of the existing GCNs are inefficient to preserve local information of graphs — a limitation that is especially problematic for graph classification. In this work, we propose a locality-preserving dense GCN with graph context-aware node representations. Specifically, our proposed model incorporates a local node feature reconstruction module to preserve initial node features into node representations, which is realized via a simple but effective encoder–decoder mechanism. To capture local structural patterns in neighborhoods representing different ranges of locality, dense connectivity is introduced to connect each convolutional layer and its corresponding readout with all previous convolutional layers. To enhance node representativeness, the output of each convolutional layer is concatenated with the output of the previous layer’s readout to form a global context-aware node representation. In addition, a self-attention module is introduced to aggregate layer-wise representations to form the final graph-level representation. Experiments on benchmark datasets demonstrate the superiority of the proposed model over state-of-the-art methods in terms of classification accuracy.

Representational Content of Oscillatory Brain Activity during Object Recognition: Contrasting Cortical and Deep Neural Network Hierarchies.

Numerous theories propose a key role for brain oscillations in visual perception. Most of these theories postulate that sensory information is encoded in specific oscillatory components (e.g., power or phase) of specific frequency bands. These theories are often tested with whole-brain recording methods of low spatial resolution (EEG or MEG), or depth recordings that provide a local, incomplete view of the brain. Opportunities to bridge the gap between local neural populations and whole-brain signals are rare. Here, using representational similarity analysis (RSA) in human participants we explore which MEG oscillatory components (power and phase, across various frequency bands) correspond to low or high-level visual object representations, using brain representations from fMRI, or layer-wise representations in seven recent deep neural networks (DNNs), as a template for low/high-level object representations. The results showed that around stimulus onset and offset, most transient oscillatory signals correlated with low-level brain patterns (V1). During stimulus presentation, sustained β (∼20 Hz) and γ (>60 Hz) power best correlated with V1, while oscillatory phase components correlated with IT representations. Surprisingly, this pattern of results did not always correspond to low-level or high-level DNN layer activity. In particular, sustained β band oscillatory power reflected high-level DNN layers, suggestive of a feed-back component. These results begin to bridge the gap between whole-brain oscillatory signals and object representations supported by local neuronal activations.

Semantic segmentation of ferrography images for automatic wear particle analysis

Automatic wear particle detection and classification has remained a high priority research area for wear condition monitoring and failure analysis. In this study, a deep convolutional neural network (DCNN) with three modules, namely, an encoder, atrous spatial pyramid pooling (ASPP), and a decoder, is constructed. Instead of using handcrafted features, the DCNN can automatically learn features through a layer-wise representation and realize semantic segmentation, i.e., segmentation and identification concurrently, of five types of wear particles in ferrograph images using end-to-end processing. Experimental results show that the DCNN achieves 82.5% accuracy. This proposed method unifies the segmentation, classification, and edge location of the wear particles into a single model, avoids the accumulation and transmission of errors caused by numerous steps applied in a traditional linear process, and improves the efficiency and accuracy of the wear particle analysis.

A Multiscale Dual-Branch Feature Fusion and Attention Network for Hyperspectral Images Classification

Recently, hyperspectral image classification based on deep learning has achieved considerable attention. Many convolutional neural network classification methods have emerged and exhibited superior classification performance. However, most methods focus on extracting features by using fixed convolution kernels and layer-wise representation, resulting in feature extraction singleness. Additionally, the feature fusion process is rough and simple. Numerous methods get accustomed to fusing different levels of features by stacking modules hierarchically, which ignore the combination of shallow and deep spectral-spatial features. In order to overcome the preceding issues, a novel multiscale dual-branch feature fusion and attention network is proposed. Specifically, we design a multiscale feature extraction (MSFE) module to extract spatial-spectral features at a granular level and expand the range of receptive fields, thereby enhancing the MSFE ability. Subsequently, we develop a dual-branch feature fusion interactive module that integrates the residual connection's feature reuse property and the dense connection's feature exploration capability, obtaining more discriminative features in both spatial and spectral branches. Additionally, we introduce a novel shuffle attention mechanism that allows for adaptive weighting of spatial and spectral features, further improving classification performance. Experimental results on three benchmark datasets demonstrate that our model outperforms other state-of-the-art methods while incurring the lower computational cost.

Stacked Marginal Time Warping for Temporal Alignment

Time warping is the popular technique of temporally aligning two sequences and has successfully applied in temporal alignment tasks such as activity recognition. However, existing time warping methods suffer from limited representation ability because aligning process is performed on either raw sequences or the projected lower-dimensional features. In this paper, we propose a stacked time warping framework (STW) to learn layer-wise representation for temporal alignment in a stacked structure. By using this structure, STW gives higher flexibility than existing methods meanwhile unifies them into a deep architecture. Based on the proposed STW framework, we explore a stacked marginal time warping (SMTW) method by using marginal stacked denoising autoencoder (mSDA) as the regularization term which enables SMTW to marginalize out noises and learn layer-wise non-linear representations with the effective closed-form solution. Benefitting from the incorporation of mSDA, SMTW achieves better alignment performance and keeps comparable time efficiency with regular time warping methods. Experiments on both synthetic data and practical human activity recognition datasets demonstrate that SMTW is superior to the state-of-the-art time warping methods in quantity.

Fuzzy Modal Control Applied to Smart Composite Structure

This paper proposes an active vibration control technique, which is based on Fuzzy Modal Control, as applied to a piezoelectric actuator bonded to a composite structure forming a so-called smart composite structure. Fuzzy Modal Controllers were found to be well adapted for controlling structures with nonlinear behavior, whose characteristics change considerably with respect to time. The smart composite structure was modelled by using a so called mixed theory. This theory uses a single equivalent layer for the discretization of the mechanical displacement field and a layerwise representation of the electrical field. Temperature effects are neglected. Due to numerical reasons it was necessary to reduce the size of the model of the smart composite structure so that the design of the controllers and the estimator could be performed. The role of the Kalman Estimator in the present contribution is to estimate the modal states of the system, which are used by the Fuzzy Modal controllers. Simulation results illustrate the effectiveness of the proposed vibration control methodology for composite structures.

Robust Optimal Control Applied to a Composite Laminated Beam

This paper proposes an active vibration control technique, which is based on linear matrix inequalities, that is applied to a piezoelectric actuator bonded to a composite structure, forming a so-called smart composite structure. Serendipity-type finite element, based on first-order shear deformation theory with rectangular shape, eight nodes, five mechanical degrees of freedom (DOF) per node and eight electrical DOF per piezoelectric layer, is established for the composite structural model. Additionally, a mixed theory that uses a single equivalent layer for the discretization of the mechanical displacement field and a layerwise representation of the electrical field is adopted. Temperature effects are neglected. Simulation results illustrate the effectiveness of the proposed vibration control methodology for composite structures.

Coupled refined layerwise theory for dynamic free and forced response of piezoelectric laminated composite and sandwich beams

This work extends a previously presented coupled refined layerwise theory to dynamic analysis of piezoelectric laminated composite and sandwich beams. Contrary to most of the available theories, all the kinematic and stress boundary conditions are satisfied at the interfaces of the piezoelectric layers with the non-zero longitudinal electric field. Moreover, both electrical transverse normal strains and transverse flexibility are taken into account for the first time in the present theory. In the presented formulation a high-order polynomial, an exponential expression and a layerwise term containing the electric field are included in the describing expression of the in-plane displacement of the beam. For the transverse displacement, the coupled refined model uses a combination of continuous piecewise fourth-order polynomials with a layerwise representation of electrical unknowns. The electric field is also approximated as linear across the thickness direction of piezoelectric layers. One of advantages of the present theory is that the mechanical number of the unknown parameters is very small and is independent of the number of the layers. For validation of the proposed model, various free and forced vibration tests for thin and thick laminated/sandwich piezoelectric beams are carried out. For various electrical and mechanical boundary conditions, excellent correlation has been found between the results obtained from the proposed formulation with those resulted from the three-dimensional theory of piezoelasticity.

A coupled refined high-order global-local theory and finite element model for static electromechanical response of smart multilayered/sandwich beams

In the present study, a coupled refined high-order global-local theory is developed for predicting fully coupled behavior of smart multilayered/sandwich beams under electromechanical conditions. The proposed theory considers effects of transverse normal stress and transverse flexibility which is important for beams including soft cores or beams with drastic material properties changes through depth. Effects of induced transverse normal strains through the piezoelectric layers are also included in this study. In the presence of non-zero in-plane electric field component, all the kinematic and stress continuity conditions are satisfied at layer interfaces. In addition, for the first time, conditions of non-zero shear and normal tractions are satisfied even while the bottom or the top layer of the beam is piezoelectric. A combination of polynomial and exponential expressions with a layerwise term containing first order differentiation of electrical unknowns is used to introduce the in-plane displacement field. Also, the transverse displacement field is formulated utilizing a combination of continuous piecewise fourth-order polynomial with a layerwise representation of electrical unknowns. Finally, a quadratic electric potential is used across the thickness of each piezoelectric layer. It is worthy to note that in the proposed shear locking-free finite element formulation, the number of mechanical unknowns is independent of the number of layers. Excellent correlation has been found between the results obtained from the proposed formulation for thin and thick piezoelectric beams with those resulted from the three-dimensional theory of piezoelasticity. Moreover, the proposed finite element model is computationally economic.

Modelling and optimization of laminated adaptive shells of revolution

In this paper two shell finite element models are presented for the structural analysis of composite laminated piezoelectric shells. One is an axisymmetric conical frustum with two nodal rings and the other is a conic shell panel with eight nodes. Both models are based in a mixed laminated theory that combines a higher order shear deformation theory for the mechanical displacement field with a layerwise representation with linear functions for the electric potential through each piezoelectric layer. In order to obtain the optimal design sensitivities analysis and optimization techniques based in the nonlinear mathematical programming are used. The design objectives can be the minimization of the deformed structure or the maximization of the natural fundamental frequency and the design variables are the electric potential difference applied to the actuators or the ply thicknesses among others.

Analysis of adaptive shell structures using a refined laminated model

This work presents the development of a shell conical panel finite element model, which has the possibility of having embedded piezoelectric actuators and/or sensors patches. A mixed laminated theory is used, which combines an equivalent single layer higher order shear deformation approach for the mechanical behavior with a layerwise representation in the thickness direction to describe the distribution of the electric potential in each of the piezoelectric layers of the finite element. The electrical potential function is represented through a linear variation across the thickness with two electric potential nodes for each piezoelectric layer. Based in this model an active damping scheme applied to laminated shell structures is presented and discussed.

An efficient coupled theory for multilayered beams with embedded piezoelectric sensory and active layers

An efficient coupled electromechanical model is developed for multilayered composite beams with embedded or surface bonded piezoelectric laminae subjected to static electromechanical excitation. The model combines third order zigzag approximations for the displacement field with a layerwise representation of the electric field. Interfacial continuity of the inplane displacement and the transverse shear stress and traction free conditions on the top and bottom surfaces are ensured under general electromechanical loading situation. The model allows for a non-uniform variation of transverse displacement in the piezoelectric layers caused by the electric field induced normal transverse strain. The theory has the same number of primary variables as first order theory and hence the computational cost is independent of the number of layers in the laminate. The governing equations of stress and charge equilibrium and the variationally consistent boundary conditions are derived from the principle of virtual work. To illustrate the accuracy, applicability and robustness of the theory, an analytical solution is obtained for hybrid beams with simply supported ends. Present results for simply supported hybrid beams with sensory and actuated piezoelectric layers are compared with the exact three dimensional solution and uncoupled first order theory solution. The present results show significant improvement over the first order solution and compare very well with the exact solution for both thin and thick piezoelectric laminated beams. Capability of the developed theory to model sensory, active and combined response of smart composite beams with general laminate configurations has been demonstrated through additional numerical examples. Feasibility of controlling deflection by applying appropriate actuation potential has been illustrated.

Mixed Laminate Theory and Finite Element for Smart Piezoelectric Composite Shell Structures

Mechanicsfortheanalysisoflaminatedcompositeshellswithpiezoelectricactuatorsandsensorsarepresented.A newmixedlaminatetheoryforpiezoelectricshellsisdevelopedincurvilinearcoordinatesthatcombinessingle-layer assumptions for the displacements and a layerwise representation for the electric potential. The resultant coupled governing equations for curvilinear piezoelectric laminates are described. Structural mechanics are subsequently developed and an eight-node ® nite element is formulated for the static and dynamic analysis of adaptivecomposite shell structures of general laminationscontaining piezoelectriclayers. Evaluations of themethod and comparisons with reported results were performed. Numerical results for cylindrical laminated piezoelectric composite panels with continuous piezoceramic actuators and cantilever shells with continuous or discrete piezoelectric actuators and sensors illustrate the advantages of the method and quantify the effects of curvature on the electromechanical response of piezoelectric shells.

Layerwise mechanics and finite element for the dynamic analysis of piezoelectric composite plates

Laminate and structural mechanics for the analysis of laminated composite plate structures with piezoelectric actuators and sensors are presented. The theories implement layerwise representations of displacements and electric potential, and can model both the global and local electromechanical response of smart composite laminates. Finite-element formulations are developed for the quasi-static and dynamic analysis of smart composite structures containing piezoelectric layers. Comparisons with an exact solution illustrate the accuracy, robustness and capability of the developed mechanics to capture the global and local response of thin and/or thick laminated piezoelectric plates. Additional correlations and numerical applications demonstrate the unique capabilities of the mechanics in analyzing the static and free-vibration response of composite plates with distributed piezoelectric actuators and sensors.

Composite Beam Element with Layerwise Plane Sections

Based on generalized laminate plate theory (GLPT), the formulation of a one-dimensional laminated beam finite element with layerwise constant shear (BLCS) is presented. BLCS formulation is equivalent to a first-order shear deformation beam theory (Timoshenko beam theory) on each layer, and a cross section of the beam therefore does not necessarily remain plane through the laminate, but only through each layer. Plane stress is assumed through both the thickness and width of the beam in the constitutive equation for a lamina. Details are presented for transforming the layerwise constant shear stresses obtained from constitutive relations into parabolic shear stress distributions. The layerwise representation of in-plane displacement through the thickness results in the formulation of a relatively simple beam element. Numerical analyses are presented for a three-node BLCS element integrated with two Gauss points. The accuracy of the element is evaluated by comparing the predictions to elasticity and experimental results.