Local Receptive Fields Research Articles

Squeeze-and-Excitation Networks.

The central building block of convolutional neural networks (CNNs) is the convolution operator, which enables networks to construct informative features by fusing both spatial and channel-wise information within local receptive fields at each layer. A broad range of prior research has investigated the spatial component of this relationship, seeking to strengthen the representational power of a CNN by enhancing the quality of spatial encodings throughout its feature hierarchy. In this work, we focus instead on the channel relationship and propose a novel architectural unit, which we term the "Squeeze-and-Excitation" (SE) block, that adaptively recalibrates channel-wise feature responses by explicitly modelling interdependencies between channels. We show that these blocks can be stacked together to form SENet architectures that generalise extremely effectively across different datasets. We further demonstrate that SE blocks bring significant improvements in performance for existing state-of-the-art CNNs at slight additional computational cost. Squeeze-and-Excitation Networks formed the foundation of our ILSVRC 2017 classification submission which won first place and reduced the top-5 error to 2.251 percent, surpassing the winning entry of 2016 by a relative improvement of ∼25 percent. Models and code are available at https://github.com/hujie-frank/SENet.

Self-attention convolutional neural network for improved MR image reconstruction

MRI is an advanced imaging modality with the unfortunate disadvantage of long data acquisition time. To accelerate MR image acquisition while maintaining high image quality, extensive investigations have been conducted on image reconstruction of sparsely sampled MRI. Recently, deep convolutional neural networks have achieved promising results, yet the local receptive field in convolution neural network raises concerns regarding signal synthesis and artifact compensation. In this study, we proposed a deep learning-based reconstruction framework to provide improved image fidelity for accelerated MRI. We integrated the self-attention mechanism, which captured long-range dependencies across image regions, into a volumetric hierarchical deep residual convolutional neural network. Basically, a self-attention module was integrated to every convolutional layer, where signal at a position was calculated as a weighted sum of the features at all positions. Furthermore, relatively dense shortcut connections were employed, and data consistency was enforced. The proposed network, referred to as SAT-Net, was applied on cartilage MRI acquired using an ultrashort TE sequence and retrospectively undersampled in a pseudo-random Cartesian pattern. The network was trained using 336 three dimensional images (each containing 32 slices) and tested with 24 images that yielded improved outcome. The framework is generic and can be extended to various applications.

A Model of Motion Processing in the Visual Cortex Using Neural Field With Asymmetric Hebbian Learning

Neurons in the dorsal pathway of the visual cortex are thought to be involved in motion processing. The first site of motion processing is the primary visual cortex (V1), encoding the direction of motion in local receptive fields, with higher order motion processing happening in the middle temporal area (MT). Complex motion properties like optic flow are processed in higher cortical areas of the Medial Superior Temporal area (MST). In this study, a hierarchical neural field network model of motion processing is presented. The model architecture has an input layer followed by either one or cascade of two neural fields (NF): the first of these, NF1, represents V1, while the second, NF2, represents MT. A special feature of the model is that lateral connections used in the neural fields are trained by asymmetric Hebbian learning, imparting to the neural field the ability to process sequential information in motion stimuli. The model was trained using various traditional moving patterns such as bars, squares, gratings, plaids, and random dot stimulus. In the case of bar stimuli, the model had only a single NF, the neurons of which developed a direction map of the moving bar stimuli. Training a network with two NFs on moving square and moving plaids stimuli, we show that, while the neurons in NF1 respond to the direction of the component (such as gratings and edges) motion, the neurons in NF2 (analogous to MT) responding to the direction of the pattern (plaids, square object) motion. In the third study, a network with 2 NFs was simulated using random dot stimuli (RDS) with translational motion, and show that the NF2 neurons can encode the direction of the concurrent dot motion (also called translational flow motion), independent of the dot configuration. This translational RDS flow motion is decoded by a simple perceptron network (a layer above NF2) with an accuracy of 100% on train set and 90% on the test set, thereby demonstrating that the proposed network can generalize to new dot configurations. Also, the response properties of the model on different input stimuli closely resembled many of the known features of the neurons found in electrophysiological studies.

Deep convolutional neural networks for eigenvalue problems in mechanics

SummaryWe show that deep convolutional neural networks (CNNs) can massively outperform traditional densely connected neural networks (NNs) (both deep or shallow) in predicting eigenvalue problems in mechanics. In this sense, we strike out in a new direction in mechanics computations with strongly predictive NNs whose success depends not only on architectures being deep but also being fundamentally different from the widely used to date. We consider a model problem: predicting the eigenvalues of one‐dimensional (1D) and two‐dimensional (2D) phononic crystals. For the 1D case, the optimal CNN architecture reaches 98% accuracy level on unseen data when trained with just 20 000 samples, compared to 85% accuracy even with 100 000 samples for the typical network of choice in mechanics research. We show that, with relatively high data efficiency, CNNs have the capability to generalize well and automatically learn deep symmetry operations, easily extending to higher dimensions and our 2D case. Most importantly, we show how CNNs can naturally represent mechanical material tensors, with its convolution kernels serving as local receptive fields, which is a natural representation of mechanical response. Strategies proposed are applicable to other mechanics' problems and may, in the future, be used to sidestep cumbersome algorithms with purely data‐driven approaches based upon modern deep architectures.

Blind Image Quality Assessment With Joint Entropy Degradation

Blind image quality assessment (BIQA) aims to evaluate the quality of an image without pristine image objectively, which is highly desired in many perception-oriented image processing systems. Distortions degrade the visual contents and cause image quality degradation. Moreover, the visual contents of an image suffer from individual degradations by different types and different levels of distortions, which makes us difficult to analyze the quality degradation. From the perspective of information theory, there is a decrease in the amount of the visual contents when images are distorted. Therefore, in this paper, the image quality is assessed through its visual entropy degradation. Researches on the neuroscience indicate that the simple cells in the local receptive field can be characterized as being spatially localized and oriented, then the local intensity, gradient, and orientation features are extracted to represent the visual contents of an image. By deducing the joint entropy equation, the joint entropy is related to the statistical distributions. Next, in order to measure the visual entropy, the joint statistical distributions of those features are calculated. Finally, by measuring the degradations on these distributions of distorted images, a novel BIQA method is proposed. The experimental results on the databases of LIVE, CSIQ, and TID2013 show that the proposed method has superior performance than other state-of-the-art BIQA methods.

Unified No-Reference Quality Assessment of Singly and Multiply Distorted Stereoscopic Images.

A challenging problem in the no-reference quality assessment of multiply distorted stereoscopic images (MDSIs) is to simulate the monocular and binocular visual properties under a mixed type of distortions. Due to the joint effects of multiple distortions in MDSIs, the underlying monocular and binocular visual mechanisms have different manifestations with those of singly distorted stereoscopic images (SDSIs). This paper presents a unified no-reference quality evaluator for SDSIs and MDSIs by learning monocular and binocular local visual primitives (MB-LVPs). The main idea is to learn MB-LVPs to characterize the local receptive field properties of the visual cortex in response to SDSIs and MDSIs. Furthermore, we also consider that the learning of primitives should be performed in a task-driven manner. For this, two penalty terms including reconstruction error and quality inconsistency are jointly minimized within a supervised dictionary learning framework, generating a set of quality-oriented MB-LVPs for each single and multiple distortion modality. Given an input stereoscopic image, feature encoding is performed using the learned MB-LVPs as codebooks, resulting in the corresponding monocular and binocular responses. Finally, responses across all the modalities are fused with probabilistic weights which are determined by the modality-specific sparse reconstruction errors, yielding the final monocular and binocular features for quality regression. The superiority of our method has been verified on several SDSI and MDSI databases.

Modeling Pitch Perception With an Active Auditory Model Extended by Octopus Cells.

Pitch is an essential category for musical sensations. Models of pitch perception are vividly discussed up to date. Most of them rely on definitions of mathematical methods in the spectral or temporal domain. Our proposed pitch perception model is composed of an active auditory model extended by octopus cells. The active auditory model is the same as used in the Stimulation based on Auditory Modeling (SAM), a successful cochlear implant sound processing strategy extended here by modeling the functional behavior of the octopus cells in the ventral cochlear nucleus and by modeling their connections to the auditory nerve fibers (ANFs). The neurophysiological parameterization of the extended model is fully described in the time domain. The model is based on latency-phase en- and decoding as octopus cells are latency-phase rectifiers in their local receptive fields. Pitch is ubiquitously represented by cascaded firing sweeps of octopus cells. Based on the firing patterns of octopus cells, inter-spike interval histograms can be aggregated, in which the place of the global maximum is assumed to encode the pitch.

Eccentricity scale independence for scene perception in the first tens of milliseconds.

Visual processing of scenes in the first tens of milliseconds relies on global image summary statistics rather than localized processing. Although natural scenes typically involve our entire visual field, scenes are usually presented experimentally at limited eccentricity. Receptive-field size increases with foveal eccentricity while increasingly pooling activity from local receptive fields. Here, we asked to what extent an observer's performance on a scene-gist perception task depends on the contents of the scene as well as on the eccentricity of the scene. We manipulated the scene content by applying window and scotoma masks. In addition, we changed presentation eccentricity independent of image content by upscaling and downscaling the scenes. We find that discrimination is strongly affected when the scene is presented with a window of 5°, showing only the central part rather than the whole scene. Performance is, however, eccentricity scale independent provided that the same scene content is presented and a comparable area of the surface of primary visual cortex is activated. We furthermore show that this eccentricity scale independence holds for shorter presentation times, down to 17 ms in some scene-discrimination tasks, but not for the naturalness-discrimination task.

Estimation of Deterioration Levels of Transmission Towers via Deep Learning Maximizing Canonical Correlation Between Heterogeneous Features

This paper presents estimation of deterioration levels of transmission towers via deep learning maximizing the canonical correlation between heterogeneous features. In the proposed method, we newly construct a correlation-maximizing deep extreme learning machine (CMDELM) based on a local receptive field (LRF). For accurate deterioration level estimation, it is necessary to obtain semantic information that effectively represents deterioration levels. However, since the amount of training data for transmission towers is small, it is difficult to perform feature transformation by using many hidden layers such as general deep learning methods. In CMDELM-LRF, one hidden layer, which maximizes the canonical correlation between visual features and text features obtained from inspection text data, is newly inserted. Specifically, by using projections obtained by maximizing the canonical correlation as weight parameters of the hidden layer, feature transformation for extracting semantic information is realized without designing many hidden layers. This is the main contribution of this paper. Consequently, CMDELM-LRF realizes accurate deterioration level estimation from a small amount of training data.

Surface Material Recognition Using Active Multi-modal Extreme Learning Machine

Visual, haptic, and auditory modalities can provide different properties about surface materials and therefore form important perception methods for common material. Nevertheless, how to effectively combine various modalities is an extremely challenging problem. To this end, the active multi-modal framework based on extreme learning machine with multi-scale local receptive fields is developed to fuse the information of different modalities. We conduct multi-modal experiments on the TUM haptic texture database. The experimental results show the highly representative characteristics can be extracted from surface material by the proposed architecture and the three modalities fusion achieves the best performance. The proposed active multi-modal fusion framework shows significant performance improvements.

Multi-Cell Multi-Task Convolutional Neural Networks for Diabetic Retinopathy Grading.

Diabetic Retinopathy (DR) is a non-negligible eye disease among patients with Diabetes Mellitus, and automatic retinal image analysis algorithm for the DR screening is in high demand. Considering the resolution of retinal image is very high, where small pathological tissues can be detected only with large resolution image and large local receptive field are required to identify those late stage disease, but directly training a neural network with very deep architecture and high resolution image is both time computational expensive and difficult because of gradient vanishing/exploding problem, we propose a Multi-Cell architecture which gradually increases the depth of deep neural network and the resolution of input image, which both boosts the training time but also improves the classification accuracy. Further, considering the different stages of DR actually progress gradually, which means the labels of different stages are related. To considering the relationships of images with different stages, we propose a Multi-Task learning strategy which predicts the label with both classification and regression. Experimental results on the Kaggle dataset show that our method achieves a Kappa of 0.841 on test set which is the 4th rank of all state-of-the-arts methods. Further, our Multi-Cell Multi-Task Convolutional Neural Networks (M2CNN) solution is a general framework, which can be readily integrated with many other deep neural network architectures.

A memristive plasticity model of voltage-based STDP suitable for recurrent bidirectional neural networks in the hippocampus

Memristive systems have gained considerable attention in the field of neuromorphic engineering, because they allow the emulation of synaptic functionality in solid state nano-physical systems. In this study, we show that memristive behavior provides a broad working framework for the phenomenological modelling of cellular synaptic mechanisms. In particular, we seek to understand how close a memristive system can account for the biological realism. The basic characteristics of memristive systems, i.e. voltage and memory behavior, are used to derive a voltage-based plasticity rule. We show that this model is suitable to account for a variety of electrophysiology plasticity data. Furthermore, we incorporate the plasticity model into an all-to-all connecting network scheme. Motivated by the auto-associative CA3 network of the hippocampus, we show that the implemented network allows the discrimination and processing of mnemonic pattern information, i.e. the formation of functional bidirectional connections resulting in the formation of local receptive fields. Since the presented plasticity model can be applied to real memristive devices as well, the presented theoretical framework can support both, the design of appropriate memristive devices for neuromorphic computing and the development of complex neuromorphic networks, which account for the specific advantage of memristive devices.

Local receptive fields based extreme learning machine with hybrid filter kernels for image classification

In this paper, an innovative method called extreme learning machine with hybrid local receptive fields (ELM-HLRF) is presented for image classification. In this method, filters generated by Gabor functions and the randomly generated convolution filters are incorporated into the convolution filter kernels of local receptive fields based extreme learning machine (ELM-LRF). Extreme learning machine (ELM) is derived from single hidden layer feed-forward neural networks, and the parameters of its hidden layer can be generated randomly. As locally connected ELM, ELM-LRF directly processes information with strong correlations such as images and speech. In this paper, two main contributions are proposed to improve the classification performance of ELM-LRF. First, the Gabor functions are used as one kind of convolution filter kernels of ELM-HLRF to execute image classification. Second, we use a data augmentation method to preprocess training images to avoid overfitting. Experiments on the Outex texture dataset, the Yale face dataset, the ORL face database and the NORB dataset demonstrate that ELM-HLRF outperforms ELM-LRF, ELM and support vector machine in classification accuracy, and the presented data augmentation method improves the classification performance.

Local receptive field based extreme learning machine with three channels for histopathological image classification

The classification of histopathological images is a challenging task in the study of real-life medicine owing to the diverse geometrical structures and different histology features. This paper proposes a framework called Local Receptive Field based Extreme Learning Machine with Three Channels (3C-LRF-ELM), which can automatically extract histopathological features to diagnose whether there is a disease. We conduct experiments on the real-world image dataset that consists of mammalian lung, kidney and spleen organ images provided by the animal diagnostics lab (ADL) Pennsylvania State University. The training sets are consisted of overlapping blocks which are randomly extracted from arbitrary 40 images of each type image in the ADL dataset. The remaining images are equally divided into 850 blocks, and then they are given to the model 3C-LRF-ELM to generate the labels. The final label of each image is defined by the optimal threshold $$\alpha$$ . The 3C-LRF-ELM can be single layer network or multi-layer network. In this paper, considering the computational complexity, we choose the single layer 3C-LRF-ELM and two layers 3C-LRF-ELM structure to analyze the influence of the number of layers on the experimental results. The experimental results show that the single layer 3C-LRF-ELM structure is better than two layers 3C-LRF-ELM. Compared to the Discriminative Feature-oriented Dictionary Learning, the single layer 3C-LRF-ELM has a better classification performance.

Spatio-Temporal Graph Convolution for Skeleton Based Action Recognition

Variations of human body skeletons may be considered as dynamic graphs, which are generic data representation for numerous real-world applications. In this paper, we propose a spatio-temporal graph convolution (STGC) approach for assembling the successes of local convolutional filtering and sequence learning ability of autoregressive moving average. To encode dynamic graphs, the constructed multi-scale local graph convolution filters, consisting of matrices of local receptive fields and signal mappings, are recursively performed on structured graph data of temporal and spatial domain. The proposed model is generic and principled as it can be generalized into other dynamic models. We theoretically prove the stability of STGC and provide an upper-bound of the signal transformation to be learnt. Further, the proposed recursive model can be stacked into a multi-layer architecture. To evaluate our model, we conduct extensive experiments on four benchmark skeleton-based action datasets, including the large-scale challenging NTU RGB+D. The experimental results demonstrate the effectiveness of our proposed model and the improvement over the state-of-the-art.

Non-uniform weighting of local motion inputs underlies dendritic computation in the fly visual system

The fly visual system offers a unique opportunity to explore computations performed by single neurons. Two previous studies characterized, in vivo, the receptive field (RF) of the vertical system (VS) cells of the blowfly (calliphora vicina), both intracellularly in the axon, and, independently using Ca2+ imaging, in hundreds of distal dendritic branchlets. We integrated this information into detailed passive cable and compartmental models of 3D reconstructed VS cells. Within a given VS cell type, the transfer resistance (TR) from different branchlets to the axon differs substantially, suggesting that they contribute unequally to the shaping of the axonal RF. Weighting the local RFs of all dendritic branchlets by their respective TR yielded a faithful reproduction of the axonal RF. The model also predicted that the various dendritic branchlets are electrically decoupled from each other, thus acting as independent local functional subunits. The study suggests that single neurons in the fly visual system filter dendritic noise and compute the weighted average of their inputs.

Multi-Scale Shape Boltzmann Machine: A Shape Model Based on Deep Learning Method

Shape modelling is very important in many tasks of computer vision in the internet of things. Shape Boltzmann Machine (SBM) is a strong shape model, having ability to capture the details of object shape by introducing the Local Receptive Fields (LRF) and weight sharing into a deep learning architecture. However, applying LRF only in a single layer restrict its capabilities of learning more de-tails of object shape and representation of local shape parts. In this paper, we propose a new shape model based on Deep Boltzmann Machine (DBM) which we call Multi-Scale Shape Boltzmann Machine (MSSBM). By introducing weight sharing and LRF hierarchically in a deep architecture, MSSBM is capable of learning the true binary distributions of training shapes and generating more realistic shapes than the existing models, such as Deep Belief Network (DBN), DBM, SBM. Such capabilities make MSSBM suitable for many vision tasks, for example, image segmentation, object detection and inpainting, by enforcing shape prior knowledge. We demonstrate the performance of MSSBM through several experiments on three different datasets, in which exploitation of the details of shape structure is important for capturing the statistical variability of the underlying shape distributions. Experimental results show that MSSBM is a strong model for representing binary shapes that contains complex structure features.

Single and combined fault diagnosis of reciprocating compressor valves using a hybrid deep belief network

In this paper, a hybrid deep belief network is proposed to diagnose single and combined faults of suction and discharge valves in a reciprocating compressor. This hybrid integrates the deep belief network structured by multiple stacked restricted Boltzmann machines for pre-training and simplified fuzzy ARTMAP (SFAM) for fault classification. In the pre-training procedure, an algorithm for selecting local receptive fields is used to group the most similar features into the receptive fields of which top values are the units of each layer, and then restricted Boltzmann machine is applied to these units to construct a network. Unsupervised learning is also carried out for each restricted Boltzmann machine layer in this procedure to compute the network weights and biases. Finally, the network output is fed into SFAM to perform fault classification. In order to diagnose the valve faults, three signal types of vibration, pressure, and current are acquired from a two-stage reciprocating air compressor under different valve conditions such as suction leakages, discharge leakages, spring deterioration, and their combination. These signals are subsequently processed so that the useful fault information from the signals can be revealed; next, statistical features in the time and frequency domains are extracted from the signals and used as the inputs for hybrid deep belief network. Performance of hybrid deep belief network in fault classification is compared with that of the original deep belief network and the deep belief network combined with generalized discriminant analysis, where softmax regression is used as a classifier for the latter two models. The results indicate that hybrid deep belief network is more capable of improving the diagnosis accuracy and is feasible in industrial applications.

Haptic recognition using hierarchical extreme learning machine with local-receptive-field

In order to perform useful tasks in people’s daily life, robots must be able to both communicate and understand the sensations they experience and may need to know the haptic properties of an object before touching it. To enable better tactile understanding for robots, we propose an effective hierarchical extreme learning machine with local-receptive-field architecture, while introducing the local receptive field concept in neuroscience and maintaining ELM’s advantages of training efficiency. In this paper, we further extend the LRF-based ELM method to a hierarchical model for haptic classification. Experimental validation on the Penn Haptic Adjective Corpus 2 dataset illustrates that the proposed hierarchical method achieves better recognition performance.

Spike Burst Coding of Translatory Optic Flow and Depth from Motion in the Fly Visual System

Many animals use the visual motion generated by traveling straight-the translatory optic flow-to successfully navigate obstacles: near objects appear larger and to move more quickly than distant objects. Flies are expert at navigating cluttered environments, and while their visual processing of rotatory optic flow is understood in exquisite detail, how they process translatory optic flow remains a mystery. We present novel cell types that have local motion receptive fields matched to translation self-motion, the vertical translation (VT) cells. One of these, the VT1 cell, encodes self-motion in the forward-sideslip direction and fires action potentials in spike bursts as well as single spikes. We show that the spike burst coding is size and speed-tuned and is selectively modulated by motion parallax-the relative motion experienced during translation. These properties are spatially organized, so that the cell is most excited by clutter rather than isolated objects. When the fly is presented with a simulation of flying past an elevated object, the spike burst activity is modulated by the height of the object, and the rate of single spikes is unaffected. When the moving object alone is experienced, the cell is weakly driven. Meanwhile, the VT2-3 cells have motion receptive fields matched to the lift axis. In conjunction with previously described horizontal cells, the VT cells have properties well suited to the visual navigation of clutter and to encode the fly's movements along near cardinal axes of thrust, lift, and forward sideslip.