Hierarchical Neural Network Architecture Research Articles

A Hybrid 3D–2D Feature Hierarchy CNN with Focal Loss for Hyperspectral Image Classification

Hyperspectral image (HSI) classification has been extensively applied for analyzing remotely sensed images. HSI data consist of multiple bands that provide abundant spatial information. Convolutional neural networks (CNNs) have emerged as powerful deep learning methods for processing visual data. In recent work, CNNs have shown impressive results in HSI classification. In this paper, we propose a hierarchical neural network architecture called feature extraction with hybrid spectral CNN (FE-HybridSN) to extract superior spectral–spatial features. FE-HybridSN effectively captures more spectral–spatial information while reducing computational complexity. Considering the prevalent issue of class imbalance in experimental datasets (IP, UP, SV) and real-world hyperspectral datasets, we apply the focal loss to mitigate these problems. The focal loss reconstructs the loss function and facilitates effective achievement of the aforementioned goals. We propose a framework (FEHN-FL) that combines FE-HybridSN and the focal loss for HSI classification and then conduct extensive HSI classification experiments using three remote sensing datasets: Indian Pines (IP), University of Pavia (UP), and Salinas Scene (SV). Using cross-entropy loss as a baseline, we assess the hyperspectral classification performance of various backbone networks and examine the influence of different spatial sizes on classification accuracy. After incorporating focal loss as our loss function, we not only compare the classification performance of the FE-HybridSN backbone network under different loss functions but also evaluate their convergence rates during training. The proposed classification framework demonstrates satisfactory performance compared to state-of-the-art end-to-end deep-learning-based methods, such as 2D-CNN, 3D-CNN, etc.

Hierarchical Graph Pattern Understanding for Zero-Shot Video Object Segmentation.

The optical flow guidance strategy is ideal for obtaining motion information of objects in the video. It is widely utilized in video segmentation tasks. However, existing optical flow-based methods have a significant dependency on optical flow, which results in poor performance when the optical flow estimation fails for a particular scene. The temporal consistency provided by the optical flow could be effectively supplemented by modeling in a structural form. This paper proposes a new hierarchical graph neural network (GNN) architecture, dubbed hierarchical graph pattern understanding (HGPU), for zero-shot video object segmentation (ZS-VOS). Inspired by the strong ability of GNNs in capturing structural relations, HGPU innovatively leverages motion cues (i.e., optical flow) to enhance the high-order representations from the neighbors of target frames. Specifically, a hierarchical graph pattern encoder with message aggregation is introduced to acquire different levels of motion and appearance features in a sequential manner. Furthermore, a decoder is designed for hierarchically parsing and understanding the transformed multi-modal contexts to achieve more accurate and robust results. HGPU achieves state-of-the-art performance on four publicly available benchmarks (DAVIS-16, YouTube-Objects, Long-Videos and DAVIS-17). Code and pre-trained model can be found at https://github.com/NUST-Machine-Intelligence-Laboratory/HGPU.

Distributed Machine Learning Based Downlink Channel Estimation for RIS Assisted Wireless Communications

The downlink channel estimation requires a huge pilot overhead in the reconfigurable intelligent surface (RIS) assisted communication system. By exploiting the powerful learning ability of the neural network, the machine learning (ML) technique can be used to estimate the high-dimensional channel from a few received pilot signals at the user. However, since the training dataset collected by the single user only contains the information of part of the channel scenarios of a cell, the neural network trained by the single user is not able to work when the user moves from one channel scenario to another. To solve this challenge, we propose to leverage the distributed machine learning (DML) technique to enable the reliable downlink channel estimation. Specifically, we firstly build a downlink channel estimation neural network shared by all users, which can be collaboratively trained by the BS and the users with the help of the DML technique. Then, we further propose a hierarchical neural network architecture to improve the channel estimation accuracy, which can extract different channel features for different channel scenarios. Simulation results show that compared with the neural network trained by the single user, the proposed DML based neural networks can achieve better estimation performance with the reduced pilot overhead for all users from different scenarios.

DrABC: deep learning accurately predicts germline pathogenic mutation status in breast cancer patients based on phenotype data

BackgroundIdentifying breast cancer patients with DNA repair pathway-related germline pathogenic variants (GPVs) is important for effectively employing systemic treatment strategies and risk-reducing interventions. However, current criteria and risk prediction models for prioritizing genetic testing among breast cancer patients do not meet the demands of clinical practice due to insufficient accuracy.MethodsThe study population comprised 3041 breast cancer patients enrolled from seven hospitals between October 2017 and 11 August 2019, who underwent germline genetic testing of 50 cancer predisposition genes (CPGs). Associations among GPVs in different CPGs and endophenotypes were evaluated using a case-control analysis. A phenotype-based GPV risk prediction model named DNA-repair Associated Breast Cancer (DrABC) was developed based on hierarchical neural network architecture and validated in an independent multicenter cohort. The predictive performance of DrABC was compared with currently used models including BRCAPRO, BOADICEA, Myriad, PENN II, and the NCCN criteria.ResultsIn total, 332 (11.3%) patients harbored GPVs in CPGs, including 134 (4.6%) in BRCA2, 131 (4.5%) in BRCA1, 33 (1.1%) in PALB2, and 37 (1.3%) in other CPGs. GPVs in CPGs were associated with distinct endophenotypes including the age at diagnosis, cancer history, family cancer history, and pathological characteristics. We developed a DrABC model to predict the risk of GPV carrier status in BRCA1/2 and other important CPGs. In predicting GPVs in BRCA1/2, the performance of DrABC (AUC = 0.79 [95% CI, 0.74–0.85], sensitivity = 82.1%, specificity = 63.1% in the independent validation cohort) was better than that of previous models (AUC range = 0.57–0.70). In predicting GPVs in any CPG, DrABC (AUC = 0.74 [95% CI, 0.69–0.79], sensitivity = 83.8%, specificity = 51.3% in the independent validation cohort) was also superior to previous models in their current versions (AUC range = 0.55–0.65). After training these previous models with the Chinese-specific dataset, DrABC still outperformed all other methods except for BOADICEA, which was the only previous model with the inclusion of pathological features. The DrABC model also showed higher sensitivity and specificity than the NCCN criteria in the multi-center validation cohort (83.8% and 51.3% vs. 78.8% and 31.2%, respectively, in predicting GPVs in any CPG). The DrABC model implementation is available online at http://gifts.bio-data.cn/.ConclusionsBy considering the distinct endophenotypes associated with different CPGs in breast cancer patients, a phenotype-driven prediction model based on hierarchical neural network architecture was created for identification of hereditary breast cancer. The model achieved superior performance in identifying GPV carriers among Chinese breast cancer patients.

A hierarchical auxiliary deep neural network architecture for large-scale indoor localization based on Wi-Fi fingerprinting

Conventional application of deep neural networks (DNNs) to multi-building and multi-floor indoor localization is based on pure regression of three-dimensional location coordinates (e.g., longitude, latitude and altitude (i.e., floor height)), classification of location labels (e.g., building, floor and room information), or hybrid classification/regression of labels and coordinates (e.g., building and floor information and two-dimensional location coordinates), which, however, does not take into account an innate hierarchical auxiliary information (e.g., building-¿floor-¿location) of indoor localization data. Such conventional application of DNNs faces scalability issues in case of large-scale indoor localization where the numbers of buildings and floors are large. Inserting classification tasks as auxiliary networks into a regression neural network, we propose a new framework called a hierarchical auxiliary deep neural network (HADNN), which not only address the scalability issues with an increasing number of classes but also could further reduce the hierarchical information error. In HADNN, hierarchical auxiliary information of given data are provided and used during the training phase. As there are two possible hierarchical information cases in indoor localization data: (1) given only floors and (2) given both buildings and floors, we propose two architectures: one utilizing only floor information and the other taking both building and floor information. At test phase, HADNN predicts building, floor and location coordinate at the same time. Experimental results show that the architecture of HADNN achieves better performance of a coordinate regression task and require a smaller number of parameters than the pure two-dimensional location coordinates regression model. In addition, HADNN does not require the training data and coarse classes (e.g., building and floor information) at test phase while previous methods still require the training data to obtain location coordinates.

Intelligent Supernovae Classification Systems in the KDUST context.

With the advent of large astronomical surveys plus multi-messenger astronomy, both automatic detection and classification of Type Ia supernovae have been addressed by different machine learning techniques. In this article we present three solutions aimed at the future spectrometer of the KDUST project, within a scope of benchmark, considering three different methodologies. The systems presented here are the following: CINTIA (based on hierarchical neural network architecture), SUZAN (which incorporates the solution known as fuzzy systems) and DANI (based on Deep Learning with Convolutional Neural Networks). The characteristics of the systems are presented and the benchmark is performed considering a data set containing 15.134 spectra. The best performance is obtained by the DANI architecture which provides 96% accuracy in the classification of Type Ia supernovae in relation to other spectral types.

A Deep Gaussian Process-Based Flight Trajectory Prediction Approach and Its Application on Conflict Detection

In this work, a deep Gaussian process (DGP) based framework is proposed to improve the accuracy of predicting flight trajectory in air traffic research, which is further applied to implement a probabilistic conflict detection algorithm. The Gaussian distribution is applied to serve as the probabilistic representation for illustrating the transition patterns of the flight trajectory, based on which a stochastic process is generated to build the temporal correlations among flight positions, i.e., Gaussian process (GP). Furthermore, to deal with the flight maneuverability of performing controller’s instructions, a hierarchical neural network architecture is proposed to improve the modeling representation for nonlinear features. Thanks to the intrinsic mechanism of the GP regression, the DGP model has the ability of predicting both the deterministic nominal flight trajectory (NFT) and its confidence interval (CI), denoting by the mean and standard deviation of the prediction sequence, respectively. The CI subjects to a Gaussian distribution, which lays the data foundation of the probabilistic conflict detection. Experimental results on real data show that the proposed trajectory prediction approach achieves higher prediction accuracy compared to other baselines. Moreover, the conflict detection approach is also validated by a obtaining lower false alarm and more prewarning time.

MFC-CNN: An automatic grading scheme for light stress levels of lettuce (Lactuca sativa L.) leaves

The identification and control of light stress is the key to the high-yield and high-quality production of leafy vegetables in a controlled environment. As a widely grown vegetable in the plant factory, lettuce is responsive to light intensity. Strong or weak light will seriously affect its yield and quality. These differences can be articulated by images and then classified by fine-grained classification methods. Deep learning is commonly used in crop image classification due to its fast and convenient advantages, but conventional fine-grained recognition approaches are still extremely challenging in dealing with this kind of inter-species classification. To address this issue, this work takes lettuce as the research object and established a set of leaf images for lettuce light stress grading. The leaves were divided into four categories depending on the changes in shoot fresh weight. Then, a hierarchical fusion convolutional neural network architecture (MFC-CNN) based on multi-scale input was constructed to grade the light stress. We firstly separate leaf patches from complete leaf and construct four-scale input to expand local leaf vein and texture information. The multi-scale dataset is fed into the main network at different depths according to the characteristics of CNN in feature extraction. Finally, the network is tested through comparative experiments. The results show that the proposed model has obvious advantages in light stress dataset and generalized dataset.

Pkg2Vec: Hierarchical package embedding for code authorship attribution

Authorship attribution of software is the task of identifying the author of a given piece of code. Code attribution is of importance in multiple scenarios, ranging from software plagiarism to cybersecurity. In this paper, we introduce authorship attribution of software packages that better reflect real-world scenarios in which code is organized in packages and written by teams. We present a novel approach for software package authorship attribution called Pkg2Vec, based on a hierarchical deep neural network (DNN) architecture, corresponding to the hierarchical nature of software (code) packages. The hierarchical neural network model consists of a token level encoder and an attention mechanism for a function level encoder, together producing package embedding. Beyond package embedding, we use keywords and API calls as resilient features, which reflect the programmer’s intention and style. Pkg2Vec is evaluated on a large dataset of public packages and compared to a number of other source code authorship attribution state-of-the-art algorithms. We find that Pkg2Vec significantly outperforms other approaches, achieving a 13% improvement in accuracy

Learning allocentric representations of space for navigation

The hippocampus of the mammalian brain supports spatial navigation by building cognitive maps of the environments in which the animal explores. Currently, there is little neurocomputational work investigating the encoding and decoding mechanisms of hippocampal neural representations in large-scale environments. We propose a biologically-inspired hierarchical neural network architecture to learn the transformation of egocentric sensorimotor inputs into allocentric spatial representation for navigation. The hierarchical network is composed of two parallel subnetworks mimicking the lateral entorhinal cortex (LEC) and medial entorhinal cortex (MEC), and one convergent subnetwork mimicking the hippocampus. LEC relays time-related visual information and MEC supplies space-related information in the form of multi-resolution grid codes as resulted from integrating movement information. The convergent subnetwork integrates all information from the parallel subnetworks and predicts the position of the agent in the environment. Synaptic weights of the vision-to-place and grid-to-place connections are learned based on the stochastic gradient descent algorithm. Simulations in a large virtual maze demonstrate that hippocampal place units in the model form multiple and irregularly-spaced place fields, similar to those observed in neurobiological experiments. The model is able to accurately decode the positions of the agent from the learned spatial representations. Moreover, the model is capable of adaptation to degraded visual inputs, and therefore is robust against perturbations. When the motion inputs are deprived, the model meets with localization difficulty, suffering from less accuracy in position predictions.

CFSRE: Context-aware based on frame-semantics for distantly supervised relation extraction

In relation extraction with distant supervision, noise labels are a bottleneck problem that hinders the performance of training models. Existing neural models solved this problem using attention mechanisms or multi-instance/multi-label learning to select sentences that are likely to express the relations. They mainly focused on the structural information of entity pairs and ignored semantic information of relation-clue words. Furthermore, these models do not consider the semantic scenario information of an entity pair (As the same entity pair may express different relations in different semantic scenarios). To bridge the gap, this paper proposes a novel model for relation extraction based on frame-semantics, namely Context-aware based on Frame-semantics for Distantly Supervised Relation Extraction (CFSRE). The model combines a hierarchical neural network architecture with FrameNet semantic knowledge base to solve the noise labels and the semantic representation of entity pairs. More specifically, a simple and effective instance selection method is used to select informative positive instances for model training. In addition, we propose a novel sentence representation method by combining sentence context representation with frame semantic representation of entities. We find that the joint representation leads to a better performance because it can obtain more comprehensive semantic representation of text instances. Then, a hierarchical-attention mechanism has been designed to select the most informative features for relation extraction. Finally, a softmax classifier is used to test the performance of the model. Experiments were conducted on two widely used benchmark datasets. The results demonstrate that the proposed method is significantly better than state-of-the-art methods for distantly supervised relation extraction.

Super-resolution visualization of subwavelength defects via deep learning-enhanced ultrasonic beamforming: A proof-of-principle study

Detecting small, subwavelength defect has known to be a challenging task mainly due to the diffraction limit, according to which the minimum resolvable size is in the order of the wavelength of a propagating wave. In this proof-of-concept study, we present a deep learning-enhanced super-resolution ultrasonic beamforming approach that computationally exceeds the diffraction limit and visualizes subwavelength defects. The proposed super-resolution approach is a novel subwavelength beamforming methodology enabled by a hierarchical deep neural network architecture. The first network (the detection network) globally detects defective regions from an ultrasonic beamforming image. Subsequently, the second network (the super-resolution network) locally resolves subwavelength-scale fine details of the detected defects. We validate the proposed approach using two independent datasets: a bulk wave array dataset generated by numerical simulations and guided wave array dataset generated by laboratory experiments. The results demonstrate that our deep learning super-resolution ultrasonic beamforming approach not only enables visualization of fine structural features of subwavelength defects, but also outperforms the existing widely-accepted super-resolution algorithm (time-reversal MUSIC). We also study key factors of the performance of our approach and discuss its applicability and limitations.

Artificial neural network based screening of cervical cancer using a hierarchical modular neural network architecture (HMNNA) and novel benchmark uterine cervix cancer database

The work reported in this paper presents a novel hierarchical modular neural network architecture (HMNNA) for automated screening of cervical cancer. HMNNA consists of three neural networks trained specifically on different areas of problem space under consideration, and the trained networks are then arranged in a tree structure forming hierarchical modular neural network architecture. The three specialized neural networks are trained by Levenberg–Maarquardt neural network algorithm. As compared to the standard back propagation algorithm, Levenberg–Maarquardt is fast and stable for convergence with only one drawback, i.e., storage requirement for estimated Hessian Matrix. For training and testing of HMNNA, a huge primary database is created which contains 8091 cervical cell images pertaining to 200 clinical cases collected from two health care institutions of northern India. The raw cases of cervical cancer in the form of Pap smear slides were photographed under a multi-headed digital microscope. Individual cells were manually cropped off from these slide images which were then passed through a feature extraction module for morphological profiling. Each cell was calibrated on the basis of 40 features from both cytoplasm and nucleus. After profiling, these cells were vigilantly assigned cell classes as per the latest 2001-Bethesda system of cervical cancer cell classification, by trained cytotechnicians and histopathologists. HMNNA is also trained and tested on the Herlev Benchmark dataset created by the Denmark University, which consists of 1417 cervical cancer cells. Both the datasets have seven classes of diagnosis, i.e., superficial squamous, intermediate squamous, columnar, mild dysplasia, moderate dysplasia, severe dysplasia, and carcinoma in situ, corresponding to the level of abnormality in cervical cells. These datasets are available in public domain at http://digitalpapsmeardb.in/ and http://mde-lab.aegean.gr/index.php/downloads. The screening potential of the HMNNA is compared with 25 well-known machine learning algorithms available in MatlabR2016 (Machine learning and statistics toolbox 10.2) and monolithic neural network algorithms available in Matlab neural network pattern recognition toolbox. The HMNNA outperformed in all the 25 algorithms for both the datasets. For the Novel Benchmark database, it produced a classification accuracy of 95.32% with an F-value of 0.949310 and classification accuracy of 88.41% with an F-value of 0.89145 for the Herlev dataset. The screening potential of HMNNA was also evaluated and compared with the other diagnostic systems available in the recently published literature and was found to be performing much better than the counterparts on multiple parameters of performance evaluation.

Learning programs is better than learning dynamics: A programmable neural network hierarchical architecture in a multi-task scenario

Distributed and hierarchical models of control are nowadays popular in computational modeling and robotics. In the artificial neural network literature, complex behaviors can be produced by composing elementary building blocks or motor primitives, possibly organized in a layered structure. However, it is still unknown how the brain learns and encodes multiple motor primitives, and how it rapidly reassembles, sequences and switches them by exerting cognitive control. In this paper we advance a novel proposal, a hierarchical programmable neural network architecture, based on the notion of programmability and an interpreter-programmer computational scheme. In this approach, complex (and novel) behaviors can be acquired by embedding multiple modules (motor primitives) in a single, multi-purpose neural network. This is supported by recent theories of brain functioning in which skilled behaviors can be generated by combining functional different primitives embedded in “reusable” areas of “recycled” neurons. Such neuronal substrate supports flexible cognitive control, too. Modules are seen as interpreters of behaviors having controlling input parameters, or programs that encode structures of networks to be interpreted. Flexible cognitive control can be exerted by a programmer module feeding the interpreters with appropriate input parameters, without modifying connectivity. Our results in a multiple T -maze robotic scenario show how this computational framework provides a robust, scalable and flexible scheme that can be iterated at different hierarchical layers permitting to learn, encode and control multiple qualitatively different behaviors.

A hierarchical attention-based neural network architecture, based on human brain guidance, for perception, conceptualisation, action and reasoning

We present a neural network software architecture, guided by that of the human and more generally primate brain, for the construction of an autonomous cognitive system (which we have named GNOSYS). GNOSYS is created so as to be able to attend to stimuli, to conceptualise them, to learn their predicted reward value and reason about them so as to attain those stimuli in the environment with greatest predicted value. We apply this software system to an embodied version in a robot, and describe the activities in the various component modules of GNOSYS, as well as the overall results. We briefly compare our system with some others proposed to have cognitive powers, and finish by discussion of future developments we propose for our system, as well as expanding on the arguments for and against our approach to creating such a software system.

Learning to generate articulated behavior through the bottom-up and the top-down interaction processes

A novel hierarchical neural network architecture for sensory-motor learning and behavior generation is proposed. Two levels of forward model neural networks are operated on different time scales while parametric interactions are allowed between the two network levels in the bottom-up and top-down directions. The models are examined through experiments of behavior learning and generation using a real robot arm equipped with a vision system. The results of the learning experiments showed that the behavioral patterns are learned by self-organizing the behavioral primitives in the lower level and combining the primitives sequentially in the higher level. The results contrast with prior work by Pawelzik et al. [Neural Comput. 8 (1996) 340], Tani and Nolfi [From animals to animats, 1998], and Wolpert and Kawato [Neural Networks 11 (1998) 1317] in that the primitives are represented in a distributed manner in the network in the present scheme whereas, in the prior work, the primitives were localized in specific modules in the network. Further experiments of on-line planning showed that the behavior could be generated robustly against a background of real world noise while the behavior plans could be modified flexibly in response to changes in the environment. It is concluded that the interaction between the bottom-up process of recalling the past and the top-down process of predicting the future enables both robust and flexible situated behavior.

On learning to estimate the block directional image of a fingerprint using a hierarchical neural network

This paper presents a hierarchical neural network architecture for computing fingerprints block directional images. Two separately trained neural networks are connected in series. First, the fingerprint image is divided into 16×16 blocks, each block is submitted to the first network which is a back propagation neural network. It has four counters in its output layer one for each direction to count the main directional codes in each fingerprint block. The output of this network is considered the feature vector for the fingerprint block, which is then submitted to the second network. The second network is a self-organized feature maps neural network uses an unsupervised learning strategy to group the fingerprint blocks into distinct directional classes. In this scheme, there is more than one sub-class for each directional class, an agglomerative hierarchical cluster algorithm for merging two clusters is used to merge two classes if their corresponding distances are below a specified threshold. Results obtained with a real world data set indicate the effectiveness of the proposed architecture.

A comparative study for character recognition using neural networks

In this paper, a comparative study is conducted on a number of networks to find the best classification performance for printed English characters. Hierarchical Artificial Neural Network (HANN) architecture is used to reduce the misclassifications due to noisy patterns. The training patterns are extracted features of the characters using Fast Fourier Transforms and/or H-V scanning features.

A hierarchical neural network architecture for handwritten numeral recognition

This paper presents a hierarchical neural network architecture for recognition of handwritten numeral characters. In this new architecture, two separately trained neural networks connected in series, use the pixels of the numeral image as input and yield ten outputs, the largest of which identifies the class to which the numeral image belongs. The first neural network generates the principal components of the numeral image using Oja's rule, while the second neural network uses an unsupervised learning strategy to group the principal components into distinct character clusters. In this scheme, there is more than one cluster for each numeral class. The decomposition of the global network into two independent neural networks facilitates rapid and efficient training of the individual neural networks. Results obtained with a large independently generated data set indicate the effectiveness of the proposed architecture.

NEURAL NETS SPEED FLOW CALCULATION

Abstract Neural Nets Successfully Solve Complex Fluid Flow Problem. Much research is being done in the area of neural networks, and industry is actively seeking successful application to real world problems. We describe here a successful application. We have used neural networks to model complex coolant flow patterns, such as those encountered in design of hypersonic aircraft. Previous calculation methods, while reasonably accurate, are iterative and extremely time consuming. Our new approach uses a hierarchical neural network architecture to model coolant flow distribution in multiple heat exchanger panels. This method is direct, fast, and accurate.