Neural Network Theory Research Articles

Single-Subject Deep-Learning Image Reconstruction With a Neural Optimization Transfer Algorithm for PET-Enabled Dual-Energy CT Imaging.

Combining dual-energy computed tomography (DECT) with positron emission tomography (PET) offers many potential clinical applications but typically requires expensive hardware upgrades or increases radiation doses on PET/CT scanners due to an extra X-ray CT scan. The recent PET-enabled DECT method allows DECT imaging on PET/CT without requiring a second X-ray CT scan. It combines the already existing X-ray CT image with a 511 keV γ -ray CT (gCT) image reconstructed from time-of-flight PET emission data. A kernelized framework has been developed for reconstructing gCT image but this method has not fully exploited the potential of prior knowledge. Use of deep neural networks may explore the power of deep learning in this application. However, common approaches require a large database for training, which is impractical for a new imaging method like PET-enabled DECT. Here, we propose a single-subject method by using neural-network representation as a deep coefficient prior to improving gCT image reconstruction without population-based pre-training. The resulting optimization problem becomes the tomographic estimation of nonlinear neural-network parameters from gCT projection data. This complicated problem can be efficiently solved by utilizing the optimization transfer strategy with quadratic surrogates. Each iteration of the proposed neural optimization transfer algorithm includes: PET activity image update; gCT image update; and least-square neural-network learning in the gCT image domain. This algorithm is guaranteed to monotonically increase the data likelihood. Results from computer simulation, real phantom data and real patient data have demonstrated that the proposed method can significantly improve gCT image quality and consequent multi-material decomposition as compared to other methods.

NN2Poly: A Polynomial Representation for Deep Feed-Forward Artificial Neural Networks.

Interpretability of neural networks (NNs) and their underlying theoretical behavior remain an open field of study even after the great success of their practical applications, particularly with the emergence of deep learning. In this work, NN2Poly is proposed: a theoretical approach to obtain an explicit polynomial model that provides an accurate representation of an already trained fully connected feed-forward artificial NN a multilayer perceptron (MLP). This approach extends a previous idea proposed in the literature, which was limited to single hidden layer networks, to work with arbitrarily deep MLPs in both regression and classification tasks. NN2Poly uses a Taylor expansion on the activation function, at each layer, and then applies several combinatorial properties to calculate the coefficients of the desired polynomials. Discussion is presented on the main computational challenges of this method, and the way to overcome them by imposing certain constraints during the training phase. Finally, simulation experiments as well as applications to real tabular datasets are presented to demonstrate the effectiveness of the proposed method.

Semantics-Guided Hierarchical Feature Encoding Generative Adversarial Network for Visual Image Reconstruction From Brain Activity.

The utilization of deep learning techniques for decoding visual perception images from brain activity recorded by functional magnetic resonance imaging (fMRI) has garnered considerable attention in recent research. However, reconstructed images from previous studies still suffer from low quality or unreliability. Moreover, the complexity inherent to fMRI data, characterized by high dimensionality and low signal-to-noise ratio, poses significant challenges in extracting meaningful visual information for perceptual reconstruction. In this regard, we proposes a novel neural decoding model, named the hierarchical semantic generative adversarial network (HS-GAN), inspired by the hierarchical encoding of the visual cortex and the homology theory of convolutional neural networks (CNNs), which is capable of reconstructing perceptual images from fMRI data by leveraging the hierarchical and semantic representations. The experimental results demonstrate that HS-GAN achieved the best performance on Horikawa2017 dataset (histogram similarity: 0.447, SSIM-Acc: 78.9%, Peceptual-Acc: 95.38%, AlexNet(2): 96.24% and AlexNet(5): 94.82%) over existing advanced methods, indicating improved naturalness and fidelity of the reconstructed image. The versatility of the HS-GAN was also highlighted, as it demonstrated promising generalization capabilities in reconstructing handwritten digits, achieving the highest SSIM (0.783±0.038), thus extending its application beyond training solely on natural images.

Financial Risk Early Warning Model for Listed Companies Using BP Neural Network and Rough Set Theory

Financial Risk Early Warning Model for Listed Companies Using BP Neural Network and Rough Set Theory

Meta-learning as a bridge between neural networks and symbolic Bayesian models

Abstract Meta-learning is even more broadly relevant to the study of inductive biases than Binz et al. suggest: Its implications go beyond the extensions to rational analysis that they discuss. One noteworthy example is that meta-learning can act as a bridge between the vector representations of neural networks and the symbolic hypothesis spaces used in many Bayesian models.

Discussion on the use of fashion elements in ceramic design in the context of big data

Abstract In the context of the significant data era, this study explores the application of fashion elements in ceramic design. By integrating modern aesthetics and traditional craftsmanship, it realizes the innovation of ceramic design and promotes the development of ceramic industry. A neural network-based assessment model, especially BP artificial neural network, is used to quantitatively analyze and apply fashion elements in ceramic design. First, the basic theory of neural network, including MP model and BP algorithm, is introduced. The study established a network model with a 5-30-5 structure by training 800 sets of color samples. The experimental results show that the network reaches the mean square error target (MSE=1e-7) after 1516 iterations, proving the model’s effectiveness in dealing with nonlinear color semantic mapping. The analysis of the prediction accuracy of the color scheme shows that the maximum errors are 2.22, 1.93, and 1.36 in the L, A, and B components, respectively. These results are lower than the national standard color difference range, indicating that the proposed model has practical application in ceramic design. The neural network model can effectively integrate fashion elements into ceramic design, providing a new perspective and technical support for traditional ceramic design.

Combining Optimal Path Search With Task-Dependent Learning in a Neural Network.

Finding optimal paths in connected graphs requires determining the smallest total cost for traveling along the graph's edges. This problem can be solved by several classical algorithms, where, usually, costs are predefined for all edges. Conventional planning methods can, thus, normally not be used when wanting to change costs in an adaptive way following the requirements of some task. Here, we show that one can define a neural network representation of path-finding problems by transforming cost values into synaptic weights, which allows for online weight adaptation using network learning mechanisms. When starting with an initial activity value of one, activity propagation in this network will lead to solutions, which are identical to those found by the Bellman-Ford (BF) algorithm. The neural network has the same algorithmic complexity as BF, and, in addition, we can show that network learning mechanisms (such as Hebbian learning) can adapt the weights in the network augmenting the resulting paths according to some task at hand. We demonstrate this by learning to navigate in an environment with obstacles as well as by learning to follow certain sequences of path nodes. Hence, the here-presented novel algorithm may open up a different regime of applications where path augmentation (by learning) is directly coupled with path finding in a natural way.

Global geometry of chemical graph neural network representations in terms of chemical moieties

The embedding vectors from a Graph Neural Network trained on quantum chemical data allow for a global geometric space with a Euclidean distance metric. Moieties that are close in chemical sense, are also close in Euclidean sense.

Generalizable Reconstruction for Accelerating MR Imaging via Federated Learning with Neural Architecture Search.

Heterogeneous data captured by different scanning devices and imaging protocols can affect the generalization performance of the deep learning magnetic resonance (MR) reconstruction model. While a centralized training model is effective in mitigating this problem, it raises concerns about privacy protection. Federated learning is a distributed training paradigm that can utilize multi-institutional data for collaborative training without sharing data. However, existing federated learning MR image reconstruction methods rely on models designed manually by experts, which are complex and computationally expensive, suffering from performance degradation when facing heterogeneous data distributions. In addition, these methods give inadequate consideration to fairness issues, namely ensuring that the model's training does not introduce bias towards any specific dataset's distribution. To this end, this paper proposes a generalizable federated neural architecture search framework for accelerating MR imaging (GAutoMRI). Specifically, automatic neural architecture search is investigated for effective and efficient neural network representation learning of MR images from different centers. Furthermore, we design a fairness adjustment approach that can enable the model to learn features fairly from inconsistent distributions of different devices and centers, and thus facilitate the model to generalize well to the unseen center. Extensive experiments show that our proposed GAutoMRI has better performances and generalization ability compared with seven state-of-the-art federated learning methods. Moreover, the GAutoMRI model is significantly more lightweight, making it an efficient choice for MR image reconstruction tasks. The code will be made available at https://github.com/ternencewu123/GAutoMRI.

Learning Kernel-Modulated Neural Representation for Efficient Light Field Compression.

Light fields capture 3D scene information by recording light rays emitted from a scene at various orientations. They offer a more immersive perception, compared with classic 2D images, but at the cost of huge data volumes. In this paper, we design a compact neural network representation for the light field compression task. In the same vein as the deep image prior, the neural network takes randomly initialized noise as input and is trained in a supervised manner in order to best reconstruct the target light field Sub-Aperture Images (SAIs). The network is composed of two types of complementary kernels: descriptive kernels (descriptors) that store scene description information learned during training, and modulatory kernels (modulators) that control the rendering of different SAIs from the queried perspectives. To further enhance compactness of the network meanwhile retain high quality of the decoded light field, we propose modulator allocation and apply kernel tensor decomposition techniques, followed by non-uniform quantization and lossless entropy coding. Extensive experiments demonstrate that our method outperforms other state-of-the-art (SOTA) methods by a significant margin in the light field compression task. Moreover, after adapting descriptors, the modulators learned from one light field can be transferred to new light fields for rendering dense views, showing the potential of the solution for view synthesis.

STATISTICA INFERENȚIALĂ SUPERIOARĂ A DATELOR EXPERIMENTALE ALE UNUI CULTIVATOR EXPERIMENTAL COMPLEX

The aim of the research is to highlight some statistical tools that favour extracting the components of the dynamic process that are dependent on the forward speed of some agricultural aggregates. The main objectives are: (I) identification of a minimum number of components in a multitude of random variables, with the help of which the other random variables can be calculated, and the application of this result to the strain gauge measurements; (II) establishing the connection between the synthetic results that partially solve the first objective and the forward speed of the agricultural aggregate. The second objective is used to obtain indications in search of the parameters’ dependencies on the forward speed of the aggregate. The first objective seeks to determine a group of three signals from the twelve, with the help of which the best multivariate linear interpolation is obtained for the other nine signals, which in physical terms means the reduction to a quarter of the measurement points and of the strain sensors used. A result associated with the first objective refers to the estimation of information loss due to the limited number of deformation sensors mounted on the tested structure. The article also presents attempts to use the results of the theory of neural networks and statistical interaction. In order to capitalise on the experimental data in this complex statistical framework, it is necessary to monitor at least the working speed (not only the average speed per experiment), fuel consumption, working depth (continuously monitored), soil moisture etc.

계산적 인지과학의 관점에서 바라본 과학적 설명

This paper introduces the process of explanation in computational cognitive science, which involves elucidating why phenomena or events occur within the framework of causal relationships. It highlights the similarity between everyday explanations and scientific explanations when examined through the lens of cognitive processes, noting that scientific explanations are more systematic, rigorous, and precise.
 The three crucial stages of explanation discussed are firstly gathering information to provision the system's structure, secondly generating hypotheses to advance the explanation, and thirdly evaluating competitive hypotheses or explanations. The paper goes on to introduce the computational implementation of cognitive science frameworks for scientific explanation through deductive, probabilistic, and neural network approaches.
 In the deductive approach, the discussion revolves around Hempel's (1965, 1966) explanatory model, illustrating its simulation as a rule-based system when given laws and conditions. The probabilistic approach, primarily utilizing Bayesian networks, is shown to involve tasks such as generating hypotheses and assessing suitable hypotheses through conditional probabilities. Similarly, the neural network theoretical approach demonstrates that hypothesis generation or explanation through identifying causes can occur through the interaction of neural cells.
 Notably, one significant advantage of neural network theory is its ability to perform abductive inference or hypothesis generation not only based on linguistic expressions but also by accepting visual, emotional, and non-linguistic information as inputs and producing non-linguistic outputs. This advantage is ultimately seen as the ability to accommodate a variety of expression forms, termed as a "multi-modal mode."

Learning image representations for anomaly detection: Application to discovery of histological alterations in drug development

We present a system for anomaly detection in histopathological images. In histology, normal samples are usually abundant, whereas anomalous (pathological) cases are scarce or not available. Under such settings, one-class classifiers trained on healthy data can detect out-of-distribution anomalous samples. Such approaches combined with pre-trained Convolutional Neural Network (CNN) representations of images were previously employed for anomaly detection (AD). However, pre-trained off-the-shelf CNN representations may not be sensitive to abnormal conditions in tissues, while natural variations of healthy tissue may result in distant representations. To adapt representations to relevant details in healthy tissue we propose training a CNN on an auxiliary task that discriminates healthy tissue of different species, organs, and staining reagents. Almost no additional labeling workload is required, since healthy samples come automatically with aforementioned labels. During training we enforce compact image representations with a center-loss term, which further improves representations for AD. The proposed system outperforms established AD methods on a published dataset of liver anomalies. Moreover, it provided comparable results to conventional methods specifically tailored for quantification of liver anomalies. We show that our approach can be used for toxicity assessment of candidate drugs at early development stages and thereby may reduce expensive late-stage drug attrition.

Neural networks with ReLU powers need less depth

Despite the widespread success of deep learning in various applications, neural network theory has been lagging behind. The choice of the activation function plays a critical role in the expressivity of a neural network but for reasons that are not yet fully understood. While the rectified linear unit (ReLU) is currently one of the most popular activation functions, ReLU squared has only recently been empirically shown to be pivotal in producing consistently superior results for state-of-the-art deep learning tasks (So et al., 2021). To analyze the expressivity of neural networks with ReLU powers, we employ the novel framework of Gribonval et al. (2022) based on the classical concept of approximation spaces. We consider the class of functions for which the approximation error decays at a sufficiently fast rate as network complexity, measured by the number of weights, increases. We show that when approximating sufficiently smooth functions that cannot be represented by sufficiently low-degree polynomials, networks with ReLU powers need less depth than those with ReLU. Moreover, if they have the same depth, networks with ReLU powers can have potentially faster approximation rates. Lastly, our computational experiments on approximating the Rastrigin and Ackley functions with deep neural networks showed that ReLU squared and ReLU cubed networks consistently outperform ReLU networks.

Exploring fMRI RDMs: enhancing model robustness through neurobiological data

Artificial neural networks (ANNs) are sensitive to perturbations and adversarial attacks. One hypothesized solution to adversarial robustness is to align manifolds in the embedded space of neural networks with biologically grounded manifolds. Recent state-of-the-art works that emphasize learning robust neural representations, rather than optimizing for a specific target task like classification, support the idea that researchers should investigate this hypothesis. While works have shown that fine-tuning ANNs to coincide with biological vision does increase robustness to both perturbations and adversarial attacks, these works have relied on proprietary datasets—the lack of publicly available biological benchmarks makes it difficult to evaluate the efficacy of these claims. Here, we deliver a curated dataset consisting of biological representations of images taken from two commonly used computer vision datasets, ImageNet and COCO, that can be easily integrated into model training and evaluation. Specifically, we take a large functional magnetic resonance imaging (fMRI) dataset (BOLD5000), preprocess it into representational dissimilarity matrices (RDMs), and establish an infrastructure that anyone can use to train models with biologically grounded representations. Using this infrastructure, we investigate the representations of several popular neural networks and find that as networks have been optimized for tasks, their correspondence with biological fidelity has decreased. Additionally, we use a previously unexplored graph-based technique, Fiedler partitioning, to showcase the viability of the biological data, and the potential to extend these analyses by extending RDMs into Laplacian matrices. Overall, our findings demonstrate the potential of utilizing our new biological benchmark to effectively enhance the robustness of models.

КОНЦЕПЦИЯ НООСУФИЗМА КАК ИНСТРУМЕНТ ИНТЕЛЛЕКТУАЛЬНОЙ МОДЕРНИЗАЦИИ

Доказывается, что для интеллектуальной модернизации Казахстана необходимо уделить повышенное внимание тем парадигмам развития науки и культуры, которые максимально полно отвечают его социокультурному коду. Показано, что существует возможность осуществить синтез идей В.И. Вернадского и практик, построенных на учении суфиев. Развитие идей В.И. Вернадского, в частности, нейросетевая теория ноосферы, позволяет, в частности, объяснить экстремально высокую эффективность деятельности суфийских орденов, которые фактически реорганизовали социокультурный код народов, населяющих огромную территорию от Алтая до Крыма. Инструменты, лежащие в основе суфийских практик, оказывали непосредственное воздействие на надличностный уровень переработки информации, существование которого также доказывается в рамках нейросетевой теории ноосферы на сугубо рациональной основе. Эти инструменты – уже на следующем витке исторического развития – могут быть использованы в целях интеллектуальной модернизации Казахстана, в чем имеется более чем острая необходимость. Синтез суфийских практик с идеями В.И. Вернадского, модернизированными с использованием современной теории нейроных сетей, приводит к концепции ноосуфизма, нацеленной на переход к когнитивному обществу за счет целенаправленного управления процессами, происходящими на надличностном уровне переработки информации. Важным направлением деятельности здесь является использование ресурсов профессионального коллективного бессознательного для повышения эффективности обучения в высшей школе. Дана трактовка обучения как направленной реконструкции интеллекта обучающегося. Основой для этого является тезис о дуальной сущности интеллекта и сознания человека, который также вытекает из нейросетевой теории ноосферы. Обоснована целесообразность издания учебников «ноосуфийского стиля», призванных обеспечить использование ресурсов надличностных информационных объектов. Сформулированы базовые критерии, которым должны отвечать такие учебники.

Response to stress via underlying deep gene regulation networks

Exposure of cells to non‐optimal growth conditions or to any environment that reduces cell viability can be considered as a stress. In this paper, we are going to highlight the main factors that determine the danger of stress to a cell considered as a biochemical system. To this end, we introduce a new mathematical concept of biosystem stability, where we take into account a signal transduction by deep gene networks. Using this concept and known results on approximations by deep networks, we find asymptotic estimates of the size and the depth of gene regulation networks that define the stress response. We propose a new algorithm to find the gene network approximating a prescribed output. It allows us, with the help of Kolmogorov ‐entropy and the deep neural network theory, to estimate the number of genes involved in regulation of responses on a stress (for example, a heat shock). We show that the main factors that increase the sensitivity of the systems with respect to a stress are the number of biochemical network parameters affected by the stress and sensitivities of kinetic rates with respect to these parameters.

A general scheme for point defect sink strength calculation and related machine-learning-based expressions

Irradiation tends to increase the concentration of point defects (PDs) in crystalline materials, whose consecutive interactions with other types of defects, such as dislocation and void, are recognised highly responsible for the characteristic plastic and damaging behaviours of materials under irradiation. Conventional treatments on evaluating the strength of PD sinks see their limitation with strong regularity requirements over the models used for summarising the key underlying microstructural behaviours, where analytical solutions are bound to be the outcome. The present article serves to introduce a general scheme for PD sink strength evaluation, where constraints on solution analyticity are fully resolved with the use of machine learning. In particular, a neural network representation of the PD sink strength due to void/bubble is derived, where PD transportation tendencies against the hydrostatic pressure gradient surrounding a bubble can be considered in details. The treatment is also applied to analyse PD sink strength due to edge dislocation clusters. For nearly uniformly distributed clusters, upon undertaking a two-scale asymptotic strategy, the corresponding sink strength formulation becomes explicit. For randomly distributed dislocations, the sink strength is found to roughly scale with the onsite dislocation density. But for patterned dislocations, such as dislocation dipoles, their sink strength is suggested to vary with the applied load. The machine-learning-based formulation is also compared well with the results obtained by other multiscale methods.

SCA-Former: transformer-like network based on stream-cross attention for medical image segmentation

Objective. Deep convolutional neural networks (CNNs) have been widely applied in medical image analysis and achieved satisfactory performances. While most CNN-based methods exhibit strong feature representation capabilities, they face challenges in encoding long-range interaction information due to the limited receptive fields. Recently, the Transformer has been proposed to alleviate this issue, but its cost is greatly enlarging the model size, which may inhibit its promotion. Approach. To take strong long-range interaction modeling ability and small model size into account simultaneously, we propose a Transformer-like block-based U-shaped network for medical image segmentation, dubbed as SCA-Former. Furthermore, we propose a novel stream-cross attention (SCA) module to enforce the network to focus on finding a balance between local and global representations by extracting multi-scale and interactive features along spatial and channel dimensions. And SCA can effectively extract channel, multi-scale spatial, and long-range information for a more comprehensive feature representation. Main results. Experimental results demonstrate that SCA-Former outperforms the current state-of-the-art (SOTA) methods on three public datasets, including GLAS, ISIC 2017 and LUNG. Significance. This work exhibits a promising method to enhance the feature representation of convolutional neural networks and improve segmentation performance.

Identification of structured nonlinear state–space models for hysteretic systems using neural network hysteresis operators

Hysteretic system behavior is ubiquitous in science and engineering fields including measurement systems and applications. In this paper, we put forth a nonlinear state–space system identification method that combines the state–space equations to capture the system dynamics with a compact and exact artificial neural network (ANN) representation of the classical Prandtl–Ishlinskii (PI) hysteresis. These ANN representations called PI hysteresis operator neurons employ recurrent ANNs with classical activation functions, and thus can be trained with classical neural network learning algorithms. The structured nonlinear state–space model class proposed in this paper, for the first time, offers a flexible interconnection of PI hysteresis operators with a linear state–space model through a linear fractional representation. This results in a comprehensive and flexible model structure. The performance is validated both on numerical simulation and on measurement data.