Small Neural Network Research Articles

Modeling health risks using neural network ensembles.

This study aims to demonstrate that demographics combined with biometrics can be used to predict obesity related chronic disease risk and produce a health risk score that outperforms body mass index (BMI)-the most commonly used biomarker for obesity. We propose training an ensemble of small neural networks to fuse demographics and biometrics inputs. The categorical outputs of the networks are then turned into a multi-dimensional risk map, which associates diverse inputs with stratified, output health risk. Our ensemble model is optimized and validated on disjoint subsets of nationally representative data (N~100,000) from the National Health and Nutrition Examination Survey (NHANES). To broaden applicability of the proposed method, we consider only non-invasive inputs that can be easily measured through modern devices. Our results show that: (a) neural networks can predict individual conditions (e.g., diabetes, hypertension) or the union of multiple (e.g., nine) health conditions; (b) Softmax model outputs can be used to stratify individual- or any-condition risk; (c) ensembles of neural networks improve generalizability; (d) multiple-input models outperform BMI (e.g., 75.1% area under the receiver operator curve for eight-input, any-condition models compared to 64.2% for BMI); (e) small neural networks are as effective as larger ones for the inference tasks considered; the proposed models are small enough that they can be expressed as human-readable equations, and they can be adapted to clinical settings to identify high-risk, undiagnosed populations.

Exploring the Impact of Model Complexity on Laryngeal Cancer Detection.

Background: Laryngeal cancer accounts for a third of all head and neck malignancies, necessitating timely detection for effective treatment and enhanced patient outcomes. Machine learning shows promise in medical diagnostics, but the impact of model complexity on diagnostic efficacy in laryngeal cancer detection can be ambiguous. Methods: In this study, we examine the relationship between model sophistication and diagnostic efficacy by evaluating three approaches: Logistic Regression, a small neural network with 4 layers of neurons and a more complex convolutional neural network with 50 layers and examine their efficacy on laryngeal cancer detection on computed tomography images. Results: Logistic regression achieved 82.5% accuracy. The 4-Layer NN reached 87.2% accuracy, while ResNet-50, a deep learning architecture, achieved the highest accuracy at 92.6%. Its deep learning capabilities excelled in discerning fine-grained CT image features. Conclusion: Our study highlights the choices involved in selecting a laryngeal cancer detection model. Logistic regression is interpretable but may struggle with complex patterns. The 4-Layer NN balances complexity and accuracy. ResNet-50 excels in image classification but demands resources. This research advances understanding affect machine learning model complexity could have on learning features of laryngeal tumor features in contrast CT images for purposes of disease prediction.

Exploring “dark-matter” protein folds using deep learning

De novo protein design explores uncharted sequence and structure space to generate novel proteins not sampled by evolution. A main challenge in de novo design involves crafting "designable" structural templates to guide the sequence searches toward adopting target structures. We present a convolutional variational autoencoder that learns patterns of protein structure, dubbed Genesis. We coupled Genesis with trRosetta to design sequences for a set of protein folds and found that Genesis is capable of reconstructing native-like distance and angle distributions for five native folds and three novel, the so-called "dark-matter" folds as a demonstration of generalizability. We used a high-throughput assay to characterize the stability of the designs through protease resistance, obtaining encouraging success rates for folded proteins. Genesis enables exploration of the protein fold space within minutes, unrestricted by protein topologies. Our approach addresses the backbone designability problem, showing that small neural networks can efficiently learn structural patterns in proteins. A record of this paper's transparent peer review process is included in the supplemental information.

Spiking activities in small neural networks induced by external forcing.

Neurons in an excitable mode do not show spiking activity and, therefore, do not contribute to information transfer transmission and its processing. However, some external influences, coupling, or time delay can lead to the appearance of oscillations in individual systems or networks. The main goal of this paper is to uncover the connection parameters and parameters of external influences that lead to the arising of spiking behavior in a small network of locally coupled FitzHugh-Nagumo oscillators. In this study, we analyze the dynamics of a small network in the absence and presence of several types of external influences. First, we consider the impact of periodic-pulse exposure generated as a periodic sequence of Gaussian pulses. Second, we show what behavior can be induced by far less regular pulsed influence (Lévy noise) and its special case called white Gaussian noise. For all types of influences, we have identified the appropriate parameters (local coupling strength, intensity, and frequency) that induce spiking activity in the small network.

Serotonergic modulation of visual processing

Neuromodulators are chemicals that alter the activity of neural networks. They can have -immediate or delayed effects on neurons, affecting their properties at both nearby and distant synapses. Neuromodulators can influence neurons in various ways, allowing even small neural Networks generate a diverse range of functional outcomes. This diversity is crucial for the flexibility and adaptability of neural functions, including sensory processing.

Deep learning for image-based detection of weeds from emergence to maturity in wheat fields

Effective weed control in wheat (Triticum aestivum L.) fields is crucial for optimizing production and ensuring food security in semi-arid regions. The implementation of deep learning for weed detection could enable precise weed management, leading to enhanced wheat yield, increased income for growers through targeted herbicide use, and a reduced environmental footprint associated with herbicide application. Charlock mustard (Sinapis arvensis L.) [CM], creeping thistle (Cirsium arvense (L.) Scop) [CT], and forking larkspur (Consolida regalis Gray) [FL]) are among the most invasive weed species in wheat fields. Effective management of these weeds is essential to ensure optimal wheat productivity in the semi-arid regions. To detect weeds, an unmanned arial vehicle was used to collect images and videos from 185 wheat fields in Tokat, Turkey. The images were collected from five phenological periods of each species including seedling, rosette, vegetative, flowering, and fruiting (in total 15 classification process). These stages are either important for timely weed control or essential for understanding weed pressure threshold, the potential for increasing weed seedbank increase, and identifying herbicide resistant weeds. A total of 145,792 objects were labeled in the images to create the dataset, divided into 80 % for training + validation and 20 % for testing. The dataset was used with the YOLOv5 (You Only Look Once) deep learning architecture. The YOLOv5 was the newest YOLO model during this study carried out. All neural networks provided by YOLOv5 (nano, small, medium, large, and xlarge) were trained and the Precision, Recall, F-1 Score, and AUC (Area under the Curve) performances of the neural networks were evaluated. The YOLOv5s (small) was the most precise neural network in detecting weeds regardless of the species. In contrast, nano was the least precise neural network in detecting each weed species. For CT, when small neural network was used, precision of detection ranged from 0.87 (fruit stage) to 0.96 (rosette and vegetative stages). For, forking larkspur, the best neural network was achieved with YOLOv5 small and the precision ranged from 0.45 (fruit stage) to 0.93 (vegetative stage). Using YOLOv5 small in Charlock mustard, the most precise detection occurred at seedling, rosette, and vegetative stages (with precision of 0.96) and the least precise detection was recorded at the fruit stage (0.81). Other performance evaluations (Recall, F-1 Score, and AUC) were in agreement with those of Precision. Our results indicated that by using YOLOv5 small neural network, all three weed species could be detected at all growth stages with the precision of 0.86 with the exception of fruit stage. Future research should (1) focus on assessing YOLOv5 small neural network with the newest versions of YOLO and also (2) evaluate this neural network for detecting weed species in wheat fields to allow for improved weed control and future recommendations.

Boolean operators and neural networks

We compute the homogeneous ideals of varieties, in a projective space of tensors, associated to different choices of the Boolean operators that describe the decision process in small neural networks. We prove that, starting with networks with three nodes, the varieties associated to different Boolean operators are all distinct.

Network Attack Classification with a Shallow Neural Network for Internet and Internet of Things (IoT) Traffic

In recent years, there has been a tremendous increase in the use of connected devices as part of the so-called Internet of Things (IoT), both in private spaces and the industry. Integrated distributed systems have shown many benefits compared to isolated devices. However, exposing industrial infrastructure to the global Internet also generates security challenges that need to be addressed to benefit from tighter systems integration and reduced reaction times. Machine learning algorithms have demonstrated their capacity to detect sophisticated cyber attack patterns. However, they often consume significant amounts of memory, computing resources, and scarce energy. Furthermore, their training relies on the availability of datasets that accurately represent real-world data traffic subject to cyber attacks. Network attacks are relatively rare events, as is reflected in the distribution of typical training datasets. Such imbalanced datasets can bias the training of a neural network and prevent it from successfully detecting underrepresented attack samples, generally known as the problem of imbalanced learning. This paper presents a shallow neural network comprising only 110 ReLU-activated artificial neurons capable of detecting representative attacks observed on a communication network. To enable the training of such small neural networks, we propose an improved attack-sharing loss function to cope with imbalanced learning. We demonstrate that our proposed solution can detect network attacks with an F1 score above 99% for various attacks found in current intrusion detection system datasets, focusing on IoT device communication. We further show that our solution can reduce the false negative detection rate of our proposed shallow network and thus further improve network security while enabling processing at line rate in low-complexity network intrusion systems.

Conforming Capacitive Load Cells for Conical Pick Cutters.

In underground coal mining, machine operators put themselves at risk when getting close to the machine or cutting face to observe the process. To improve the safety and efficiency of machine operators, a cutting force sensor is proposed. A linear cutting machine is used to cut two separate coal samples cast in concrete with conical pick cutters to simulate mining with a continuous miner. Linear and neural network regression models are fit using 100 random 70:30 test/train splits. The normal force exceeds 60 kN during the rock-cutting tests, and it is averaged using a low pass filter with a 10 Hertz cutoff frequency. The sensor uses measurements of the resonant frequency of capacitive cells in a steel case to determine cutting forces. When used in the rock-cutting experiments, the sensor conforms to the tooling and the stiffness and sensitivity are increased compared to the initial configuration. The sensor is able to track the normal force on the conical picks with a mean absolute error less than 6 kN and an R2 score greater than 0.60 using linear regression. A small neural network with a second-order polynomial expansion is able to improve this to a mean absolute error of less than 4 kN and an R2 score of around 0.80. Filtering measurements before regression fitting is explored. This type of sensor could allow operators to assess tool wear and material type using objective force measurements while maintaining a greater distance from the cutting interface.

Score distillation for anomaly detection

Recently, significant performance improvements have been achieved in deep learning-based anomaly detection methods by introducing large neural network architectures and complex anomaly scoring functions. However, the computational cost and memory usage required in the inference phase have also increased significantly, thereby limiting their use in real-time applications. In this paper, we propose a score distillation method that adopts the concept of knowledge distillation. An existing high-performance anomaly detection method is used as the teacher. A small neural network is then trained as the student to mimic the scoring function of the teacher. In the inference phase, the anomaly score for a query instance is obtained by a single forward pass through the student network without requiring any complicated computation processes. We demonstrate that the proposed method makes anomaly detection faster and more efficient while maintaining high performance.

Online Neural Path Guiding with Normalized Anisotropic Spherical Gaussians

Importance sampling techniques significantly reduce variance in physically based rendering. In this article, we propose a novel online framework to learn the spatial-varying distribution of the full product of the rendering equation, with a single small neural network using stochastic ray samples. The learned distributions can be used to efficiently sample the full product of incident light. To accomplish this, we introduce a novel closed-form density model, called the Normalized Anisotropic Spherical Gaussian mixture, that can model a complex light field with a small number of parameters and that can be directly sampled. Our framework progressively renders and learns the distribution, without requiring any warm-up phases. With the compact and expressive representation of our density model, our framework can be implemented entirely on the GPU, allowing it to produce high-quality images with limited computational resources. The results show that our framework outperforms existing neural path guiding approaches and achieves comparable or even better performance than state-of-the-art online statistical path guiding techniques.

Generative Model-Based Feature Knowledge Distillation for Action Recognition

Knowledge distillation (KD), a technique widely employed in computer vision, has emerged as a de facto standard for improving the performance of small neural networks. However, prevailing KD-based approaches in video tasks primarily focus on designing loss functions and fusing cross-modal information. This overlooks the spatial-temporal feature semantics, resulting in limited advancements in model compression. Addressing this gap, our paper introduces an innovative knowledge distillation framework, with the generative model for training a lightweight student model. In particular, the framework is organized into two steps: the initial phase is Feature Representation, wherein a generative model-based attention module is trained to represent feature semantics; Subsequently, the Generative-based Feature Distillation phase encompasses both Generative Distillation and Attention Distillation, with the objective of transferring attention-based feature semantics with the generative model. The efficacy of our approach is demonstrated through comprehensive experiments on diverse popular datasets, proving considerable enhancements in video action recognition task. Moreover, the effectiveness of our proposed framework is validated in the context of more intricate video action detection task. Our code is available at https://github.com/aaai-24/Generative-based-KD.

When to Grow? A Fitting Risk-Aware Policy for Layer Growing in Deep Neural Networks

Neural growth is the process of growing a small neural network to a large network and has been utilized to accelerate the training of deep neural networks. One crucial aspect of neural growth is determining the optimal growth timing. However, few studies investigate this systematically. Our study reveals that neural growth inherently exhibits a regularization effect, whose intensity is influenced by the chosen policy for growth timing. While this regularization effect may mitigate the overfitting risk of the model, it may lead to a notable accuracy drop when the model underfits. Yet, current approaches have not addressed this issue due to their lack of consideration of the regularization effect from neural growth. Motivated by these findings, we propose an under/over fitting risk-aware growth timing policy, which automatically adjusts the growth timing informed by the level of potential under/overfitting risks to address both risks. Comprehensive experiments conducted using CIFAR-10/100 and ImageNet datasets show that the proposed policy achieves accuracy improvements of up to 1.3% in models prone to underfitting while achieving similar accuracies in models suffering from overfitting compared to the existing methods.

Neural Data Augmentation for Legal Overruling Task: Small Deep Learning Models vs. Large Language Models

Deep learning models produce impressive results in any natural language processing applications when given a better learning strategy and trained with large labeled datasets. However, the annotation of massive training data is far too expensive, especially in the legal domain, due to the need for trained legal professionals. Data augmentation solves the problem of learning without labeled big data. In this paper, we employ pre-trained language models and prompt engineering to generate large-scale pseudo-labeled data for the legal overruling task using 100 data samples. We train small recurrent and convolutional deep-learning models using this data and fine-tune a few other transformer models. We then evaluate the effectiveness of the models, both with and without data augmentation, using the benchmark dataset and analyze the results. We also test the performance of these models with the state-of-the-art GPT-3 model under few-shot setting. Our experimental findings demonstrate that data augmentation results in better model performance in the legal overruling task than models trained without augmentation. Furthermore, our best-performing deep learning model trained on augmented data outperforms the few-shot GPT-3 by 18% in the F1-score. Additionally, our results highlight that the small neural networks trained with augmented data achieve outcomes comparable to those of other large language models.

Swift Prediction of Battery Performance: Applying Machine Learning Models on Microstructural Electrode Images for Lithium-Ion Batteries

In this study, we investigate the use of artificial neural networks as a potentially efficient method to determine the rate capability of electrodes for lithium-ion batteries with different porosities. The performance of a lithium-ion battery is, to a large extent, determined by the microstructure (i.e., layer thickness and porosity) of its electrodes. Tailoring the microstructure to a specific application is a crucial process in battery development. However, unravelling the complex correlations between microstructure and rate performance using either experiments or simulations is time-consuming and costly. Our approach provides a swift method for predicting the rate capability of battery electrodes by using machine learning on microstructural images of electrode cross-sections. We train multiple models in order to predict the specific capacity based on the batteries’ microstructure and investigate the decisive parts of the microstructure through the use of explainable artificial intelligence (XAI) methods. Our study shows that even comparably small neural network architectures are capable of providing state-of-the-art prediction results. In addition to this, our XAI studies demonstrate that the models are using understandable human features while ignoring present artefacts.

Neural Subspaces for Light Fields.

We introduce a framework for compactly representing light field content with the novel concept of neural subspaces. While the recently proposed neural light field representation achieves great compression results by encoding a light field into a single neural network, the unified design is not optimized for the composite structures exhibited in light fields. Moreover, encoding every part of the light field into one network is not ideal for applications that require rapid transmission and decoding. We recognize this problem's connection to subspace learning. We present a method that uses several small neural networks, specializing in learning the neural subspace for a particular light field segment. Moreover, we propose an adaptive weight sharing strategy among those small networks, improving parameter efficiency. In effect, this strategy enables a concerted way to track the similarity among nearby neural subspaces by leveraging the layered structure of neural networks. Furthermore, we develop a soft-classification technique to enhance the color prediction accuracy of neural representations. Our experimental results show that our method better reconstructs the light field than previous methods on various light field scenes. We further demonstrate its successful deployment on encoding light fields with irregular viewpoint layout and dynamic scene content.

A neural network model for the evolution of learning in changing environments.

Learning from past experience is an important adaptation and theoretical models may help to understand its evolution. Many of the existing models study simple phenotypes and do not consider the mechanisms underlying learning while the more complex neural network models often make biologically unrealistic assumptions and rarely consider evolutionary questions. Here, we present a novel way of modelling learning using small neural networks and a simple, biology-inspired learning algorithm. Learning affects only part of the network, and it is governed by the difference between expectations and reality. We use this model to study the evolution of learning under various environmental conditions and different scenarios for the trade-off between exploration (learning) and exploitation (foraging). Efficient learning readily evolves in our individual-based simulations. However, in line with previous studies, the evolution of learning is less likely in relatively constant environments, where genetic adaptation alone can lead to efficient foraging, or in short-lived organisms that cannot afford to spend much of their lifetime on exploration. Once learning does evolve, the characteristics of the learning strategy (i.e. the duration of the learning period and the learning rate) and the average performance after learning are surprisingly little affected by the frequency and/or magnitude of environmental change. In contrast, an organism's lifespan and the distribution of resources in the environment have a clear effect on the evolved learning strategy: a shorter lifespan or a broader resource distribution lead to fewer learning episodes and larger learning rates. Interestingly, a longer learning period does not always lead to better performance, indicating that the evolved neural networks differ in the effectiveness of learning. Overall, however, we show that a biologically inspired, yet relatively simple, learning mechanism can evolve to lead to an efficient adaptation in a changing environment.

Reduced-Order Neural Network Synthesis With Robustness Guarantees.

In the wake of the explosive growth in smartphones and cyber-physical systems, there has been an accelerating shift in how data are generated away from centralized data toward on-device-generated data. In response, machine learning algorithms are being adapted to run locally on board, potentially hardware-limited, devices to improve user privacy, reduce latency, and be more energy efficient. However, our understanding of how these device-orientated algorithms behave and should be trained is still fairly limited. To address this issue, a method to automatically synthesize reduced-order neural networks (having fewer neurons) approximating the input-output mapping of a larger one is introduced. The reduced-order neural network's weights and biases are generated from a convex semidefinite program that minimizes the worst case approximation error with respect to the larger network. Worst case bounds for this approximation error are obtained and the approach can be applied to a wide variety of neural networks architectures. What differentiates the proposed approach to existing methods for generating small neural networks, e.g., pruning, is the inclusion of the worst case approximation error directly within the training cost function, which should add robustness to out-of-sample data points. Numerical examples highlight the potential of the proposed approach. The overriding goal of this article is to generalize recent results in the robustness analysis of neural networks to a robust synthesis problem for their weights and biases.

Fluid Simulation on Neural Flow Maps

We introduce Neural Flow Maps, a novel simulation method bridging the emerging paradigm of implicit neural representations with fluid simulation based on the theory of flow maps, to achieve state-of-the-art simulation of in-viscid fluid phenomena. We devise a novel hybrid neural field representation, Spatially Sparse Neural Fields (SSNF), which fuses small neural networks with a pyramid of overlapping, multi-resolution, and spatially sparse grids, to compactly represent long-term spatiotemporal velocity fields at high accuracy. With this neural velocity buffer in hand, we compute long-term, bidirectional flow maps and their Jacobians in a mechanistically symmetric manner, to facilitate drastic accuracy improvement over existing solutions. These long-range, bidirectional flow maps enable high advection accuracy with low dissipation, which in turn facilitates high-fidelity incompressible flow simulations that manifest intricate vortical structures. We demonstrate the efficacy of our neural fluid simulation in a variety of challenging simulation scenarios, including leapfrogging vortices, colliding vortices, vortex reconnections, as well as vortex generation from moving obstacles and density differences. Our examples show increased performance over existing methods in terms of energy conservation, visual complexity, adherence to experimental observations, and preservation of detailed vortical structures.

Recursive-NeRF: An Efficient and Dynamically Growing NeRF.

View synthesis methods using implicit continuous shape representations learned from a set of images, such as the Neural Radiance Field (NeRF) method, have gained increasing attention due to their high quality imagery and scalability to high resolution. However, the heavy computation required by its volumetric approach prevents NeRF from being useful in practice; minutes are taken to render a single image of a few megapixels. Now, an image of a scene can be rendered in a level-of-detail manner, so we posit that a complicated region of the scene should be represented by a large neural network while a small neural network is capable of encoding a simple region, enabling a balance between efficiency and quality. Recursive-NeRF is our embodiment of this idea, providing an efficient and adaptive rendering and training approach for NeRF. The core of Recursive-NeRF learns uncertainties for query coordinates, representing the quality of the predicted color and volumetric intensity at each level. Only query coordinates with high uncertainties are forwarded to the next level to a bigger neural network with a more powerful representational capability. The final rendered image is a composition of results from neural networks of all levels. Our evaluation on public datasets and a large-scale scene dataset we collected shows that Recursive-NeRF is more efficient than NeRF while providing state-of-the-art quality. The code will be available at https://github.com/Gword/Recursive-NeRF.