Simple Neural Network Architecture Research Articles

A non-memory-based functional neural framework for animal caching behavior

The brain’s extraordinary abilities are often attributed to its capacity to learn and adapt. But memory has its limitations, especially when faced with tasks such as retrieving thousands of food items—a common behavior in scatter-hoarding animals. Here, we propose a brain mechanism that may facilitate caching and retrieval behaviors, with a focus on hippocampal spatial cells. Rather than memorizing the locations of their caches, as previously hypothesized, we suggest that cache-hoarding animals employ a static mechanism akin to hash functions commonly used in computing. Our mathematical model aligns with the activity of hippocampal spatial cells, which respond to an animal’s positional attention. We know that the region that activates each spatial cell remains consistent across subsequent visits to the same area but not between areas. This remapping, combined with the uniqueness of cognitive maps, produces persistent hash functions that can serve both food caching and retrieval. We present a simple neural network architecture that can generate such a probabilistic hash that is unique to the animal and not sensitive to environmental changes. This mechanism could serve a virtually boundless capacity for the encoding of any structured data.

Spatiotemporal self-supervised predictive learning for atmospheric variable prediction via multi-group multi-attention

Atmospheric variable prediction (AVP) is crucial for atmospheric and environmental science, as well as practical applications related to production and daily life. Traditional AVP commonly relies on numerical modeling methods. In recent years, deep learning (DL) technologies have introduced new approaches for AVP. With the support of massive high-quality data products, DL methods have achieved remarkable progress. Simultaneously, studying and annotating complex phenomena and processes within vast datasets require specialized researchers and come with substantial costs. The self-supervised learning (SSL) mechanism in DL can facilitate representation learning from unlabeled data, thereby significantly reducing the expenses associated with manual annotation. However, there has been limited exploration of the SSL mechanism in atmospheric variable prediction and analysis. Furthermore, the prevailing large-scale predicting applications based on AVP mainly focus on massive data processing, multi-modal approaches, and foundation model technologies, often neglecting the critical aspects of spatiotemporal representation learning. In light of the above, based on the cutting-edge DL techniques but diverges from the current popular contrastive or generative SSL frameworks, we devise a spatiotemporal (ST) self-supervised predictive learning (SSPL) method (dubbed ST-SSPL) for AVP, which enables the learning of predictive characteristics from unlabeled atmospheric data. Additionally, we improve a multi-group multi-attention (MGMA) method, integrating it into our predictive framework to enhance spatiotemporal representation learning. In experiments conducted on the widely used ERA5 dataset, the proposed ST-SSPL method has achieved superior performance and high efficiency using a simple convolutional neural network (CNN) architecture. These advancements offer crucial methods and references for relevant large-scale applications.

A good result of brain tumor classification based on simple convolutional neural network architecture

A good result of brain tumor classification based on simple convolutional neural network architecture

ProteusNeRF: Fast Lightweight NeRF Editing using 3D-Aware Image Context

Neural Radiance Fields (NeRFs) have recently emerged as a popular option for photo-realistic object capture due to their ability to faithfully capture high-fidelity volumetric content even from handheld video input. Although much research has been devoted to efficient optimization leading to real-time training and rendering, options for interactive editing NeRFs remain limited. We present a very simple but effective neural network architecture that is fast and efficient while maintaining a low memory footprint. This architecture can be incrementally guided through user-friendly image-based edits. Our representation allows straightforward object selection via semantic feature distillation at the training stage. More importantly, we propose a local 3D-aware image context to facilitate view-consistent image editing that can then be distilled into fine-tuned NeRFs, via geometric and appearance adjustments. We evaluate our setup on a variety of examples to demonstrate appearance and geometric edits and report 10-30x speedup over concurrent work focusing on text-guided NeRF editing. Video results and code can be found on our project webpage at https://proteusnerf.github.io.

Developing RPC-Net: Leveraging High-Density Electromyography and Machine Learning for Improved Hand Position Estimation.

The purpose of this study was to develop and evaluate the performance of RPC-Net (Recursive Prosthetic Control Network), a novel method using simple neural network architectures to translate electromyographic activity into hand position with high accuracy and computational efficiency. RPC-Net uses a regression-based approach to convert forearm electromyographic signals into hand kinematics. We tested the adaptability of the algorithm to different conditions and compared its performance with that of solutions from the academic literature. RPC-Net demonstrated a high degree of accuracy in predicting hand position from electromyographic activity, outperforming other solutions with the same computational cost. Including previous position data consistently improved results across subjects and conditions. RPC-Net showed robustness against a reduction in the number of electromyography electrodes used and shorter input signals, indicating potential for further reduction in computational cost. The results demonstrate that RPC-Net is capable of accurately translating forearm electromyographic activity into hand position, offering a practical and adaptable tool that may be accessible in clinical settings. The development of RPC-Net represents a significant advancement. In clinical settings, its application could enable prosthetic devices to be controlled in a way that feels more natural, improving the quality of life for individuals with limb loss.

SCOV-CNN: A Simple CNN Architecture for COVID-19 Identification Based on the CT Images

Since the coronavirus was first discovered in Wuhan, it has widely spread and was finally declared a global pandemic by the WHO. Image processing plays an essential role in examining the lungs of affected patients. Computed Tomography (CT) and X-ray images have been popularly used to examine the lungs of COVID-19 patients. This research aims to design a simple Convolution Neural Network (CNN) architecture called SCOV-CNN for the classification of the virus based on CT images and implementation on the web-based application. The data used in this work were CT images of 120 patients from hospitals in Brazil. SCOV-CNN was inspired by the LeNet architecture, but it has a deeper convolution and pooling layer structure. Combining seven and five kernel sizes for convolution and padding schemes can preserve the feature information from the images. Furthermore, it has three fully connected (FC) layers with a dropout of 0.3 on each. In addition, the model was evaluated using the sensitivity, specificity, precision, F1 score, and ROC curve values. The results showed that the architecture we proposed was comparable to some prominent deep learning techniques in terms of accuracy (0.96), precision (0.98), and F1 score (0.95). The best model was integrated into a website-based system to help and facilitate the users' activities. We use Python Flask Pam tools as a web server on the server side and JavaScript for the User Interface (UI) Design

Hybrid convolutional long short‐term memory models for sales forecasting in retail

AbstractThis study proposes novel sales forecasting approaches that merge deep learning methods in a hybrid model. Long short‐term memory (LSTM) is adopted for modeling the temporal characteristics of the data, whereas the convolutional neural network (CNN) focuses on identifying and extracting relevant exogenous information. We propose stacked (S‐CNN‐LSTM) and parallel (P‐CNN‐LSTM) hybrid architectures to understand complex time series data with varying seasonal patterns and multiple products correlations. The performance drivers of both architectures were empirically tested with a real‐world multivariate retail dataset and outperformed when compared with simple neural network architectures and standard autoregressive methods for short and long‐term forecasting horizons. When compared with traditional predictive approaches, the proposed hybrid models reduce the computational complexity while providing flexibility and robustness.

Simple CNN as an alternative for large pretrained models for medical image classification - MedMNIST case study

For real-life image classification problems, publicly available pretrained deep convolutional neural networks (CNN) are often applied. Such families of architectures as ResNet, VGG, or Inception are usually the first choice because they are widely used and can provide satisfying results. Though universal, such approaches can be too sophisticated and computationally expensive when fitting to certain medical image classification tasks, especially those where images are not varied and do not contain features learnt by the models in the pretraining phase. The question of whether some simplified models, dedicated to such tasks, can be designed is of great concern. We performed a case study, using three datasets from the MedMNIST data collection, to test neural network classifiers with simple architectures. This paper presents simple convolutional neural network architectures that perform for the considered data sets similarly to much more sophisticated pre-trained models and automatic ML environments.

Novel MobileNet based Multipath Convolutional Neural Network for defect detection in fabrics

Automatic fabric defect detection and classification is the most important process in the textile industry to ensure the fabric quality. In the existing systems, a learning based method is used for detecting defects in plain weave fabrics. In this paper, a novel MobileNet based Multipath Convolutional Neural Network (MMPCNN) architecture is proposed for detection and classification of simple and complex patterned fabric defects. In the proposed MMPCNN architecture, MobileNet model is used in the first path. In this, Gabor filter bank is used instead of conventional filters in the first convolution layer. A simple convolutional neural network architecture with Gray Level Co-occurrence Matrix (GLCM) features as an input is used in the second path of the MMPCNN architecture. Gabor filters are more useful for analyzing the texture with different orientations and scales. Each Gabor filter parameter has its own impact on analyzing the texture and extracting the information from the texture. Therefore, in this paper, the use of Gabor filter parameters in MMPCNN architecture is analyzed. The proposed model is experimented on the TILDA textile image database and it is able to achieve 100% accuracy with reduced trainable parameters for fabric defect detection and classification.

Market index price prediction using Deep Neural Networks with a Self-Similarity approach

Stock indexes are of vital importance to understand financial and economic markets of sectors and nations. Given the importance of market indexes, researchers, investors, and policy makers are continuously working to improve models to forecast market movements; small improvements in modeling have the potential for large financial gains. To this end, machine learning approaches have gained in popularity as both software and hardware performance have increased. In this work, we apply machine learning applications to the S&P 500, DAX, AEX and the SMI indexes to improve forecasting performance. In particular, we take advantage of the fractal and Self-Similarity behaviors that exist in the time series of these indexes using simple Recurrent Neural Network, Multilayer Perceptron, and Long-Short Term Memory architectures. We apply the architecture using 60-, 30-, and 15-min windows of daily series for each approach and index. The results indicate that for the S&P 500 the proposed self-similarity models outperform all base approaches. However, results are not uniform across all models and indexes. Overall, performance differences across models and indexes are presented. To check the difference between the forecasts of the self-similarity and base models, Model Confidence Set was used.

CRIMINAL FACIAL DETECTION USING DEEP LEARNING

Deep learning models have enhanced picture-based semantic pattern identification. Facial photos can assess a person's emotional state and character traits. Due to this motive, we are applying a variety of deep learning architectures to try to infer criminal tendency from face pictures. This study used a simple convolutional neural network (CNN) architecture and multiple pre-trained CNN designs, including VGG-16, VGG-19, and InceptionV3. We compared these systems' effectiveness in identifying criminal traits from human faces. Deep learning models were tested using the National Institute of Standards and Technology's public database. (NIST). Our initiative only used men's images to avoid misunderstanding. VGG CNN models performed best, even with low data, detecting criminal faces with 99.5% accuracy. This index includes Image Classification, Facial Images, Personality Traits, Semantic Pattern Recognition, Deep Learning, and Image Processing.

Human Activity Recognition in a Realistic and Multiview Environment Based on Two-Dimensional Convolutional Neural Network

Recognition of human activity based on convolutional neural network has received the interest of researchers in recent years due to its significant improvement in accuracy. A large number of algorithms based on the deep learning approach have been proposed for activity recognition purpose. However, with the increasing advancements in technologies having limited computational resources, it needs to design an efficient deep learning-based approaches with improved utilization of computational resources. This paper presents a simple and efficient 2-Dimensional convolutional neural network (2-D CNN) architecture with very small size convolutional kernel for human activity recognition. The merit of the proposed CNN architecture over standard deep learning architectures is fewer trainable parameters and lesser memory requirement which enables it to train the proposed CNN architecture on low GPU memory-based devices and also works well with smaller as well as larger size datasets. The proposed approach consists of mainly four stages: namely (1) creation of dataset and data augmentation, (2) designing 2-D convolutional neural network (CNN) architecture, (3) the proposed 2-D CNN architecture trained from scratch up to optimum stage, and (4) evaluation of the trained 2-D CNN architecture. To illustrate the effectiveness of the proposed architecture several extensive experiments are conducted on three publicly available datasets, namely IXMAS, YouTube, and UCF101 dataset. The results of the proposed method and its comparison with other state-of-the-art methods [8-12,14,18-26,29-33] demonstrate the usefulness of the proposed method.

Implicit Solutions of the Electrical Impedance Tomography Inverse Problem in the Continuous Domain with Deep Neural Networks.

Electrical impedance tomography (EIT) is a non-invasive imaging modality used for estimating the conductivity of an object Ω from boundary electrode measurements. In recent years, researchers achieved substantial progress in analytical and numerical methods for the EIT inverse problem. Despite the success, numerical instability is still a major hurdle due to many factors, including the discretization error of the problem. Furthermore, most algorithms with good performance are relatively time consuming and do not allow real-time applications. In our approach, the goal is to separate the unknown conductivity into two regions, namely the region of homogeneous background conductivity and the region of non-homogeneous conductivity. Therefore, we pose and solve the problem of shape reconstruction using machine learning. We propose a novel and simple jet intriguing neural network architecture capable of solving the EIT inverse problem. It addresses previous difficulties, including instability, and is easily adaptable to other ill-posed coefficient inverse problems. That is, the proposed model estimates the probability for a point of whether the conductivity belongs to the background region or to the non-homogeneous region on the continuous space Rd∩Ω with d∈{2,3}. The proposed model does not make assumptions about the forward model and allows for solving the inverse problem in real time. The proposed machine learning approach for shape reconstruction is also used to improve gradient-based methods for estimating the unknown conductivity. In this paper, we propose a piece-wise constant reconstruction method that is novel in the inverse problem setting but inspired by recent approaches from the 3D vision community. We also extend this method into a novel constrained reconstruction method. We present extensive numerical experiments to show the performance of the architecture and compare the proposed method with previous analytic algorithms, mainly the monotonicity-based shape reconstruction algorithm and iteratively regularized Gauss-Newton method.

Boosting auxiliary task guidance: a probabilistic approach

This work aims to introduce a novel approach for auxiliary task guidance (ATG). In this approach, our goal is to achieve effective guidance from a suitable auxiliary task by utilizing the uncertainty in calculated gradients for a mini-batch of samples. Our method calculates a probabilistic fitness factor of the auxiliary task gradient for each of the shared weights to guide the main task at every training step of mini-batch gradient descent. We have shown that this proposed factor incorporates task specific confidence of learning to manipulate ATG in an effective manner. For studying the potency of the method, monocular visual odometry (VO) has been chosen as an application. Substantial experiments have been done on the KITTI VO dataset for solving monocular VO with a simple convolutional neural network (CNN) architecture. Corresponding results show that our ATG method significantly boosts the performance of supervised learning for VO. It also out performs state-of-the-art (SOTA) auxiliary guided methods we applied for VO. The proposed method is able to achieve decent scores (in some cases competitive)compared to existing SOTA supervised monocular VO algorithms, while keeping an exceptionally low parameter space in supervised regime.

Improved estimation of rail–wheel contact forces from instrumented wheel-set data through higher harmonic cancellation and a back-propagation neural network scheme

Instrumented wheelsets, equipped with numbers of strain gauges, are widely used to estimate the rail–wheel contact parameters. The accuracy in estimation of rail–wheel contact parameters from the measured strain signals depends on the layout of the strain gauges and correlation techniques adopted for processing measured signals. Harmonic components estimation is the most widely used conventional correlation technique. However, its accuracy in estimation of rail–wheel contact parameters is limited to the order of 80%–90%. This study proposes a signal modification algorithm, initially utilizing cancellation of next significant higher harmonics than those considered in the existing algorithms and then further, a novel Artificial Neural Network architecture for the same purpose.The signal modification approach is based on the realization that the seventh harmonic has significant contribution on the strain signals. The proposed algorithm is aimed at estimating and cancelling the effects of the seventh harmonic in strain signals. The accuracy thus improves to 91% in vertical contact force estimates, while the accuracy in estimation of lateral contact force and lateral contact position are improved to 97% and 95%, respectively. Due to the highly non-linear nature of the rail–wheel contact parameters, an optimized nonlinear mapping is required for further improvement in the accuracy.The viability of using artificial neural networks in estimation of rail–wheel contact parameters have been explored earlier. The proposed architectures are complicated and require extensive manual optimization of tuning parameters. A simple back-propagation artificial neural network architecture, using Bayesian regularization training function with Levenberg–Marquardt optimization technique is proposed in this study. The proposed neural network consists of 18 measured strain signals as input, three hidden layers with 20, 6 and 3 neurons, respectively, and vertical force, lateral force and lateral contact position as three outputs. Accuracies of the order of 99%, 99% and 96% are achieved in the estimation of vertical contact force, lateral contact force and lateral contact position respectively. The input and training data is generated from the multi-body dynamics and finite element simulations.

Predicting asthma patients' total cost using neural networks and linear regression

In this paper, we analyze a population of asthma patients trying to predict the total cost of their treatment based on various demographic, clinical, and pharmacological data. We are comparing a neural network architecture with a simple linear regression using data from a healthcare insurance provider based in Louisiana. Our first focus was to explore the factors associated with the total cost of asthma treatment. Then we identified a sufficient threshold of data for which the neural networks outperform the linear regression models in terms of predictive accuracy. We showed that even with a simple Neural Network architecture, after approximately 6,000 randomly selected data points Neural Networks outperform Linear Regression almost always.

Efficient Neural Style Transfer for Volumetric Simulations

Artistically controlling fluids has always been a challenging task. Recently, volumetric Neural Style Transfer (NST) techniques have been used to artistically manipulate smoke simulation data with 2D images. In this work, we revisit previous volumetric NST techniques for smoke, proposing a suite of upgrades that enable stylizations that are significantly faster, simpler, more controllable and less prone to artifacts. Moreover, the energy minimization solved by previous methods is camera dependent. To avoid that, a computationally expensive iterative optimization performed for multiple views sampled around the original simulation is needed, which can take up to several minutes per frame. We propose a simple feed-forward neural network architecture that is able to infer view-independent stylizations that are three orders of the magnitude faster than its optimization-based counterpart.

Simple dilated convolutional neural network for quantitative modeling based on near infrared spectroscopy techniques

A simple deep convolutional neural network architecture had been developed and applied to the establishment of a quantitative model based on near infrared spectroscopy techniques. The network architecture only contained general convolutional layers and dilated convolutional layers. The determination coefficient was used as a criterion to stop the training process of the neural network. Three data sets were used to test the performance of the neural network. The major results are (1) quantitative model can be established within one million iterations; (2) the quantitative model established on one spectrometer can be applied to other spectrometers of same manufacturer and even the same kind of spectrometer of different manufacturer; (3) simultaneous quantitative analysis of four components in grain samples. This study provided a new strategy to apply techniques of deep neural network in quantitative analysis.

High-resolution downscaling with interpretable deep learning: Rainfall extremes over New Zealand

The gap in resolution between existing global climate model output and that sought by decision-makers drives an ongoing need for climate downscaling. Here we test the extent to which developments in deep learning can out-perform existing statistical approaches for downscaling historical rainfall in the highly complex terrain setting of New Zealand. While deep learning removes the need for manual feature selection when extracting spatial-temporal information from predictor fields, several key considerations need to be addressed. These include: the chosen complexity of the network architecture, suitable loss functions tailored to the problem, as well as input data considerations of domain size and amount of training data required to provide adequate out-of-sample generalization. Sensitivity testing to these considerations reveals that a relatively simple convolutional neural network (CNN) architecture with carefully selected loss functions can considerably outperform existing statistical downscaling models based on multiple linear regression with manual feature selection. When aggregated across the entire region, the fraction of explained variance on wet days increased from 0.35 to 0.52, the root-mean-squared error reduced by over 20% and percentage biases for the 90th percentile of rainfall improved by over 25%. Using interpretable machine learning methods, we demonstrate that the CNN has been capable of self-learning physically plausible relationships between the large-scale atmospheric environment and extreme localized rainfall events. The historical performance and physical interpretability documented here lends support for wider development and application of deep learning in climate downscaling.

Multifarious aspects of the chaos game representation and its applications in biological sequence analysis

Chaos game representation (CGR) has been successfully applied in bioinformatics for over 30 years. Since then, many further extensions were announced. Numerical encoding of biological sequences is especially convenient in the visualisation process, free-alignment methods and input preparation for machine learning techniques. The development and applications of CGR have embraced mainly linear nucleotide sequences. However, there were also some attempts to create a representation of proteins. The latter need to be more sophisticated, as arbitrary coordinates for amino acids do not reflect their properties which is crucial during the encoding process. In this paper, the authors summarised various variations of CGRs and their limitations. We began by studying the PROSITE motifs and showed the immense number of amino acid properties employed by different proteins. To this aim, we harnessed the Principal Component Analysis (PCA) and studied the relation between explained variance and the number of features that describe them. It appeared that even after many reductions, about 50 features are non-redundant. This was the reason we introduced an embedding concept from natural language processing which enables adjusting features for a given list of sequences. We presented a simple neural network architecture with one hidden layer and one neuron within it and showed it provides satisfactory results in phylogenetic tree construction in ND5 and SPARC protein cases. To this aim, we transformed CGR representations for all considered sequences using Discrete Fourier Transform (DFT) and applied Unweighted Pair Group Method with Arithmetic Mean (UPGMA) algorithm. Moreover, we indicated some similarities between CGR and Recurrent Neural Networks (RNN). In the end, we attempted to include information about the RNA secondary structure and defined some measures to validate biological significance. We studied their properties and showed on ALMV-3 example its usefulness.