Simple Neural Network Research Articles

Conv2Former: A Simple Transformer-Style ConvNet for Visual Recognition.

Vision Transformers have been the most popular network architecture in visual recognition recently due to the strong ability of encode global information. However, its high computational cost when processing high-resolution images limits the applications in downstream tasks. In this paper, we take a deep look at the internal structure of self-attention and present a simple Transformer style convolutional neural network (ConvNet) for visual recognition. By comparing the design principles of the recent ConvNets and Vision Transformers, we propose to simplify the self-attention by leveraging a convolutional modulation operation. We show that such a simple approach can better take advantage of the large kernels ( ≥ 7×7) nested in convolutional layers and we observe a consistent performance improvement when gradually increasing the kernel size from 5×5 to 21×21. We build a family of hierarchical ConvNets using the proposed convolutional modulation, termed Conv2Former. Our network is simple and easy to follow. Experiments show that our Conv2Former outperforms existent popular ConvNets and vision Transformers, like Swin Transformer and ConvNeXt in all ImageNet classification, COCO object detection and ADE20 k semantic segmentation.

Automated MRI quantification of pediatric abdominal adipose tissue using convolutional neural networks and novel total intensity maps

ObjectiveChildhood obesity is a significant global health concern. Visceral adipose tissue (VAT) is more closely linked to the development of metabolic and cardiovascular diseases than abdominal subcutaneous adipose tissue (ASAT). This study aimed to develop a straightforward and automated method to quantify both ASAT and VAT in children from MRI. MethodsDixon-based 3 T-MRI scans were conducted on Mexican boys aged 7–9 years. Novel Total Intensity Maps were generated from these scans for input into simple 2D Convolutional Neural Networks (CNNs) for automatic prediction/quantification of ASAT and VAT. The results were compared with the commercial semi-automated quantification and validated method provided by AMRA® Researcher which is the gold standard. ResultsThe CNN-based method produced ASAT and VAT quantifications comparable to those obtained by AMRA® Researcher (ASAT, p = 0.1765; VAT, p = 0.7757), with a high correlation between the two methods for ASAT (R2 = 0.98, p < 10−14) and VAT (R2 = 0.95, p < 10−11). The reproducibility coefficients (RPC) were 0.32 L for ASAT and 0.17 L for VAT with our novel methodology. ConclusionThis study presents a simple automated method for quantifying ASAT and VAT in children from MRI data using 2D-CNN analysis of novel inputs, Total Intensity Maps. The new method produced measurements comparable to those obtained by AMRA® Researcher on the same dataset. SignificanceThe proposed automatic and straightforward quantification method, with minimal computational resource requirements, may serve as an accessible tool for researchers and clinicians in the diagnosis, follow-up, and prevention of childhood obesity.

Neural network modelling of kinematic and dynamic features for signature verification

Online signature parameters, which are based on human characteristics, broaden the applicability of an automatic signature verifier. Although kinematic and dynamic features have previously been suggested, accurately measuring features such as arm and forearm torques remains challenging. We present two approaches for estimating angular velocities, angular positions, and force torques. The first approach involves using a physical UR5e robotic arm to reproduce a signature while capturing those parameters over time. The second method, a cost-effective approach, uses a neural network to estimate the same parameters. Our findings demonstrate that a simple neural network model can extract effective parameters for signature verification. Training the neural network with the MCYT300 dataset and cross-validating with other databases, namely, BiosecurID, Visual, Blind, OnOffSigDevanagari-75 and OnOffSigBengali-75 confirm the model’s generalization capability. The trained model is available at: https://github.com/gvessio/SignatureKinematics.

The risk of shortcutting in deep learning algorithms for medical imaging research

While deep learning (DL) offers the compelling ability to detect details beyond human vision, its black-box nature makes it prone to misinterpretation. A key problem is algorithmic shortcutting, where DL models inform their predictions with patterns in the data that are easy to detect algorithmically but potentially misleading. Shortcutting makes it trivial to create models with surprisingly accurate predictions that lack all face validity. This case study shows how easily shortcut learning happens, its danger, how complex it can be, and how hard it is to counter. We use simple ResNet18 convolutional neural networks (CNN) to train models to do two things they should not be able to do: predict which patients avoid consuming refried beans or beer purely by examining their knee X-rays (AUC of 0.63 for refried beans and 0.73 for beer). We then show how these models’ abilities are tied to several confounding and latent variables in the image. Moreover, the image features the models use to shortcut cannot merely be removed or adjusted through pre-processing. The end result is that we must raise the threshold for evaluating research using CNNs to proclaim new medical attributes that are present in medical images.

GWO-elman based composite detection of transformer oil flow temperature

This paper addresses the issue of nonlinear coupling error caused by cross-sensitivity of strain and temperature in the composite detection of flow rate and temperature in transformer oil using a Fiber Bragg Grating (FBG) sensor. This paper focuses on the FBG sensor as the research object, elucidating the underlying principles of composite detection. Subsequently, a composite detection experimental platform is established for the purpose of analyzing the coupling error of flow rate and temperature. Ultimately, a nonlinear decoupling algorithm based on the Grey Wolf Optimizer (GWO) is proposed to enhance the nonlinear decoupling algorithm of the Elman neural network (simple recurrent neural network, Elman). The findings demonstrate that within the flow rate range of 0–5 m/s and the temperature range of 30 °C–150 °C, the maximum error is reduced by 72.0 % and 81.3 %, and the average error is reduced by 74.4 % and 79.4 %. This significantly enhances the precision and reliability of the sensor’s detection capabilities.

Point spread function deconvolution using a convolutional autoencoder

A major issue in optical astronomical image analysis is the combined effect of the instrument’s point spread function (PSF) and the atmospheric seeing that blurs images and changes their shape in a way that is band and time-of-observation dependent. In this work we present a very simple neural network based approach to nonblind image deconvolution that relies on feeding a convolutional autoencoder (CAE) input images that have been preprocessed by convolution with the corresponding PSF and its regularized inverse, a method which is both conceptually simple and computationally less intensive. We also present here, a new approach for dealing with limited input dynamic range of neural networks compared to the dynamic range present in astronomical images. Published by the American Physical Society 2024

SimSANet: a simple sequential attention-aided deep neural network for vehicle make and model recognition

SimSANet: a simple sequential attention-aided deep neural network for vehicle make and model recognition

The HYBRID CONTENT-BASED FILTERING AND CLASSIFICATION RNN WITH PARTICLE SWARM OPTIMIZATION FOR TOURISM RECOMMENDATION SYSTEM

Economic recovery in the tourism sector after the COVID-19 pandemic is one of the main focuses of the Indonesian government at the moment, especially in Bandung City. This research aims to develop a personalized tourist spot recommendation system, by addressing the gaps in the existing literature through the integration of Content-Based Filtering (CBF) and Simple Recurrent Neural Network (RNN) methods that aim to improve recommendation accuracy. This study uses a hybrid approach that combines Term Frequency - Inverse Document Frequency (TF-IDF) and word embedding with the Robustly Optimized BERT (RoBERTa) model to identify similarities between tourist destinations based on their content characteristics. Simple RNN is used to analyze user preference patterns over time, which is then further optimized using Particle Swarm Optimization (PSO). As a result, the Simple RNN model that has been optimized with PSO shows an increased accuracy of up to 94.37%, outperforming other optimizations such as Adam and SGD. This research makes a novel contribution by applying advanced machine learning techniques to improve personalization in travel recommendation systems.

Enhanced Tomato Disease Classification Using Hybrid Deep Learning Approach

Tomato disease classification is important for agricultural productivity and food security, yet achieving high accuracy stays challenging due to the complexity of disease symptoms. This study proposed a model classification utilizing DenseNet121 combined with an Autoencoder for feature extraction and Simple neural network (SNN) for classifier. The dataset contains 10 classes of tomato diseases. Experimental results shows that the proposed approach achieved a high accuracy of 99%, outperforming the performance of several approaches. The integration of the Autoencoder enhances feature extraction and dimensionality reduction, important for improved classification outcomes. This study highlights the efficacy of the proposed work in achieving state-of-the-art accuracy in tomato disease detection. Future work may focus on refining the model using advanced augmentation techniques and expanding the dataset to enhance its applicability across diverse agricultural conditions.

Prediction and evaluation of key parameters in coalbed methane pre-extraction based on transformer and inversion model

Accurate parameter prediction in the coalbed methane (CBM) pre-extraction process is crucial for formulating effective control measures and preventing CBM-related accidents. Traditional prediction methods rely on feature extraction or complex physical model parameter calculations, which require extensive manual intervention and have limited practical applicability. Additionally, simple neural network methods are prone to overfitting and gradient vanishing when handling parameters, and they lack the capability to dynamically monitor gas pressure during extraction, leading to inefficient and blind extraction operations. This study proposes a CBM pre-extraction parameter and completion time prediction method based on the Transformer model. By integrating autoregressive models and wavelet denoising techniques, the approach effectively captures temporal features and long-term dependencies in CBM data. Experimental results demonstrate that the proposed model outperforms traditional methods in short-, medium-, and long-term predictions, with a median R2 value of 0.99072, and 76% of the training results exceeding 0.9. Furthermore, a CBM pressure inversion model was developed, combining dimensional analysis and physical similarity principles with the Transformer model, enabling the dynamic detection of high- and low-pressure regions in coal seams. In single borehole compliance time predictions, the median compliance time for the first stage is 4 days, with an average of 49 days and a maximum of 277 days, providing adjustment guidance for boreholes with extended compliance times. The proposed model significantly improves prediction accuracy and stability, offering critical support for developing scientifically sustainable pre-extraction plans and advancing intelligent CBM management.

Predicting rainfall using machine learning, deep learning, and time series models across an altitudinal gradient in the North-Western Himalayas

Predicting rainfall is a challenging and critical task due to its significant impact on society. Timely and accurate predictions are essential for minimizing human and financial losses. The dependence of approximately 60% of agricultural land in India on monsoon rainfall implies the crucial nature of accurate rainfall prediction. Precise rainfall forecasts can facilitate early preparedness for disasters associated with heavy rains, enabling the public and government to take necessary precautions. In the North-Western Himalayas, where meteorological data are limited, the need for improved accuracy in traditional modeling methods for rainfall forecasting is pressing. To address this, our study proposes the application of advanced machine learning (ML) algorithms, including random forest (RF), support vector regression (SVR), artificial neural network (ANN), and k-nearest neighbour (KNN) along with various deep learning (DL) algorithms such as long short-term memory (LSTM), bi-directional LSTM, deep LSTM, gated recurrent unit (GRU), and simple recurrent neural network (RNN). These advanced techniques hold the potential to significantly improve the accuracy of rainfall prediction, offering hope for more reliable forecasts. Additionally, time series techniques, including autoregressive integrated moving average (ARIMA) and trigonometric, Box-Cox transform, arma errors, trend, and seasonal components (TBATS), are proposed for predicting rainfall across the altitudinal gradients of India’s North-Western Himalayas. This approach can potentially revolutionise how we approach rainfall forecasting, ushering in a new era of accuracy and reliability. The effectiveness and accuracy of the proposed algorithms were assessed using meteorological data obtained from six weather stations at different elevations spanning from 1980 to 2021. The results indicate that DL methods exhibit the highest accuracy in predicting rainfall, as measured by the root mean squared error (RMSE) and mean absolute error (MAE), followed by ML algorithms and time series techniques. Among the DL algorithms, the accuracy order was bi-directional LSTM, LSTM, RNN, deep LSTM, and GRU. For the ML algorithms, the accuracy order was ANN, KNN, SVR, and RF. These findings suggest that altitude significantly affects the accuracy of the models, highlighting the need for additional weather stations in this mountainous region to enhance the precision of rainfall prediction.

Predicting Mutual Fund Stress Levels Utilizing SEBI’s Stress Test Parameters in MidCap and SmallCap Funds Using Deep Learning Models

Abstract: The Association of Mutual Funds of India (AMFI), under the direction of the Securities and Exchange Board of India (SEBI), provided open access to various risk parameters with respect to MidCap and SmallCap funds for the first time from February 2024. Our study utilizes AMFI datasets from February 2024 to September 2024 which consisted of 14 variables. Among these, the primary variable identified in grading mutual funds is the stress test parameter, expressed as number of days required to liquidate between 50% and 25% of the portfolio, respectively, on a pro-rata basis under stress conditions as a response variable. The objective of our paper is to build and test various neural network models which can help in predicting stress levels with the highest accuracy and specificity in MidCap and SmallCap mutual funds based on AMFI’s 14 parameters as predictors. The results suggest that the simpler neural network model architectures show higher accuracy. We used Artificial Neural Networks (ANN) over other machine learning methods due to its ability to analyze the impact of dynamic interrelationships among 14 variables on the dependent variable, independent of the statistical distribution of parameters considered. Predicting stress levels with the highest accuracy in MidCap and SmallCap mutual funds will benefit investors by reducing information asymmetry while allocating investments based on their risk tolerance. It will help policy makers in designing controls to protect smaller investors and provide warnings for funds with unusually high risk.

A Simple Neural Network for Estimating Fine Sediment Sources Using XRF and XRD

Suspended sediment (SS) has a wide range of negative effects such as increased water turbidity, altered habitat structures, sedimentation, and effects on hydraulic systems and environmental engineering projects. Nevertheless, the methods for accurately determining SS sources on a basin-scale are poorly understood. Herein, we used a simplified neural network analysis (NNA) model to identify the sources of SS in Japan’s Oromushi River Catchment Basin. Fine soil samples were collected from different locations of the catchment basin, processed, and separately analysed using X-ray fluorescence (XRF) and X-ray diffraction (XRD). The sampling stations were grouped according to the type of soil cover, vegetation type and land-use pattern. The geochemical components of each group were fed into the same neural network layer, and a series of equations were applied to estimate the sediment contribution from each group to the downstream side of the river. Samples from the same sampling locations were also analysed by XRD, and the obtained peak intensity values were used as the input in the NNA model. SS mainly originated from agricultural fields, with regions where the ground is covered with volcanic ash identified as the key sources through XRF and XRD analysis, respectively. Therefore, based on the nature of the surface soil cover and the land use pattern in the catchment basin, NNA was found to be a reliable data analytical technique. Moreover, XRD analysis does not incorporate carbon, and also provides detailed information on crystalline phases. The results obtained in this study, therefore, do not depend on seasonal uncertainty due to organic matter.

Analysis of the Hyperparameter Selection in Machine Learning

Abstract. A Machine learning is now playing a role in various fields. With the development of the internet, the volume of data is increasing, and the algorithms of machine learning are becoming more complex. Adjusting hyperparameters is a challenge for beginners who are new to machine learning. This research implemented a simple neural network model through code. Moreover, this study has changed some of the most basic parameters of the model and trained different models. This study has used visualization methods to show how the model training time and performance change with the alteration of hyperparameters, and derived their respective accuracy rates. This article provides entry-level data and code for newcomers to the field of machine learning. Since this experiment uses a federated learning model, employs local_update SGD, and also utilizes parallel methods, it is quite comprehensive, covering many algorithm codes for machine learning beginners. Testing the basic parameters of the model can help newcomers to machine learning understand more clearly the impact of hyperparameters on the model. It also aids in the development of more algorithms for adjusting hyperparameters.

Research on Autonomous Navigation of Unmanned Aerial Vehicles Based on Correlation-Based Image Comparison Methods

Relevance. Autonomous navigation of unmanned aerial vehicles (UAVs) is one of the key challenges in the modern aerospace industry. Specifically, for small UAVs, the task of autonomous navigation becomes even more complex due to limitations in computational resources and sensor capabilities. Optimizing navigation methods under such conditions is a pressing issue that requires solutions capable of providing high performance with minimal resource consumption.Objective. Improving the efficiency of autonomous navigation for small UAVs by using correlation methods for image comparison. Achieving this objective is linked to the development and evaluation of algorithms that provide high-speed and accurate navigation with limited computational resources. The work uses methods such as the autocorrelation function, Pearson correlation, the structural similarity index, and a simple neural network for image comparison.Solution. The research showed that the autocorrelation-based approach demonstrates the best performance under low computational resources. It ensures high-speed processing of the entire image and shows optimal detection accuracy. Compared to other methods presented in the study, the autocorrelation approach is capable of working not only with noise in the "reference" map but also of using alternative areas with altered patterns as detected regions.The scientific novelty of the study is determined by the systematic comparison of various methods applied to the task of image comparison for small UAVs with limited computational resources. Unlike well-known works in the field of correlation-extreme systems, this research focuses on the use of a "reference map" and a search area, representing two different images of the same terrain taken from different sources. This is a key difference, as most methods are not highly efficient in processing such images where patterns may differ significantly.The practical significance of the developed algorithm lies in the fact that the proposed autocorrelation-based method can be used by developers of autonomous small UAV control systems to reduce computational load and increase data processing speed.

The role of motor cortex in motor sequence execution depends on demands for flexibility.

The role of the motor cortex in executing motor sequences is widely debated, with studies supporting disparate views. Here we probe the degree to which the motor cortex's engagement depends on task demands, specifically whether its role differs for highly practiced, or 'automatic', sequences versus flexible sequences informed by external cues. To test this, we trained rats to generate three-element motor sequences either by overtraining them on a single sequence or by having them follow instructive visual cues. Lesioning motor cortex showed that it is necessary for flexible cue-driven motor sequences but dispensable for single automatic behaviors trained in isolation. However, when an automatic motor sequence was practiced alongside the flexible task, it became motor cortex dependent, suggesting that an automatic motor sequence fails to consolidate subcortically when the same sequence is produced also in a flexible context. A simple neural network model recapitulated these results and offered a circuit-level explanation. Our results critically delineate the role of the motor cortex in motor sequence execution, describing the conditions under which it is engaged and the functions it fulfills, thus reconciling seemingly conflicting views about motor cortex's role in motor sequence generation.

Flow2GNN: Flexible Two-Way Flow Message Passing for Enhancing GNNs Beyond Homophily.

Message passing (MP) is crucial for effective graph neural networks (GNNs). Most local message-passing schemes have been shown to underperform on heterophily graphs due to the perturbation of updated representations caused by local redundant heterophily information. However, our experiment findings indicate that the distribution of heterophily information during MP can be disrupted by disentangling local neighborhoods. This finding can be applied to other GNNs, improving their performance on heterophily graphs in a more flexible manner compared to most heterophily GNNs with complex designs. This article proposes a new type of simple message-passing neural network called Flow2GNN. It uses a two-way flow message-passing scheme to enhance the ability of GNNs by disentangling and redistributing heterophily information in the topology space and the attribute space. Our proposed message-passing scheme consists of two steps in topology space and attribute space. First, we introduce a new disentangled operator with binary elements that disentangle topology information in-flow and out-flow between connected nodes. Second, we use an adaptive aggregation model that adjusts the flow amount between homophily and heterophily attribute information. Furthermore, we rigorously prove that disentangling in message-passing can reduce the generalization gap, offering a deeper understanding of how our model enhances other GNNs. The extensive experiment results show that the proposed model, Flow2GNN, not only outperforms state-of-the-art GNNs, but also helps improve the performance of other commonly used GNNs on heterophily graphs, including GCN, GAT, GCNII, and H2GCN, specifically for GCN, with up to a 25.88% improvement on the Wisconsin dataset.

Predicting Movie Box Office Based on Machine Learning, Deep Learning, and Statistical Methods

Abstract. The global film industry has been proved to impose a significant impact on both culture and the economy, with box office revenue serving as a crucial indicator of a film's commercial success. This study utilizes data from the Kaggle "TMDB Box Office Prediction" competition, encompassing 3,000 films released between 1990 and 2018, to predict movie box office revenue using Random Forest, XGBoost algorithms, and deep learning models such as Bidirectional Long Short-Term Memory (Bidirectional LSTM) and Simple Recurrent Neural Network (SimpleRNN). The goal is to develop a model that accurately predicts movie box office. By comprehensively considering multiple factors such as budget, popularity and film characteristics, this study not only significantly improves the accuracy of box office prediction, but also provides a scientific basis for the formulation of film market strategies. The results demonstrate that the Bidirectional LSTM excels in handling complex time-series data, showing strong trend-capturing capabilities, while XGBoost exhibits greater robustness in dealing with complex data and outliers. These findings can not only provide guidance for making more effective strategies on film production and distribution, but also provide new directions for future research, such as delving into the impact of social media on box office and developing more sophisticated predictive models to adapt to changing market dynamics.

Neural Network-Based Optimization of LEO Transfers

This study investigates the application of neural networks to the evaluation of minimum-time low-thrust transfers in low Earth orbit. The findings demonstrate the effectiveness of utilizing costates to regularize the training loss, significantly enhancing the accuracy of the predictions of the neural networks, even when working with limited datasets. Remarkably precise estimates of transfer times are achieved by training the regularized networks on datasets comprising one million samples. The incorporation of a warm-started guess strategy, involving simpler neural networks to provide transfer time and costates predictions for new transfers, accelerates the data collection process, making this approach highly practical for real-world applications. Overall, the methodology employed in this research study holds significant promise for low-thrust space missions, particularly when the evaluation of multiple minimum-time transfers is necessary in mission planning. In fact, the trained neural networks significantly speed up convergence when solving optimal control problems with indirect optimization methods. Furthermore, the remarkable accuracy in estimating both minimum transfer times and costates provides the flexibility of relying entirely on neural networks for determining minimum time.

Methodology for the automated selection of time-frequency representations

Data preprocessing is a key step in extracting useful information from sound and vibration data and often involves selecting a time-frequency representation. No single time-frequency representation is always optimal, and no standard method exists for selecting the appropriate time-frequency representation, so selecting the time-frequency representation requires expert knowledge and is susceptible to human bias. To address this, this work introduces a methodology to automate the selection of a time-frequency representation for a dataset using only a subset of the healthy, or normal, class of data. To select the parameters for each type of time-frequency representation, Bayesian optimization is used. With a candidate from each type of time-frequency representation, the average similarity is used to select the final candidate. Additionally, the use of multiple time-frequency representations within a single model is explored. Because there is currently no objective method to compare the selected time frequency representations against, the proposed methodology is evaluated in two case studies. In the case studies, the time frequency representations are used as inputs to a simple convolutional neural network that achieved 100% accuracy in classifying bearing faults and 94% accuracy in classifying the contact tip to workpiece distance in wire arc additive manufacturing. Additionally, the proposed methodology presents a 75% and 94% reduction in the data size for the two case studies. This offers further benefits for reducing costs of data transmission and storage in modern digital manufacturing architectures.