Modern Neural Networks Research Articles

SSDSNet: A Dual-Stream State-Space Network for Efficient Image Denoising with Long-Range Dependency Modeling

Abstract In recent years, significant progress has been made in image denoising, driven by advancements in modern deep neural networks such as Convolutional Neural Networks
(CNNs) and Transformers. However, existing denoising models often face limitations due to inherent local inductive biases or the quadratic computational complexity of their architectures. Recently, Selective Structured State Space Models, exemplified by Mamba, have shown considerable potential in modeling long-range dependencies with linear complexity. Despite this promise, their application in low-level computer vision tasks remains underexplored. In this study, we introduce a straightforward yet effective model, the Dual-Stream State-Space Module (DSSM), specifically designed for image denoising. The DSSM serves as the core component of our model, combining convolutional operations with the
Mamba model to enhance the capability of capturing both local features and global context.
This dual-stream approach leverages the inherent structure of local image patches while effectively modeling long-range dependencies, resulting in feature representations tailored for denoising tasks. Extensive experiments demonstrate the effectiveness of our method. For instance, on the SIDD dataset, our model achieves performance on par with state-of-the-art methods, maintaining a global receptive field while operating with similar computational costs.

Generative Models for Grid-Based and Image-Based Pathfinding

Pathfinding is a challenging problem which generally asks to find a sequence of valid moves for an agent provided with a representation of the environment, i.e. a map, in which it operates. In this work, we consider pathfinding on binary grids and on image representations of the digital elevation models. In the former case, the transition costs are known, while in latter scenario, they are not. A widespread method to solve the first problem is to utilize a search algorithm that systematically explores the search space to obtain a solution. Ideally, the search should also be complemented with an informative heuristic to focus on the goal and prune the unpromising regions of the search space, thus decreasing the number of search iterations. Unfortunately, the widespread heuristic functions for grid-based pathfinding, such as Manhattan distance or Chebyshev distance, do not take the obstacles into account and in obstacle-rich environments demonstrate inefficient performance. As for pathfinding with image inputs, the heuristic search cannot be applied straightforwardly as the transition costs, i.e. the costs of moving from one pixel to the other, are not known. To tackle both challenges, we suggest utilizing modern deep neural networks to infer the instance-dependent heuristic functions at the pre-processing step and further use them for pathfinding with standard heuristic search algorithms. The principal heuristic function that we suggest learning is the path probability, which indicates how likely the grid cell (pixel) is lying on the shortest path (for binary grids with known transition costs, we also suggest another variant of the heuristic function that can speed up the search). Learning is performed in a supervised fashion (while we have also explored the possibilities of end-to-end learning that includes a planner in the learning pipeline). At the test time, path probability is used as the secondary heuristic for the Focal Search, a specific heuristic search algorithm that provides the theoretical guarantees on the cost bound of the resultant solution. Empirically, we show that the suggested approach significantly outperforms state-of-the-art competitors in a variety of different tasks (including out-of-the distribution instances).

Artificial intelligence in medicine: neural networks for analyzing systemic hemodynamics

Artificial neural networks are capable of efficiently processing large data sets, as well as solving the tasks of prediction, classification and data recovery. The article considers each of the above tasks in detail and studies literature sources devoted to the topic under study. Artificial neural networks cope with the tasks with a high degree of accuracy. The methods of application of neural networks for the analysis of systemic haemodynamics are described. Modern neural networks can analyse medical data and are able to work with incomplete data, find hidden patterns in them, and can be adapted to solve a wide range of problems. Our laboratory is developing an artificial neural network capable of classifying indicators describing the state of haemodynamics of subjects and recovering missing or incomplete data. Thus, artificial neural networks can act as an efficient method of analysing systemic hemodynamic parameters.

A Multi-label Artificial Intelligence Approach for Improving Breast Cancer Detection With Mammographic Image Analysis.

Breast cancer remains a major global health concern. This study aimed to develop a deep-learning-based artificial intelligence (AI) model that predicts the malignancy of mammographic lesions and reduces unnecessary biopsies in patients with breast cancer. In this retrospective study, we used deep-learning-based AI to predict whether lesions in mammographic images are malignant. The AI model learned the malignancy as well as margins and shapes of mass lesions through multi-label training, similar to the diagnostic process of a radiologist. We used the Curated Breast Imaging Subset of Digital Database for Screening Mammography. This dataset includes annotations for mass lesions, and we developed an algorithm to determine the exact location of the lesions for accurate classification. A multi-label classification approach enabled the model to recognize malignancy and lesion attributes. Our multi-label classification model, trained on both lesion shape and margin, demonstrated superior performance compared with models trained solely on malignancy. Gradient-weighted class activation mapping analysis revealed that by considering the margin and shape, the model assigned higher importance to border areas and analyzed pixels more uniformly when classifying malignant lesions. This approach improved diagnostic accuracy, particularly in challenging cases, such as American College of Radiology Breast Imaging-Reporting and Data System categories 3 and 4, where the breast density exceeded 50%. This study highlights the potential of AI in improving the diagnosis of breast cancer. By integrating advanced techniques and modern neural network designs, we developed an AI model with enhanced accuracy for mammographic image analysis.

Device placement using Laplacian PCA and graph attention networks

Abstract The exponential growth in data and parameters in modern neural networks has created the need to distribute these models across multiple devices for efficient training, resulting in the device placement problem. Existing graph encoding approaches for device placement suffer from low efficiency when searching for optimal parallel strategies, primarily due to suboptimal positional information retrieval. To address these challenges, we propose the Laplacian Principal Component Analysis-graph attention networks (LPCA-GAT) model. Firstly, we employ GAT to capture complex relationships between nodes and generate node encodings. Secondly, we leverage LPCA on the graph Laplacian matrix to extract crucial low-dimensional positional information. Finally, by integrating these two components, we obtain the final node encodings. This enhances the representation capability of nodes within the graph, enabling efficient device placement. The experimental results demonstrate that LPCA-GAT achieves superior device placement results, specifically accelerating execution and computation time by 13.24% and 96.38%, respectively, leading to significant improvements in both operational efficiency and performance.

What should an ordinary person understand about the work of generative artificial intelligence? Proceedings of the competition "TRIZformashka-2024"

Three basic aspects of the operation of generative neural network models (generative artificial intelligence) are discussed in the article: the concept of a "token", the probabilistic nature of the generated response, and the concept of a "large model", the size of which ensures the pseudo-intelligent behavior of neural network chatbots. The issues of implementing generative models, areas and methods of their application are not discussed in principle.The materials of the "TRIZformashka-2024" competition, which was dedicated to neural network models, are provided. The fact of pseudo-intelligence of generative models is demonstrated. It turns out that a model trained on a single phrase "mama myla ramu" ("mama washed the frame") and using a context of a single letter for generation can sometimes behave as if it knows the rules of declension in the Russian language and is able to change a word by case!The concept of a "token" is considered in relation to the generation of texts, pictures and passwords. On the basis of "tokens" a practically useful method of generating passwords is built, difficult to guess, but easy to reproduce (difficult to forget).The concept of a "large model" is presented clearly and intelligibly due to its "visualization" by comparing it with physical quantities. (If one parameter of a neural network weighed one gram, then 200 freight trains would be needed to transport it. If one parameter had a length one millimeter, then the neural network would revolve around the Earth at the equator 25 times. If one second was required to learn one parameter, then it would be necessary to start training a modern neural network in the times of the Cro-Magnons.)The materials will be useful for studying generative artificial intelligence at any age.

Efficient Privacy-Preserving Machine Learning with Lightweight Trusted Hardware

In this paper, we propose a new secure machine learning inference platform assisted by a small dedicated security processor, which will be easier to protect and deploy compared to today's TEEs integrated into high-performance processors. Our platform provides three main advantages over the state-of-the-art: (i) We achieve significant performance improvements compared to state-of-the-art distributed Privacy-Preserving Machine Learning (PPML) protocols, with only a small security processor that is comparable to a discrete security chip such as the Trusted Platform Module (TPM) or on-chip security subsystems in SoCs similar to the Apple enclave processor. In the semi-honest setting with WAN/GPU, our scheme is 4X-63X faster than Falcon (PoPETs'21) and AriaNN (PoPETs'22) and 3.8X-12X more communication efficient. We achieve even higher performance improvements in the malicious setting. (ii) Our platform guarantees security with abort against malicious adversaries under honest majority assumption. (iii) Our technique is not limited by the size of secure memory in a TEE and can support high-capacity modern neural networks like ResNet18 and Transformer. While previous work investigated the use of high-performance TEEs in PPML, this work represents the first to show that even tiny secure hardware with very limited performance can be leveraged to significantly speed-up distributed PPML protocols if the protocol can be carefully designed for lightweight trusted hardware.

Потенциальные угрозы, связанные с применением искусственного интеллекта в преступных действиях

The article is devoted to the issues of assessing the threats associated with the use of artificial intelligence in the implementation of criminal actions, taking into account the rapid development of technical capabilities and areas of application of modern neural networks, since the degree of readiness for counteraction directly affects the effectiveness of such counteraction and the efficiency of its application.

Lifelong Generative Adversarial Autoencoder.

Lifelong learning describes an ability that enables humans to continually acquire and learn new information without forgetting. This capability, common to humans and animals, has lately been identified as an essential function for an artificial intelligence system aiming to learn continuously from a stream of data during a certain period of time. However, modern neural networks suffer from degenerated performance when learning multiple domains sequentially and fail to recognize past learned tasks after being retrained. This corresponds to catastrophic forgetting and is ultimately induced by replacing the parameters associated with previously learned tasks with new values. One approach in lifelong learning is the generative replay mechanism (GRM) that trains a powerful generator as the generative replay network, implemented by a variational autoencoder (VAE) or a generative adversarial network (GAN). In this article, we study the forgetting behavior of GRM-based learning systems by developing a new theoretical framework in which the forgetting process is expressed as an increase in the model's risk during the training. Although many recent attempts have provided high-quality generative replay samples by using GANs, they are limited to mainly downstream tasks due to the lack of inference. Inspired by the theoretical analysis while aiming to address the drawbacks of existing approaches, we propose the lifelong generative adversarial autoencoder (LGAA). LGAA consists of a generative replay network and three inference models, each addressing the inference of a different type of latent variable. The experimental results show that LGAA learns novel visual concepts without forgetting and can be applied to a wide range of downstream tasks.

A Hybrid Approach for CT Image Noise Reduction Combining Method Noise-CNN and Shearlet Transform

The presence of gaussian noise commonly weakens the diagnostic precision of low-dose CT imaging. A novel CT image denoising technique that integrates the non-subsampled shearlet transform (NSST) with Bayesian thresholding, and incorporates a modern method noise Deep Convolutional neural network (DCNN) based post-processing operation on denoised images to strengthen low-dose CT imaging quality. The hybrid method commences with NSST and Bayesian thresholding to mitigate the initial noise while preserving crucial image features, such as corners and edges. The novel aspect of the proposed approach is its successive application of a DnCNN on initial denoised image, which learns and removes residual noise patterns from denoised images, thereby enhancing fine detail preservation. This dual-phase strategy addresses both noise suppression and image-detail preservation. The proposed technique is evaluated through the use of metrics, such as PSNR, SNR, SSIM, ED, and UIQI. The results demonstrate that the hybrid approach outperforms standard denoising techniques in preserving image quality and fine details.

Learnability of state spaces of physical systems is undecidable

Despite an increasing role of machine learning in science, there is a lack of results on limits of empirical exploration aided by machine learning. In this paper, we construct one such limit by proving undecidability of learnability of state spaces of physical systems. We characterize state spaces as binary hypothesis classes of the computable Probably Approximately Correct learning framework. This leads to identifying the first limit for learnability of state spaces in the agnostic setting. Further, using the fact that finiteness of the combinatorial dimension of hypothesis classes is undecidable, we derive undecidability for learnability of state spaces as well. Throughout the paper, we try to connect our formal results with modern neural networks. This allows us to bring the limits close to the current practice and make a first step in connecting scientific exploration aided by machine learning with results from learning theory.

ShuffleNeMt: modern lightweight convolutional neural network architecture

ShuffleNeMt: modern lightweight convolutional neural network architecture

Factorizers for distributed sparse block codes

Distributed sparse block codes (SBCs) exhibit compact representations for encoding and manipulating symbolic data structures using fixed-width vectors. One major challenge however is to disentangle, or factorize, the distributed representation of data structures into their constituent elements without having to search through all possible combinations. This factorization becomes more challenging when SBCs vectors are noisy due to perceptual uncertainty and approximations made by modern neural networks to generate the query SBCs vectors. To address these challenges, we first propose a fast and highly accurate method for factorizing a more flexible and hence generalized form of SBCs, dubbed GSBCs. Our iterative factorizer introduces a threshold-based nonlinear activation, conditional random sampling, and an ℓ ∞ -based similarity metric. Its random sampling mechanism, in combination with the search in superposition, allows us to analytically determine the expected number of decoding iterations, which matches the empirical observations up to the GSBC’s bundling capacity. Secondly, the proposed factorizer maintains a high accuracy when queried by noisy product vectors generated using deep convolutional neural networks (CNNs). This facilitates its application in replacing the large fully connected layer (FCL) in CNNs, whereby C trainable class vectors, or attribute combinations, can be implicitly represented by our factorizer having F-factor codebooks, each with C F fixed codevectors. We provide a methodology to flexibly integrate our factorizer in the classification layer of CNNs with a novel loss function. With this integration, the convolutional layers can generate a noisy product vector that our factorizer can still decode, whereby the decoded factors can have different interpretations based on downstream tasks. We demonstrate the feasibility of our method on four deep CNN architectures over CIFAR-100, ImageNet-1K, and RAVEN datasets. In all use cases, the number of parameters and operations are notably reduced compared to the FCL.

EcoFlow: Efficient Convolutional Dataflows on Low-Power Neural Network Accelerators

Dilated and transposed convolutions are widely used in modern convolutional neural networks (CNNs). These kernels are used extensively during CNN training and inference of applications such as image segmentation and high-resolution image generation. We find that commonly-used low-power CNN inference accelerators are <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">not</i> optimized for both these convolutional kernels. Dilated and transposed convolutions introduce significant zero padding when mapped to the underlying spatial architecture, significantly degrading performance and energy efficiency. Existing approaches that address this issue require significant design changes to the otherwise simple, efficient, and well-adopted architectures used to compute direct convolutions. To address this challenge, we propose EcoFlow, a new set of dataflows and mapping algorithms for dilated and transposed convolutions. These algorithms are tailored to execute efficiently on existing low-cost, small-scale spatial architectures and requires minimal changes to existing accelerators. At its core, EcoFlow eliminates zero padding through careful dataflow orchestration and data mapping tailored to the spatial architecture. We evaluate EcoFlow on CNN training workloads and Generative Adversarial Network (GAN) workloads. Experiments in our new cycle-accurate simulator show that, using a common CNN inference accelerator, EcoFlow 1) reduces end-to-end CNN training time between 7-85%, and 2) improves end-to-end GAN training performance between 29-42%, compared to state-of-the-art CNN dataflows.

Equivariant tensor network potentials

Abstract Machine-learning interatomic potentials (MLIPs) have made a significant contribution to the recent progress in the fields of computational materials and chemistry due to the MLIPs’ ability of accurately approximating energy landscapes of quantum-mechanical models while being orders of magnitude more computationally efficient. However, the computational cost and number of parameters of many state-of-the-art MLIPs increases exponentially with the number of atomic features. Tensor (non-neural) networks, based on low-rank representations of high-dimensional tensors, have been a way to reduce the number of parameters in approximating multidimensional functions, however, it is often not easy to encode the model symmetries into them. In this work we develop a formalism for rank-efficient equivariant tensor networks (ETNs), i.e. tensor networks that remain invariant under actions of SO(3) upon contraction. All the key algorithms of tensor networks like orthogonalization of cores and DMRG-based algorithms carry over to our equivariant case. Moreover, we show that many elements of modern neural network architectures like message passing, pulling, or attention mechanisms, can in some form be implemented into the ETNs. Based on ETNs, we develop a new class of polynomial-based MLIPs that demonstrate superior performance over existing MLIPs for multicomponent systems.

In Search of Beauty Formula

The main category of aesthetics — beauty is the focus of the study. Our epoch is a revision of some spiritual meanings both in the sphere of human life and in science and art. The aim of this paper is to derive a universal formula of beauty through the image of Venus on the basis of a comparative analysis of works of classical art, art of postmodernity, as well as to determine the possibilities of modern neural networks. Art of the 21st century is multifaceted, neuro-art is another of its aspects, artificial intelligence and Man of Art compete here. Neural network is a program built on the principle of organisation and functioning of nerve cells of a living organism. The neural network was tasked to represent the modern ideal of beauty in the image of Venus, similar to the way painters of previous epochs acted. Can modern neural networks create? Create spiritual values? Create beauty in art? In total, more than a thousand images were generated according to our requests, which allows us to draw certain conclusions about the representation of the image of Venus by artificial intelligence. An attempt to derive a formula of beauty for each period of art development and to find a universal one is presented. The modern understanding of beauty has absorbed all historical paradigms, but on a qualitative, essential level. Beauty is a subjective sensual perception of the appearance of an object that is capable of evoking positive emotions. It is not the beauty itself that is important, but the feeling of beauty. Beauty is the dominant value of being, which contains a harmonious combination of absolute values, what makes a person a Personality.

COMPARISON OF K-NEAREST NEIGHBOR AND NEURAL NETWORK FOR PREDICTION INTERNATIONAL VISITOR IN EAST JAVA

Tourism is one of the government's priority sectors for economic growth. East Java is one of Indonesia's provinces and is attractive to international visitors. International visitors will appreciate the natural beauty and multiculturalism offered by East Java. In this study, predictions of international visitor visits in East Java from the entrance of Juanda International Airport were carried out using k-NN (k-Nearest Neighbor) and a neural network. The dataset used is based on BPS statistics of Jawa Timur Province in the form of the number of international visitor arrivals from January 2000 to February 2024. The datasets were distributed by dividing the data into 70% for training data and 30% for testing data. The creation of the k-NN model is carried out using k-values 2 to 7. The creation of a modern neural network using hidden layers 1 to 3. The prediction results that were made using k-NN obtained optimal RMSE at k-values 2, resulting in an RMSE of 1594,674 or an error of 3,98%. Meanwhile, the prediction results that have been made using neural networks obtained optimal RMSE at two hidden layers, which resulted in an RMSE of 1873, 355 or an error of 4,68%. So, it is recommended that the k-NN algorithm be used to predict the number of international visitors in East Java. The results of this study can be used to provide quantitative information for the government and stakeholders in adjusting the program to the development of international visitors visiting East Java.

An Empirical Study on the Effect of Training Data Perturbations on Neural Network Robustness.

The vulnerability of modern neural networks to random noise and deliberate attacks has raised concerns about their robustness, particularly as they are increasingly utilized in safety- and security-critical applications. Although recent research efforts were made to enhance robustness through retraining with adversarial examples or employing data augmentation techniques, a comprehensive investigation into the effects of training data perturbations on model robustness remains lacking. This paper presents the first extensive empirical study investigating the influence of data perturbations during model retraining. The experimental analysis focuses on both random and adversarial robustness, following established practices in the field of robustness analysis. Various types of perturbations in different aspects of the dataset are explored, including input, label, and sampling distribution. Single-factor and multi-factor experiments are conducted to assess individual perturbations and their combinations. The findings provide insights into constructing high-quality training datasets for optimizing robustness and recommend the appropriate degree of training set perturbations that balance robustness and correctness, and contribute to understanding model robustness in deep learning and offer practical guidance for enhancing model performance through perturbed retraining, promoting the development of more reliable and trustworthy deep learning systems for safety-critical applications.

Question Answering System Based on Bideirectional Long-Short-Term Memory (Bilstm)

In the modern world, Q&amp;A systems are essential for promoting better communication between people and technology. These systems play an important role in collecting information quickly and efficiently, and this leads to great progress in learning, teaching and development in many areas of life. Using deep learning techniques, this research addresses the problem of excellent prediction of the questions that need to be answered. We created a question-and-answer system using "Bidirectional Long Short-Term Memory (BiLSTM)", a modern neural network known for its accuracy and results in text analysis and natural language understanding. This technique is more effective in understanding questions and producing very accurate answers because of its special ability to pay attention to preceding and following information in a sentence. Preprocessing was used to remove unnecessary, unimportant and time-consuming data. The "Stanford Question Answering Dataset version 2 (SQuAD 2.0)"was used, which is considered one of the important datasets used in the field of machine learning and natural language processing. The following evaluation metrics were used to evaluate the model’s performance: “Mean Average Precision(MAP), Mean Reciprocal Rank(MRR), Recall, Precision, Loss, F1 Score, and Exact Match (EM). The results, based on 150 epochs (EPOGs) and 128 batch sizes with a cleaned dataset split into 70% training and 30% test/validation (15% each), are as follows:"Precision (0.966), Loss (0.591), F1-score (0.966), Recall (0.967), EM (0.967), MRR(0.918), MAP (0.776), and accuracy(0.966)". Interestingly, the highest performance was observed when using the accuracy measure.

Learning Robust Shape Regularization for Generalizable Medical Image Segmentation.

Generalizable medical image segmentation enables models to generalize to unseen target domains under domain shift issues. Recent progress demonstrates that the shape of the segmentation objective, with its high consistency and robustness across domains, can serve as a reliable regularization to aid the model for better cross-domain performance, where existing methods typically seek a shared framework to render segmentation maps and shape prior concurrently. However, due to the inherent texture and style preference of modern deep neural networks, the edge or silhouette of the extracted shape will inevitably be undermined by those domain-specific texture and style interferences of medical images under domain shifts. To address this limitation, we devise a novel framework with a separation between the shape regularization and the segmentation map. Specifically, we first customize a novel whitening transform-based probabilistic shape regularization extractor namely WT-PSE to suppress undesirable domain-specific texture and style interferences, leading to more robust and high-quality shape representations. Second, we deliver a Wasserstein distance-guided knowledge distillation scheme to help the WT-PSE to achieve more flexible shape extraction during the inference phase. Finally, by incorporating domain knowledge of medical images, we propose a novel instance-domain whitening transform method to facilitate a more stable training process with improved performance. Experiments demonstrate the performance of our proposed method on both multi-domain and single-domain generalization.