Architecture Model Research Articles

Performance comparison of state-of-the-art deep learning model architectures in Indonesian food image classification

Performance comparison of state-of-the-art deep learning model architectures in Indonesian food image classification

MemberShield: A framework for federated learning with membership privacy

Federated Learning (FL) allows multiple data owners to build high-quality deep learning models collaboratively, by sharing only model updates and keeping data on their premises. Even though FL offers privacy-by-design, it is vulnerable to membership inference attacks (MIA), where an adversary tries to determine whether a sample was included in the training data. Existing defenses against MIA cannot offer meaningful privacy protection without significantly hampering the model’s utility and causing a non-negligible training overhead. In this paper we analyze the underlying causes of the differences in the model behavior for member and non-member samples, which arise from model overfitting and facilitate MIAs. Accordingly, we propose MemberShield, a generalization-based defense method for MIAs that consists of: (i) one-time preprocessing of each client’s training data labels that transforms one-hot encoded labels to soft labels and eventually exploits them in local training, and (ii) early stopping the training when the local model’s validation accuracy does not improve on that of the global model for a number of epochs. Extensive empirical evaluations on three widely used datasets and four model architectures demonstrate that MemberShield outperforms state-of-the-art defense methods by delivering substantially better practical privacy protection against all forms of MIAs, while better preserving the target model utility. On top of that, our proposal significantly reduces training time and is straightforward to implement, by just tuning a single hyperparameter.

Polytopic Autoencoders with Smooth Clustering for Reduced-order Modelling of Flows

With the advancement of neural networks, there has been a notable increase, both in terms of quantity and variety, in research publications concerning the application of autoencoders to reduced-order models. We propose a polytopic autoencoder architecture that includes a lightweight nonlinear encoder, a convex combination decoder, and a smooth clustering network. Supported by several proofs, the model architecture ensures that all reconstructed states lie within a polytope, accompanied by a metric indicating the quality of the constructed polytopes, referred to as polytope error. Additionally, it offers a minimal number of convex coordinates for polytopic linear-parameter varying systems while achieving acceptable reconstruction errors compared to proper orthogonal decomposition (POD). To validate our proposed model, we conduct simulations involving two flow scenarios with the incompressible Navier-Stokes equation. Numerical results demonstrate the guaranteed properties of the model, low reconstruction errors compared to POD, and the improvement in error using a clustering network.

A comprehensive review of advances in physics-informed neural networks and their applications in complex fluid dynamics

Physics-informed neural networks (PINNs) represent an emerging computational paradigm that incorporates observed data patterns and the fundamental physical laws of a given problem domain. This approach provides significant advantages in addressing diverse difficulties in the field of complex fluid dynamics. We thoroughly investigated the design of the model architecture, the optimization of the convergence rate, and the development of computational modules for PINNs. However, efficiently and accurately utilizing PINNs to resolve complex fluid dynamics problems remain an enormous barrier. For instance, rapidly deriving surrogate models for turbulence from known data and accurately characterizing flow details in multiphase flow fields present substantial difficulties. Additionally, the prediction of parameters in multi-physics coupled models, achieving balance across all scales in multiscale modeling, and developing standardized test sets encompassing complex fluid dynamic problems are urgent technical breakthroughs needed. This paper discusses the latest advancements in PINNs and their potential applications in complex fluid dynamics, including turbulence, multiphase flows, multi-field coupled flows, and multiscale flows. Furthermore, we analyze the challenges that PINNs face in addressing these fluid dynamics problems and outline future trends in their growth. Our objective is to enhance the integration of deep learning and complex fluid dynamics, facilitating the resolution of more realistic and complex flow problems.

Fake News Detection Using Optimized Convolutional Neural Network and Bidirectional Long Short-Term Memory

The spread of fake news in the digital age threatens the integrity of online information, influences public opinion, and creates confusion. This study developed and tested a fake news detection model using an enhanced CNN-BiLSTM architecture with GloVe word embedding techniques. The WELFake dataset comprising 72,000 samples was used, with training and testing data ratios of 90:10, 80:20, and 70:30. Preprocessing involved GloVe 100-dimensional word embedding, tokenization, and stopword removal. The CNN-BiLSTM model was optimized with hyperparameter tuning, achieving an accuracy of 96%. A larger training data ratio demonstrated better performance. Results indicate the effectiveness of this model in distinguishing fake news from real news. This study shows that the CNN-BiLSTM architecture with GloVe embedding can achieve high accuracy in fake news detection, with recommendations for further research to explore preprocessing techniques and alternative model architectures for further improvement.

NASPrecision: Neural Architecture Search-Driven Multi-Stage Learning for surface roughness prediction in ultra-precision machining

Accurate surface roughness prediction is critical for ensuring high product quality, especially in sectors such as manufacturing, aerospace, and medical devices, where the smallest imperfections can compromise performance or safety. However, this is very challenging due to complex, non-linear interactions among variables, which is further exacerbated when working with limited and imbalanced datasets. Existing methods leveraging traditional machine learning algorithms require extensive domain knowledge for feature engineering and substantial human intervention for model selection. To address these issues, we propose a Neural Architecture Search (NAS)-Driven Multi-Stage Learning Framework, named NASPrecision. This innovative approach autonomously identifies the most suitable features and models for various surface roughness prediction tasks and significantly enhances the performance by multi-stage learning. Our framework operates in three stages: (1) architecture search stage, employing NAS to automatically identify the most effective model architecture; (2) initial training stage, where we train the neural network for initial predictions; (3) refinement stage, where a subsequent model is appended to refine and capture subtle variations overlooked by the initial training stage. In light of limited and imbalanced datasets, we adopt a generative data augmentation technique to balance and generate new data by learning the underlying data distribution. We perform extensive experiments on three distinct real-world datasets, each associated with a different machining technique, and compare with various machine learning algorithms. The experimental results underscore the superiority of our framework, which achieves an average improvement of 25.7%, 22.7%, and 39.1% in terms of Mean Absolute Error (MAE), Root Mean Square Error (RMSE), and Standard Deviation of Residuals (STDR), respectively. This significant performance enhancement not only confirms the robustness of our framework but also establishes it as a generic solution for accurate surface roughness prediction. The success of this approach can lead to improved production efficiency and product quality in critical industries while also reducing the need for extensive domain knowledge and human intervention.

A review of multiscale numerical modeling of rock mechanics and rock engineering

AbstractRock is geometrically and mechanically multiscale in nature, and the traditional phenomenological laws at the macroscale cannot render a quantitative relationship between microscopic damage of rocks and overall rock structural degradation. This may lead to problems in the evaluation of rock structure stability and safe life. Multiscale numerical modeling is regarded as an effective way to gain insight into factors affecting rock properties from a cross‐scale view. This study compiles the history of theoretical developments and numerical techniques related to rock multiscale issues according to different modeling architectures, that is, the homogenization theory, the hierarchical approach, and the concurrent approach. For these approaches, their benefits, drawbacks, and application scope are underlined. Despite the considerable attempts that have been made, some key issues still result in multiple challenges. Therefore, this study points out the perspectives of rock multiscale issues so as to provide a research direction for the future. The review results show that, in addition to numerical techniques, for example, high‐performance computing, more attention should be paid to the development of an advanced constitutive model with consideration of fine geometrical descriptions of rock to facilitate solutions to multiscale problems in rock mechanics and rock engineering.

Кодовая паремиологическая когнитивная модель концепта МИЛОСЕРДИЕ в английском и русском языках

The research aims to determine the characteristics of constructing code-based paremiological cognitive models of the MERCY concept in the non-cognate language formats of English and Russian worldviews. The scientific novelty lies in identifying the architecture of code-based paremiological cognitive models representing the culturally marked spectrum of the nominative field of the MERCY concept in English and Russian, in constructing classifications of code-based cognitive models in the paremiological stocks of non-cognate languages, in determining the differences in the functioning of code-based paremiological cognitive models of the MERCY concept in the formats of English and Russian linguistic worldviews, and in enriching the scientific thesaurus of cognitive linguistics with such concepts as “a code-based paremiological cognitive model”, “the paremiological polarization of the code format of a cognitive model”. As a result, it has been proved that the functioning of code-based paremiological cognitive models in English and Russian linguistic worldviews occurs along similar and different lines. The similarity is manifested in the presence of models such as “mercy – God”, “mercy – help”, “mercy – truth”, while the difference is seen in the frequency of nominations of code models. Among the latter, the models “mercy – cruelty”, “mercy – forgiveness”, and “mercy – truth” prevail in the English paremiological stock. At the same time, the models “mercy – God”, “mercy – soul/heart”, “mercy – good” predominate in the Russian paremiological stock. The development and introduction of a new approach to interpreting the paremiological stock through the construction of code-based paremiological cognitive models enriches the theoretical and methodological basis of modern cognitive linguistics.

ARCHITECTURAL AND CIRCUIT MODEL OF NAVIGATION DEVICE FOR ULTRA-LIGHT LAUNCH VEHICLE

The paper considered the architectural and circuitry model of a navigation device for an ultra-lightweight launch vehicle designed to improve the accuracy and reliability of positioning and orientation in flight conditions. The study utilized modern approaches to the integration of global navigation satellite system (GNSS) and inertial navigation system (INS) to achieve high accuracy and stability of the system. The key components of the device including UM980 module, LSM6DSV32X inertial measurement module, STM32 F7 series microcontroller, E220-400T30S radio module, LMR50410XDBVR converter based power supply and RS422 interface were described in detail. The UM980 module, including SAW and LNA filter blocks, demonstrated the ability to efficiently process incoming RF signals. The NebulasIV main function block combined GNSS BB, GNSS RF, interfaces and power management unit (PMU) modules, supporting multiple communication interfaces such as UART, SPI and I²C, as well as RTK and PPS signals. The study analyzed the integration and accuracy aspects of the module in navigation systems. The signal processing scheme of the inertial navigation system module was also presented, including digital filters, free-fall and activity/inactivity detection functions, and interfaces for data communication. User bias functions and filter modes are considered to ensure the accuracy and stability of the navigation device. The work lays the groundwork for further improvements in navigation technology, especially in the context of increasing availability and accuracy for small scale space launches.

MATHEMATICAL MODEL AND STRUCTURE OF A NEURAL NETWORK FOR DETECTION OF CYBER ATTACKS ON INFORMATION AND COMMUNICATION SYSTEMS

The paper discusses the principles of creating a mathematical model and system architecture by applying the method of artificial intelligence to detect cyberattacks on information and communication systems, where a neural network capable of learning and detecting cyberattacks is used. The proposed approach, based on the application of the developed mathematical model and architecture of artificial neural networks, as a detector of network attacks on information and communication systems, allows to increase the level of detection of network intrusions into computer systems, Web and Internet resources. An algorithm for processing network traffic parameters in real-time systems by structuring a neural network is proposed, which allows to optimize the redundancy of its multi-level structure at the level of inter-element connections.

Do Vision and Language Models Share Concepts? A Vector Space Alignment Study

Abstract Large-scale pretrained language models (LMs) are said to “lack the ability to connect utterances to the world” (Bender and Koller, 2020), because they do not have “mental models of the world” (Mitchell and Krakauer, 2023). If so, one would expect LM representations to be unrelated to representations induced by vision models. We present an empirical evaluation across four families of LMs (BERT, GPT-2, OPT, and LLaMA-2) and three vision model architectures (ResNet, SegFormer, and MAE). Our experiments show that LMs partially converge towards representations isomorphic to those of vision models, subject to dispersion, polysemy, and frequency. This has important implications for both multi-modal processing and the LM understanding debate (Mitchell and Krakauer, 2023).1

Collaborative Optimization of a Matrix Manufacturing System Based on Overall Equipment Effectiveness

When several traditional flow-shop lines operate in parallel, the operation mode with no communication between production lines will no longer be the optimal production paradigm. This paper describes matrix manufacturing systems (MMS) in a general manner from the perspective of related works, comparing different manufacturing organizational forms and their characteristics. Subsequently, MMS are extracted during the parallel production of multiple surface mount technology (SMT) lines. An overall equipment effectiveness (OEE) online calculation model and a collaborative optimization method are proposed based on the OEE of the MMS. The innovative idea of this study is to divide existing multiple parallel SMT lines into MMS. The efficiency of each matrix unit (MU) was calculated, and a collaborative optimization method was proposed based on an indicator (OEE). In this paper, an example of eight SMT lines is presented. The partitioning of MUs, OEE calculation of each MU, and the low OEE unit collaborative optimization method are described in detail. Through a case study, the architecture of the collaborative optimization model for the MMS was constructed and discussed. Finally, the improvement in the OEE proved the effectiveness and usability of the proposed architecture.

Evolving Adversarial Training (EAT) for AI-Powered Intrusion Detection Systems (IDS)

Intrusion Detection Systems (IDS) are crucial components of network security, yet traditional IDS models often fail to cope with rapidly evolving adversarial attacks that exploit their static nature. This study proposes a novel approach, Evolving Adversarial Training (EAT), to enhance the adaptability and robustness of AI-powered IDS against dynamic threats. The EAT framework integrates continuous model evolution with advanced adversarial training techniques, enabling the IDS to dynamically adjust to new attack patterns. Experimental results demonstrate that the EAT framework significantly enhances IDS performance, leading to increased detection accuracy and reduced false positive rates compared to conventional methods. These findings emphasize the potential of EAT in fortifying network defenses against evolving cyber threats, offering a promising avenue for future research in scalable and adaptive IDS solutions that can effectively combat the complexities of modern cyber adversaries. The research explores three key objectives: dynamic adaptation and adversarial training, continuous learning and enhanced threat detection, and robustness and generalization. By focusing on these objectives, the study aims to develop AI-powered IDS that can effectively navigate the ever-changing cyber threat landscape. The research methodology includes data collection, model architecture design, training and evaluation, continuous learning, simulation, and real-world testing, all aimed at enhancing the resilience of AI-powered IDS against adversarial attacks. By systematically following this framework, the study intends to enhance the security system of IDS through the effective implementation of EAT.

Addressing cold start problems in new store locations with transfer learning in spatial GNNs

The cold start problem poses a significant challenge for retailers opening new store locations, primarily due to the lack of historical sales data necessary for accurate demand forecasting and effective inventory management. This paper explores the application of transfer learning within spatial Graph Neural Networks (GNNs) as a solution to this issue. By leveraging existing data from established stores that share similar characteristics, our proposed methodology enhances the forecasting accuracy and helps mitigate the risks associated with new store openings. We detail the architecture of the spatial GNN model, which captures complex spatial relationships and customer interactions, providing richer insights into demand patterns. Experimental results demonstrate substantial improvements in forecasting performance compared to traditional methods, highlighting the potential of transfer learning to inform strategic decision-making in retail. This research aims to provide actionable insights for retailers seeking to optimize their operations in new markets.

Comparative Approach to De-Noising TEMPEST Video Frames

Analysis of unintended compromising emissions from Video Display Units (VDUs) is an important topic in research communities. This paper examines the feasibility of recovering the information displayed on the monitor from reconstructed video frames. The study holds particular significance for our understanding of security vulnerabilities associated with the electromagnetic radiation of digital displays. Considering the amount of noise that reconstructed TEMPEST video frames have, the work in this paper focuses on two different approaches to de-noising images for efficient optical character recognition. First, an Adaptive Wiener Filter (AWF) with adaptive window size implemented in the spatial domain was tested, and then a Convolutional Neural Network (CNN) with an encoder–decoder structure that follows both classical auto-encoder model architecture and U-Net architecture (auto-encoder with skip connections). These two techniques resulted in an improvement of more than two times on the Structural Similarity Index Metric (SSIM) for AWF and up to four times for the SSIM for the Deep Learning (DL) approach. In addition, to validate the results, the possibility of text recovery from processed noisy frames was studied using a state-of-the-art Tesseract Optical Character Recognition (OCR) engine. The present work aims to bring to attention the security importance of this topic and the non-negligible character of VDU information leakages.

Deep Learning for Pediatric Sleep Staging from Photoplethysmography: A Transfer Learning Approach from Adults to Children.

Sleep staging is critical for diagnosing sleep disorders. Traditional methods in clinical settings involve time-intensive scoring procedures. Recent advancements in data-driven algorithms using photoplethysmogram (PPG) time series have shown promise in automating sleep staging in adults. However, for children, algorithm development is hindered by the limited availability of datasets, with the Childhood Adenotonsillectomy Trial (CHAT) being the only substantial source, comprising recordings from children aged 5-10. This limitation constrains the evaluation of algorithmic generalization performance. We employed a deep learning model for sleep staging from PPG, initially trained using a large dataset of adult sleep recordings, and fine-tuned it on 80% of the CHAT dataset (CHAT-train) for the task of three-class sleep staging (wake, REM, non-REM). The resulting algorithm performance was compared to the same model architecture but trained from scratch on CHAT-train (benchmark). The algorithms are evaluated on the local test set, denoted CHAT-test, as well as on a newly introduced independent dataset. Our deep learning algorithm achieved a Cohen's Kappa of 0.88 on CHAT-test (versus 0.65), and demonstrated generalization capabilities with a Kappa of 0.72 on the external Ichilov dataset for children above 5 years old (versus 0.64) and 0.64 for those below 5 (versus 0.53). This research establishes a new state-of-the-art performance for the task of sleep staging in children using raw PPG. The findings underscore the value of transfer learning from the adults to children domain. However, the reduced performance in children under 5 suggests the need for further research and additional datasets covering a broader pediatric age range to fully address generalization limitations.

Распознавание фишинговых ссылок с использованием методов машинного обучения

In recent years, phishing has become one of the most widespread and dangerous cyber threats. These attacks aim to obtain users' confidential information, such as passwords and credit card details, through deceptive messages or websites, making the issue of protection against them more relevant than ever. Traditional methods of phishing protection, such as blacklists and heuristic analysis, can no longer keep up with the evolving pace of phishing attacks. Therefore, there is a need to develop more advanced and intelligent methods, among which machine learning (ML) techniques play a significant role. This article discusses various ML methods used for automatic detection of phishing URLs. The study presents the main approaches, model architectures, advantages and disadvantages of each method, and provides a comparative analysis of their effectiveness on real data.

Wave masked autoencoder: An electrocardiogram signal diagnosis model based on wave making strategy

ECG signal labeling is time-consuming and laborious, significantly constraining the performance of deep learning models on ECG diagnosis. Self-supervised learning methods offer a means to mitigate the reliance on annotated ECG data. Nevertheless, the direct transposition of these methods to ECG diagnostics may encounter challenges stemming from deficient domain knowledge and incongruous model architectures. Therefore, a self-supervised learning model that introduces ECG prior knowledge, called wave masked autoencoder (WMAE), is proposed. Firstly, considering that various waves that make up the ECG signal are key features for diagnosing arrhythmias, we design a wave masking strategy as an auxiliary task for self-supervised learning to introduce ECG-related knowledge. Secondly, sliding window slicing is utilized to divide each ECG signal into a series of subsequences, which are mapped into deep features with a convolutional feature mapper to increase the information density of basic semantic units and learn complete waveform representations. In this way, the proposed WMAE can capture robust and transferable ECG signal features from unlabeled data, thereby alleviating the problem of insufficient annotated ECG data and improving the performance of the model on cardiovascular disease diagnosis. Experimental results show that WMAE is significantly better than other benchmark models and has high practical value.

LungNeXt: A novel lightweight network utilizing enhanced mel-spectrogram for lung sound classification

Lung auscultation is essential for early lung condition detection. Categorizing adventitious lung sounds requires expert discrimination by medical specialists. This paper details the features of LungNeXt, a novel classification model specifically designed for lung sound analysis. Furthermore, we propose two auxiliary methods: RandClipMix (RCM) for data augmentation and Enhanced Mel-Spectrogram for Feature Extraction (EMFE). RCM addresses the issue of data imbalance by randomly mixing clips within the same category to create new adventitious lung sounds. EMFE augments specific frequency bands in spectrograms to highlight adventitious features. These contributions enable LungNeXt to achieve outstanding performance. LungNeXt optimally integrates an appropriate number of NeXtblocks, ensuring superior performance and a lightweight model architecture. The proposed RCM and EMFE methods, along with the LungNeXt classification network, have been evaluated on the SPRSound dataset. Experimental results revealed a commendable score of 0.5699 for the lung sound five-category task on SPRSound. Specifically, the LungNeXt model is characterized by its efficiency, with only 3.804M parameters and a computational complexity of 0.659G FLOPS. This lightweight and efficient model is particularly well-suited for applications in electronic stethoscope back-end processing equipment, providing efficient diagnostic advice to physicians and patients.

Exploring Deep Learning Models for Lyric Generation and Addressing Biases in Word Embeddings

The aim of this project is to explore the performance of different model architectures (such as RNN, LSTM, GRU) by generating lyrics using deep learning models, and to use the Word2Vec model for distributed semantic analysis to understand semantic phenomena and potential biases in word embedding models. The experimental results show that LSTM and GRU perform better than traditional RNN models when processing long sequence data. In addition, by analyzing word embeddings, we revealed potential gender and racial biases and proposed corresponding solutions.