Architecture Model Research Articles

Intelligent prediction of rail corrugation evolution trend based on self-attention bidirectional TCN and GRU

Analyzing the evolution trend of rail corrugation using signal processing and deep learning is critical for railway safety, as current traditional methods struggle to capture the complex evolution of corrugation. This present study addresses the challenge of accurately capturing this trend, which relies significantly on expert judgment, by proposing an intelligent prediction method based on self-attention (SA), a bidirectional temporal convolutional network (TCN), and a bidirectional gated recurrent unit (GRU). First, multidomain feature extraction and adaptive feature screening were used to obtain the optimal feature set. These features were then combined with principal component analysis (PCA) and the Mahalanobis distance (MD) method to construct a comprehensive health indicator (CHI) that reflects the evolution of rail corrugation. A bidirectional fusion model architecture was employed to capture the temporal correlations between forward and backward information during corrugation evolution, with SA embedded in the model to enhance the focus on key information. The outcome was a rail corrugation trend prediction network that combined a bidirectional TCN, bidirectional GRU, and SA. Subsequently, a multi-strategy improved crested porcupine optimizer (CPO) algorithm was constructed to automatically obtain the optimal network hyperparameters. The proposed method was validated with on-site rail corrugation data, demonstrating superior predictive performance compared to other advanced methods. In summary, the proposed method can accurately predict the evolution trend of rail corrugation, offering a valuable tool for on-site railway maintenance.

Data‐efficient graph learning: Problems, progress, and prospects

AbstractGraph‐structured data, ranging from social networks to financial transaction networks, from citation networks to gene regulatory networks, have been widely used for modeling a myriad of real‐world systems. As a prevailing model architecture to model graph‐structured data, graph neural networks (GNNs) have drawn much attention in both academic and industrial communities in the past decades. Despite their success in different graph learning tasks, existing methods usually rely on learning from “big” data, requiring a large amount of labeled data for model training. However, it is common that real‐world graphs are associated with “small” labeled data as data annotation and labeling on graphs is always time and resource‐consuming. Therefore, it is imperative to investigate graph machine learning (graph ML) with low‐cost human supervision for low‐resource settings where limited or even no labeled data is available. This paper investigates a new research field—data‐efficient graph learning, which aims to push forward the performance boundary of graph ML models with different kinds of low‐cost supervision signals. Specifically, we outline the fundamental research problems, review the current progress, and discuss the future prospects of data‐efficient graph learning, aiming to illuminate the path for subsequent research in this field.

Generalizable and automated classification of TNM stage from pathology reports with external validation

Cancer staging is an essential clinical attribute informing patient prognosis and clinical trial eligibility. However, it is not routinely recorded in structured electronic health records. Here, we present BB-TEN: Big Bird – TNM staging Extracted from Notes, a generalizable method for the automated classification of TNM stage directly from pathology report text. We train a BERT-based model using publicly available pathology reports across approximately 7000 patients and 23 cancer types. We explore the use of different model types, with differing input sizes, parameters, and model architectures. Our final model goes beyond term-extraction, inferring TNM stage from context when it is not included in the report text explicitly. As external validation, we test our model on almost 8000 pathology reports from Columbia University Medical Center, finding that our trained model achieved an AU-ROC of 0.815–0.942. This suggests that our model can be applied broadly to other institutions without additional institution-specific fine-tuning.

Many-body localized hidden generative models

Quantum generative models hold the promise of accelerating or improving machine learning tasks by leveraging the probabilistic nature of quantum states, but the successful optimization of these models remains a difficult challenge. To tackle this challenge, we present a new architecture for quantum generative modeling that combines insights from classical machine learning and quantum phases of matter. In particular, our model utilizes both many-body localized (MBL) dynamics and hidden units to improve the optimization of the model. We demonstrate the applicability of our model on a diverse set of classical and quantum tasks, including a toy version of MNIST handwritten digits, quantum data obtained from quantum many-body states, and nonlocal parity data. Our architecture and algorithm provide novel strategies of utilizing quantum many-body systems as learning resources and reveal a powerful connection between disorder, interaction, and learning in quantum many-body systems. Published by the American Physical Society 2024

N exus: a framework for controlled simulations of idealized galaxies

ABSTRACT Motivated by the need for realistic, dynamically self-consistent, evolving galaxy models that avoid the complexity of full, and zoom-in, cosmological simulations, we have developed Nexus, an integral framework to create and evolve synthetic galaxies made of collisionless and gaseous components. Nexus leverages the power of publicly available, tried-and-tested packages: the stellar-dynamics, action-based library Action-based Galaxy Modelling Architecture (AGAMA); and the adaptive mesh refinement, N-body/hydrodynamical code Ramses, modified to meet our needs. In addition, we make use of a proprietary module to account for galaxy formation physics, including gas cooling and heating, star formation, stellar feedback, and chemical enrichment. Nexus’ basic functionality consists in the generation of bespoke initial conditions (ICs) for a diversity of galaxy models, which are advanced in time to simulate the galaxy’s evolution. The fully self-consistent ICs are generated with a distribution-function-based approach, as implemented in the galaxy modelling module of AGAMA – up to now restricted to collisionless components, extended in this work to treat two types of gaseous configurations: hot haloes and gas discs. Nexus allows constructing equilibrium models with disc gas fractions $0~\le ~f_{\rm {\rm gas}}~\le ~1$, appropriate to model both low- and high-redshift galaxies. Similarly, the framework is ideally suited to the study of galactic ecology, i.e. the dynamical interplay between stars and gas over billions of years. As a validation and illustration of our framework, we reproduce several isolated galaxy model setups reported in earlier studies, and present a new, ‘nested bar’ galaxy simulation. Future upgrades of Nexus will include magnetohydrodynamics and highly energetic particle (‘cosmic ray’) heating.

Programmatic Pitch

Diffusion models have firmly established their excellence for image generation tasks, as evidenced by the success of renowned models such as DALL-E, Midjourney, and Stable Diffusion. This advanced development of diffusion models for visual content raises an interesting question: can diffusion models be adapted for audio generation tasks? In this study, we introduce a novel diffusion model architecture designed to generate mel spectrograms, visual representations of sound which can subsequently be converted into audible music. Given the exceptional capability of diffusion models to produce high- quality images, their application to mel spectrogram generation is particularly promising. Our proposed diffusion model deviates minimally from the conventional architectures employed for visual content, making this research especially useful for examining the potential for cross-domain application between image and audio generation. The proposed model has been trained on a dataset consisting of over 186 hours of Lofi audio, offering the model diverse samples for generalized learning. However, a combination of research limitations led to subpar results, paving the way for further studies to build on this one.

Application of neural network for automatic symbol recognition in production of electronic navigation charts from paper charts

Abstract This research boarded on a novel initiative to replace the error-prone and labour-intensive process of converting Paper Nautical Chart (PNC) symbols to Electronic Navigational Chart (ENC) symbols with a more efficient and automated manner using Artificial Intelligence (AI). The proposed method applies the Convolutional Neural Network and YOLOv5 model to recognise and convert symbols from PNC into their corresponding ENC formats. The model's competence was evaluated with performance metrics including Precision, Recall, Average Precision, and mean Average Precision. Among the different variations of the YOLOv5 models tested, the YOLOv5m version revealed the best performance achieving a mean Average Precision of 0 ⋅ 837 for all features. A confusion matrix was used to visualise the model's classification accuracy for various chart symbols, underlining strengths and identifying areas for improvements. While the model has demonstrated high ability in identifying symbols like ‘Obstruction’ and ‘Major/Minor Lights’, it exhibited lesser accuracy with ‘Visible Wreck’ and ‘Background’ categories. Further, the developed graphical user interface (GUI) allowed users to interact with the artificial neural network model without demanding detailed knowledge of the underlying programming or model architecture.

Historical Architecture and BIM Modelling: Between Representation of Reality and Conceptual Abstraction

The essay aims to explore the applications of BIM to design regarding historic buildings, which will become mandatory in Italy from 2025 for public works above €1 million. Despite the advantages in coordinated data management and information sharing among the actors involved in contracts, problems related to the geometric representation and semantic definition of the virtual entities that populate the models have been recognized by the scientific community. Critical issues that have emerged from the study of significant experiences can be traced back to the difficulty of adapting tools designed for the projects of new buildings to the characteristics of the cultural heritage. In particular, the complexity of proposing congruent categorisations and parameterisations seems related to only partial correspondence between historical building components, IFC classes and BIM software categories. These limitations often lead to the identification of contingent solutions and expedients, which solve only the problems of geometric representation. In contrast, the ability to adequately share information stored in databases, interoperable through the IFC format, is conveyed by an appropriate semantic description. Formal representation in the BIM environment effectively refers to standardized industrial production, reproducibility of building elements and, more generally, to the goals of globalisation. Thus, the proposed reflections aim to encourage a conscious use of these tools and to outline implementation perspectives useful to bring the digital environment closer to the concrete reality of historical architecture, unique and irreproducible in origin and transformation, often realized through artisanal processes and anchored to specific contexts in multiple aspects.

A Novel Approach to Predict the Asian Exchange Stock Market Index Using Artificial Intelligence

This study uses real-world illustrations to explore the application of deep learning approaches to predict economic information. In this, we investigate the effect of deep learning model architecture and time-series data properties on prediction accuracy. We aim to evaluate the predictive power of several neural network models using a financial time-series dataset. These models include Convolutional RNNs, Convolutional LSTMs, Convolutional GRUs, Convolutional Bi-directional RNNs, Convolutional Bi-directional LSTMs, and Convolutional Bi-directional GRUs. Our main objective is to utilize deep learning techniques for simultaneous predictions on multivariable time-series datasets. We utilize the daily fluctuations of six Asian stock market indices from 1 April 2020 to 31 March 2024. This study’s overarching goal is to evaluate deep learning models constructed using training data gathered during the early stages of the COVID-19 pandemic when the economy was hit hard. We find that the limitations prove that no single deep learning algorithm can reliably forecast financial data for every state. In addition, predictions obtained from solitary deep learning models are more precise when dealing with consistent time-series data. Nevertheless, the hybrid model performs better when analyzing time-series data with significant chaos.

HiCDiffusion - diffusion-enhanced, transformer-based prediction of chromatin interactions from DNA sequences

Prediction of chromatin interactions from DNA sequence has been a significant research challenge in the last couple of years. Several solutions have been proposed, most of which are based on encoder-decoder architecture, where 1D sequence is convoluted, encoded into the latent representation, and then decoded using 2D convolutions into the Hi-C pairwise chromatin spatial proximity matrix. Those methods, while obtaining high correlation scores and improved metrics, produce Hi-C matrices that are artificial - they are blurred due to the deep learning model architecture. In our study, we propose the HiCDiffusion, sequence-only model that addresses this problem. We first train the encoder-decoder neural network and then use it as a component of the diffusion model - where we guide the diffusion using a latent representation of the sequence, as well as the final output from the encoder-decoder. That way, we obtain the high-resolution Hi-C matrices that not only better resemble the experimental results - improving the Fréchet inception distance by an average of 11 times, with the highest improvement of 56 times - but also obtain similar classic metrics to current state-of-the-art encoder-decoder architectures used for the task.

Sedimentary characteristics and reservoir architecture of a lacustrine mixed carbonate/siliciclastic system: the lower member of the ShangGanchaigou Formation, Neogene, in the western Qaidam Basin, China

Lacustrine mixed carbonate/siliciclastic sediment is an important type of oil and gas reservoir with significant potential. Although previous studies have investigated the sedimentary characteristics of the mixed depositional system in numerous oil and gas-bearing basins worldwide, a detailed sedimentary architecture model is still lacking for guiding reservoir characterization at the hydrocarbon reservoir scale. In this paper, a typical lacustrine mixed carbonate/siliciclastic sedimentary system, preserved in the lower member of the ShangGanchaigou Formation, Neogene, in the western Qaidam Basin, Western China, was deeply investigated based on well logging data from 640 wells, 438 m of cores from 3 core wells, and outcrop studies. The results demonstrate that 1) seven types of architecture elements, namely, distributary channel, channel mouth bar, distal bar, sheet-like sand, shallow water mud, algal mound, and marl flat characterized by different lithofacies associations, were recognized based on core and well logging data. 2) The lacustrine mixed carbonate/siliciclastic depositional system can be divided into three facies belts. Along the lakeward direction, the proximal facies belt is dominated by delta front deposits and characterized by gradually downstream bifurcating distributary channels and associated lateral amalgamated delta lobes. The middle facies belt is characterized by isolated and small-scale delta lobes and inter-lobe deposits, including sheet-like sand, small-scale algal mounds, and marl flats, and the distal facies belt is a combination of large-scale algal mounds and marl flats. 3) Within a depression, short-term base-level cycles controlled the facies belt transition, and the proximal, middle, and distal facies belts formed under relatively low, middle, and high base-level conditions, respectively. 4) The scale and connectivity of reservoirs gradually decreased from the proximal to the distal belt.

Bolt loosening status classification via wave energy dissipation measurements with temperature-compensated deep learning by piezoelectric active sensing

Abstract In industrial applications, bolts, serving as crucial components, endure substantial loads and are susceptible to loosening problems exacerbated by intricate external environmental factors. The active sensing method based on wave energy dissipation exhibits pronounced sensitivity to axial load fluctuations in bolts and demonstrates extensive applicability, with its measurement indicators directly applicable for discerning the bolt’s status. However, environmental factors, notably temperature, can significantly influence signal energy measurements, and the oversight of temperature impact may result in erroneous state discrimination. To tackle this challenge, this paper introduces a cascaded model comprising a temperature compensation subnetwork and a bolt state discrimination subnetwork. The temperature compensation subnetwork takes temperature and signal energy as inputs and outputs the temperature-compensated signal energy, and conveys the outcomes to the bolt state discrimination subnetwork for state classification. In model design, we quantitatively analyzed the number of convolutional blocks and training epochs for the temperature compensation subnetwork with the aim of enhancing the model’s generalization ability, ultimately determining the model architecture. By comparing the experimental results between a single-task model and a model incorporating the temperature compensation subnetwork, we verified the effectiveness of the temperature compensation subnetwork. The experimental outcomes demonstrate that the proposed cascaded model achieved a state classification accuracy of 98.6% on over 1300 temperature-generalized data points. To comprehensively assess our approach, we conducted detailed comparisons with other bolt loosening monitoring methods, elucidating the effectiveness based on data trends and model design. Experimental results demonstrate that the proposed temperature compensation cascaded model accurately identifies bolt states amidst complex temperature variations.

Maximizing wind farm production through pitch control using graph neural networks and hybrid learning methods

AbstractThis article presents a novel methodology to maximize wind farm power generation by integrating graph neural networks (GNN), supervised learning, and reinforcement learning techniques. First, the article introduces a graph‐based representation of the wind farm, capturing wind turbines as vertices and the inter‐turbine wake interactions as edges. The construction of this graph representation integrates the Jensen wake model, which includes insights derived from prior knowledge of wind farm aerodynamics. Subsequently, a detailed description of the GNN model's architecture, incorporating a message passing mechanism, is outlined. This GNN model is trained initially with supervised learning using a dataset of optimal pitch angles generated from the analytical results derived from Jensen wake model. Moreover, to improve the GNN model's accuracy and adaptability, reinforcement learning techniques are employed. The GNN model interacts with a high‐fidelity wind farm simulation environment, receiving feedback in the form of rewards derived from the wind farm's actual power output. Through a policy gradient approach, the GNN parameters undergo iterative updates, enabling the model to learn and adapt to dynamic wind conditions and intricate turbine interactions. The effectiveness and advantages of the proposed methodology are demonstrated through comprehensive case studies across various wind farm layouts.

Knowledge graph-enhanced multi-agent reinforcement learning for adaptive scheduling in smart manufacturing

AbstractSelf-organizing manufacturing network has emerged as a viable solution for adaptive manufacturing control within the mass personalization paradigm. This approach involves three critical elements: system modeling and control architecture, interoperable communication, and adaptive manufacturing control. However, current research often separates interoperable communication from adaptive manufacturing control as isolated areas of study. To address this gap, this paper introduces Knowledge Graph-enhanced Multi-Agent Reinforcement Learning (MARL) method that integrates interoperable communication via Knowledge Graphs with adaptive manufacturing control through Reinforcement Learning. We hypothesize that implicit domain knowledge obtained from historical production job allocation records can guide each agent to learn more effective scheduling policies with accelerated learning rates. This is based on the premise that machine assignment preferences effectively could reduce the Reinforcement Learning search space. Specifically, we redesign machine agents with new observation, action, reward, and cooperation mechanisms considering the preference of machines, building upon our previous MARL base model. The scheduling policies are trained under extensive simulation experiments that consider manufacturing requirements. During the training process, our approach demonstrates improved training speed compared with individual Reinforcement Learning methods under the same training hyperparameters. The obtained scheduling policies generated by our Knowledge Graph-enhanced MARL also outperform both individual Reinforcement Learning methods and heuristic rules under dynamic manufacturing settings.

A Neural Architecture Search CNN for Alzheimer’s Disease Classification

The evolution of automated machine learning (AutoML) is gradually reengineering the design of deep learning architectures for various imaging tasks. AutoML effectively develops model architectures and tunes hyperparameters through neural architecture search (NAS). Deep learning model architecture design is generally considered a tedious and time-consuming task that requires mastery skills to develop robust and better-performing models for imaging tasks. Again, the model's hyperparameters must be well-tuned to ensure optimal performances, which can be tedious and time-consuming if the hyperparameters are manually selected; using existing hyperparameter optimization algorithms can be expensive regarding resources. This study addresses these challenges in developing an optimal convolutional neural network (CNN) for classifying Alzheimer's (AD). The study, therefore, adopted a NAS approach to generate a CNN model architecture using a customized search space comprising only CNN patterns implemented with a NAS framework. The search was done for ten (10) trials, yielding a CNN architecture with an accuracy of 95.85% and a loss of 0.22. Training the model with a 10-fold cross-validation approach using a 0.0009 learning rate for 150 epochs improved the model's performance. The model recorded 97.17% accuracy, 97.21% precision, 97.14% recall, and a 0.99 area under the curve (AUC) in classifying AD as one of AD, mild cognitive impairment (MCI), and normal control (NC). The model obtained 98.06%, 98.66%, and 98.62% accuracy on binary classes of AD/NC, AD/MCI, and NC/MCI, respectively. The model generally showed robustness and better performance than existing CNN architectures.

Evolution and future directions of Artificial Intelligence Generated Content (AIGC): A comprehensive review

Artificial Intelligence Generated Content (AIGC) has rapidly evolved, revolutionizing the creation of text, images, audio, and video content. Despite these advancements, research on the development process of AIGC technology remains scarce, necessitating a systematic discussion of its current state and future directions. So this paper delves into the significant advancements and foundational technologies driving AIGC, emphasizing the contributions of state-of-the-art models such as DALL-E 3 [1] and Sora [2]. We analyze the evolution of generative models from single-modal approaches to the current multimodal generative models. The paper further explores the application prospects of AIGC across various domains such as office work, art, education, and film, while addressing the existing limitations and challenges in the field. We propose potential improvement directions, including more efficient model architectures and enhanced multimodal capabilities. Emphasis is placed on the environmental impact of AIGC technologies and the need for sustainable practices. Our comprehensive review aims to provide researchers and professionals with a deeper understanding of AIGC, inspiring further exploration and innovation in this transformative domain.

A community effort to optimize sequence-based deep learning models of gene regulation.

A systematic evaluation of how model architectures and training strategies impact genomics model performance is needed. To address this gap, we held a DREAM Challenge where competitors trained models on a dataset of millions of random promoter DNA sequences and corresponding expression levels, experimentally determined in yeast. For a robust evaluation of the models, we designed a comprehensive suite of benchmarks encompassing various sequence types. All top-performing models used neural networks but diverged in architectures and training strategies. To dissect how architectural and training choices impact performance, we developed the Prix Fixe framework to divide models into modular building blocks. We tested all possible combinations for the top three models, further improving their performance. The DREAM Challenge models not only achieved state-of-the-art results on our comprehensive yeast dataset but also consistently surpassed existing benchmarks on Drosophila and human genomic datasets, demonstrating the progress that can be driven by gold-standard genomics datasets.

State-of-the-Art Results with the Fashion-MNIST Dataset

In September 2024, the Fashion-MNIST dataset will be 7 years old. Proposed as a replacement for the well-known MNIST dataset, it continues to be used to evaluate machine learning model architectures. This paper describes new results achieved with the Fashion-MNIST dataset using classical machine learning models and a relatively simple convolutional network. We present the state-of-the-art results obtained using the CNN-3-128 convolutional network and data augmentation. The developed CNN-3-128 model containing three convolutional layers achieved an accuracy of 99.65% in the Fashion-MNIST test image set. In addition, this paper presents the results of computational experiments demonstrating the dependence between the number of adjustable parameters of the convolutional network and the maximum acceptable classification quality, which allows us to optimise the computational cost of model training.

Advances in medical image analysis: A comprehensive survey of lung infection detection

AbstractThis research investigates advanced approaches in medical image analysis, specifically focusing on segmentation and classification techniques, as well as their integration into multi‐task architectures for lung infections. This research begins by explaining key architectural models used in segmentation and classification tasks. The study extends to the enhancement of these architectures through attention modules and conditional random fields. Relevant datasets and evaluation metrics, incorporating discussions on loss functions are also reviewed. This review encompasses recent advancements in single‐task and multi‐task models, highlighting innovations in semi‐supervised, self‐supervised, few‐shot, and zero‐shot learning techniques. Empirical analysis is conducted on both single‐task and multi‐task architectures, predominantly utilizing the U‐Net framework, and is applied across multiple datasets for segmentation and classification tasks. Results demonstrate the effectiveness of these models and provide insights into the strengths and limitations of different approaches. This research contributes to improved detection and diagnosis of lung infections by offering a comprehensive overview of current methodologies and their practical applications.

Non-blocking functional dependency discovery from data streams

With the proliferation of sensors and IoT technologies, there is an increasing need to analyze information from data streams that they produce dynamically. However, the volume and velocity of this data require algorithms that mine knowledge as data are read from streams. The capability of dynamically extracting functional dependencies (fds) from data streams would not only permit to assess and improve the quality of data, but also provide knowledge on the evolution of data correlations within streams, allowing to understand the relevance that each feature has in predicting unknown features. In this paper, we propose a new discovery algorithm, namely COD3, which allows to continuous discovery fds holding on a data stream, as the data are read from it. COD3 represents the first proposal to use a non-blocking architectural model for discovering fds from data streams. Furthermore, we present novel data structures and a validation method to handle dynamic discovery and reduce data load inbound streams. Experimental evaluations demonstrate its effectiveness on both adapted real-world datasets and real data streams, such as those from air quality sensors. Moreover, by integrating COD3 with Bleach, a well-known fd-based data stream cleansing framework, we demonstrate its effectiveness in a real-world use case.