Architecture Model Research Articles

Programmatic Pitch

Diffusion models have firmly established their excellence for image generation tasks, as evidenced by the success of renowned models such as DALL-E, Midjourney, and Stable Diffusion. This advanced development of diffusion models for visual content raises an interesting question: can diffusion models be adapted for audio generation tasks? In this study, we introduce a novel diffusion model architecture designed to generate mel spectrograms, visual representations of sound which can subsequently be converted into audible music. Given the exceptional capability of diffusion models to produce high- quality images, their application to mel spectrogram generation is particularly promising. Our proposed diffusion model deviates minimally from the conventional architectures employed for visual content, making this research especially useful for examining the potential for cross-domain application between image and audio generation. The proposed model has been trained on a dataset consisting of over 186 hours of Lofi audio, offering the model diverse samples for generalized learning. However, a combination of research limitations led to subpar results, paving the way for further studies to build on this one.

NASPrecision: Neural Architecture Search-Driven Multi-Stage Learning for surface roughness prediction in ultra-precision machining

Accurate surface roughness prediction is critical for ensuring high product quality, especially in sectors such as manufacturing, aerospace, and medical devices, where the smallest imperfections can compromise performance or safety. However, this is very challenging due to complex, non-linear interactions among variables, which is further exacerbated when working with limited and imbalanced datasets. Existing methods leveraging traditional machine learning algorithms require extensive domain knowledge for feature engineering and substantial human intervention for model selection. To address these issues, we propose a Neural Architecture Search (NAS)-Driven Multi-Stage Learning framework, named NASPrecision. This innovative approach autonomously identifies the most suitable features and models for various surface roughness prediction tasks and significantly enhances the performance by multi-stage learning. Our framework operates in three stages: (1) architecture search stage, employing NAS to automatically identify the most effective model architecture; (2) initial training stage, where we train the neural network for initial predictions; (3) refinement stage, where a subsequent model is appended to refine and capture subtle variations overlooked by the initial training stage. In light of limited and imbalanced datasets, we adopt a generative data augmentation technique to balance and generate new data by learning the underlying data distribution. We perform extensive experiments on three distinct real-world datasets, each associated with a different machining technique, and compare with various machine learning algorithms. The experimental results underscore the superiority of our framework, which achieves an average improvement of 25.7%, 22.7%, and 39.1% in terms of Mean Absolute Error (MAE), Root Mean Square Error (RMSE), and Standard Deviation of Residuals (STDR), respectively. This significant performance enhancement not only confirms the robustness of our framework but also establishes it as a generic solution for accurate surface roughness prediction. The success of this approach can lead to improved production efficiency and product quality in critical industries while also reducing the need for extensive domain knowledge and human intervention.

Application of neural network for automatic symbol recognition in production of electronic navigation charts from paper charts

Abstract This research boarded on a novel initiative to replace the error-prone and labour-intensive process of converting Paper Nautical Chart (PNC) symbols to Electronic Navigational Chart (ENC) symbols with a more efficient and automated manner using Artificial Intelligence (AI). The proposed method applies the Convolutional Neural Network and YOLOv5 model to recognise and convert symbols from PNC into their corresponding ENC formats. The model's competence was evaluated with performance metrics including Precision, Recall, Average Precision, and mean Average Precision. Among the different variations of the YOLOv5 models tested, the YOLOv5m version revealed the best performance achieving a mean Average Precision of 0 ⋅ 837 for all features. A confusion matrix was used to visualise the model's classification accuracy for various chart symbols, underlining strengths and identifying areas for improvements. While the model has demonstrated high ability in identifying symbols like ‘Obstruction’ and ‘Major/Minor Lights’, it exhibited lesser accuracy with ‘Visible Wreck’ and ‘Background’ categories. Further, the developed graphical user interface (GUI) allowed users to interact with the artificial neural network model without demanding detailed knowledge of the underlying programming or model architecture.

Historical Architecture and BIM Modelling: Between Representation of Reality and Conceptual Abstraction

The essay aims to explore the applications of BIM to design regarding historic buildings, which will become mandatory in Italy from 2025 for public works above €1 million. Despite the advantages in coordinated data management and information sharing among the actors involved in contracts, problems related to the geometric representation and semantic definition of the virtual entities that populate the models have been recognized by the scientific community. Critical issues that have emerged from the study of significant experiences can be traced back to the difficulty of adapting tools designed for the projects of new buildings to the characteristics of the cultural heritage. In particular, the complexity of proposing congruent categorisations and parameterisations seems related to only partial correspondence between historical building components, IFC classes and BIM software categories. These limitations often lead to the identification of contingent solutions and expedients, which solve only the problems of geometric representation. In contrast, the ability to adequately share information stored in databases, interoperable through the IFC format, is conveyed by an appropriate semantic description. Formal representation in the BIM environment effectively refers to standardized industrial production, reproducibility of building elements and, more generally, to the goals of globalisation. Thus, the proposed reflections aim to encourage a conscious use of these tools and to outline implementation perspectives useful to bring the digital environment closer to the concrete reality of historical architecture, unique and irreproducible in origin and transformation, often realized through artisanal processes and anchored to specific contexts in multiple aspects.

A Novel Approach to Predict the Asian Exchange Stock Market Index Using Artificial Intelligence

This study uses real-world illustrations to explore the application of deep learning approaches to predict economic information. In this, we investigate the effect of deep learning model architecture and time-series data properties on prediction accuracy. We aim to evaluate the predictive power of several neural network models using a financial time-series dataset. These models include Convolutional RNNs, Convolutional LSTMs, Convolutional GRUs, Convolutional Bi-directional RNNs, Convolutional Bi-directional LSTMs, and Convolutional Bi-directional GRUs. Our main objective is to utilize deep learning techniques for simultaneous predictions on multivariable time-series datasets. We utilize the daily fluctuations of six Asian stock market indices from 1 April 2020 to 31 March 2024. This study’s overarching goal is to evaluate deep learning models constructed using training data gathered during the early stages of the COVID-19 pandemic when the economy was hit hard. We find that the limitations prove that no single deep learning algorithm can reliably forecast financial data for every state. In addition, predictions obtained from solitary deep learning models are more precise when dealing with consistent time-series data. Nevertheless, the hybrid model performs better when analyzing time-series data with significant chaos.

HiCDiffusion - diffusion-enhanced, transformer-based prediction of chromatin interactions from DNA sequences

Prediction of chromatin interactions from DNA sequence has been a significant research challenge in the last couple of years. Several solutions have been proposed, most of which are based on encoder-decoder architecture, where 1D sequence is convoluted, encoded into the latent representation, and then decoded using 2D convolutions into the Hi-C pairwise chromatin spatial proximity matrix. Those methods, while obtaining high correlation scores and improved metrics, produce Hi-C matrices that are artificial - they are blurred due to the deep learning model architecture. In our study, we propose the HiCDiffusion, sequence-only model that addresses this problem. We first train the encoder-decoder neural network and then use it as a component of the diffusion model - where we guide the diffusion using a latent representation of the sequence, as well as the final output from the encoder-decoder. That way, we obtain the high-resolution Hi-C matrices that not only better resemble the experimental results - improving the Fréchet inception distance by an average of 11 times, with the highest improvement of 56 times - but also obtain similar classic metrics to current state-of-the-art encoder-decoder architectures used for the task.

Sedimentary characteristics and reservoir architecture of a lacustrine mixed carbonate/siliciclastic system: the lower member of the ShangGanchaigou Formation, Neogene, in the western Qaidam Basin, China

Lacustrine mixed carbonate/siliciclastic sediment is an important type of oil and gas reservoir with significant potential. Although previous studies have investigated the sedimentary characteristics of the mixed depositional system in numerous oil and gas-bearing basins worldwide, a detailed sedimentary architecture model is still lacking for guiding reservoir characterization at the hydrocarbon reservoir scale. In this paper, a typical lacustrine mixed carbonate/siliciclastic sedimentary system, preserved in the lower member of the ShangGanchaigou Formation, Neogene, in the western Qaidam Basin, Western China, was deeply investigated based on well logging data from 640 wells, 438 m of cores from 3 core wells, and outcrop studies. The results demonstrate that 1) seven types of architecture elements, namely, distributary channel, channel mouth bar, distal bar, sheet-like sand, shallow water mud, algal mound, and marl flat characterized by different lithofacies associations, were recognized based on core and well logging data. 2) The lacustrine mixed carbonate/siliciclastic depositional system can be divided into three facies belts. Along the lakeward direction, the proximal facies belt is dominated by delta front deposits and characterized by gradually downstream bifurcating distributary channels and associated lateral amalgamated delta lobes. The middle facies belt is characterized by isolated and small-scale delta lobes and inter-lobe deposits, including sheet-like sand, small-scale algal mounds, and marl flats, and the distal facies belt is a combination of large-scale algal mounds and marl flats. 3) Within a depression, short-term base-level cycles controlled the facies belt transition, and the proximal, middle, and distal facies belts formed under relatively low, middle, and high base-level conditions, respectively. 4) The scale and connectivity of reservoirs gradually decreased from the proximal to the distal belt.

Bolt loosening status classification via wave energy dissipation measurements with temperature-compensated deep learning by piezoelectric active sensing

Abstract In industrial applications, bolts, serving as crucial components, endure substantial loads and are susceptible to loosening problems exacerbated by intricate external environmental factors. The active sensing method based on wave energy dissipation exhibits pronounced sensitivity to axial load fluctuations in bolts and demonstrates extensive applicability, with its measurement indicators directly applicable for discerning the bolt’s status. However, environmental factors, notably temperature, can significantly influence signal energy measurements, and the oversight of temperature impact may result in erroneous state discrimination. To tackle this challenge, this paper introduces a cascaded model comprising a temperature compensation subnetwork and a bolt state discrimination subnetwork. The temperature compensation subnetwork takes temperature and signal energy as inputs and outputs the temperature-compensated signal energy, and conveys the outcomes to the bolt state discrimination subnetwork for state classification. In model design, we quantitatively analyzed the number of convolutional blocks and training epochs for the temperature compensation subnetwork with the aim of enhancing the model’s generalization ability, ultimately determining the model architecture. By comparing the experimental results between a single-task model and a model incorporating the temperature compensation subnetwork, we verified the effectiveness of the temperature compensation subnetwork. The experimental outcomes demonstrate that the proposed cascaded model achieved a state classification accuracy of 98.6% on over 1300 temperature-generalized data points. To comprehensively assess our approach, we conducted detailed comparisons with other bolt loosening monitoring methods, elucidating the effectiveness based on data trends and model design. Experimental results demonstrate that the proposed temperature compensation cascaded model accurately identifies bolt states amidst complex temperature variations.

Maximizing wind farm production through pitch control using graph neural networks and hybrid learning methods

AbstractThis article presents a novel methodology to maximize wind farm power generation by integrating graph neural networks (GNN), supervised learning, and reinforcement learning techniques. First, the article introduces a graph‐based representation of the wind farm, capturing wind turbines as vertices and the inter‐turbine wake interactions as edges. The construction of this graph representation integrates the Jensen wake model, which includes insights derived from prior knowledge of wind farm aerodynamics. Subsequently, a detailed description of the GNN model's architecture, incorporating a message passing mechanism, is outlined. This GNN model is trained initially with supervised learning using a dataset of optimal pitch angles generated from the analytical results derived from Jensen wake model. Moreover, to improve the GNN model's accuracy and adaptability, reinforcement learning techniques are employed. The GNN model interacts with a high‐fidelity wind farm simulation environment, receiving feedback in the form of rewards derived from the wind farm's actual power output. Through a policy gradient approach, the GNN parameters undergo iterative updates, enabling the model to learn and adapt to dynamic wind conditions and intricate turbine interactions. The effectiveness and advantages of the proposed methodology are demonstrated through comprehensive case studies across various wind farm layouts.

Knowledge graph-enhanced multi-agent reinforcement learning for adaptive scheduling in smart manufacturing

AbstractSelf-organizing manufacturing network has emerged as a viable solution for adaptive manufacturing control within the mass personalization paradigm. This approach involves three critical elements: system modeling and control architecture, interoperable communication, and adaptive manufacturing control. However, current research often separates interoperable communication from adaptive manufacturing control as isolated areas of study. To address this gap, this paper introduces Knowledge Graph-enhanced Multi-Agent Reinforcement Learning (MARL) method that integrates interoperable communication via Knowledge Graphs with adaptive manufacturing control through Reinforcement Learning. We hypothesize that implicit domain knowledge obtained from historical production job allocation records can guide each agent to learn more effective scheduling policies with accelerated learning rates. This is based on the premise that machine assignment preferences effectively could reduce the Reinforcement Learning search space. Specifically, we redesign machine agents with new observation, action, reward, and cooperation mechanisms considering the preference of machines, building upon our previous MARL base model. The scheduling policies are trained under extensive simulation experiments that consider manufacturing requirements. During the training process, our approach demonstrates improved training speed compared with individual Reinforcement Learning methods under the same training hyperparameters. The obtained scheduling policies generated by our Knowledge Graph-enhanced MARL also outperform both individual Reinforcement Learning methods and heuristic rules under dynamic manufacturing settings.

A Neural Architecture Search CNN for Alzheimer’s Disease Classification

The evolution of automated machine learning (AutoML) is gradually reengineering the design of deep learning architectures for various imaging tasks. AutoML effectively develops model architectures and tunes hyperparameters through neural architecture search (NAS). Deep learning model architecture design is generally considered a tedious and time-consuming task that requires mastery skills to develop robust and better-performing models for imaging tasks. Again, the model's hyperparameters must be well-tuned to ensure optimal performances, which can be tedious and time-consuming if the hyperparameters are manually selected; using existing hyperparameter optimization algorithms can be expensive regarding resources. This study addresses these challenges in developing an optimal convolutional neural network (CNN) for classifying Alzheimer's (AD). The study, therefore, adopted a NAS approach to generate a CNN model architecture using a customized search space comprising only CNN patterns implemented with a NAS framework. The search was done for ten (10) trials, yielding a CNN architecture with an accuracy of 95.85% and a loss of 0.22. Training the model with a 10-fold cross-validation approach using a 0.0009 learning rate for 150 epochs improved the model's performance. The model recorded 97.17% accuracy, 97.21% precision, 97.14% recall, and a 0.99 area under the curve (AUC) in classifying AD as one of AD, mild cognitive impairment (MCI), and normal control (NC). The model obtained 98.06%, 98.66%, and 98.62% accuracy on binary classes of AD/NC, AD/MCI, and NC/MCI, respectively. The model generally showed robustness and better performance than existing CNN architectures.

Volatility forecasting of clean energy ETF using GARCH-MIDAS with neural network model

This paper utilizes a hybrid model to analyze the impression of information from the GECON indicator on the volatility prediction of the clean energy market. The model architecture is constructed by embedding a recurrent neural network (RNN) into the GARCH-MIDAS model. The results show that RNN-GARCH-MIDAS-GECON achieves optimal ranking in volatility prediction. This work confirms the advantages of embedded hybrid integrated models in capturing nonlinear information in financial markets and achieving significant progress in volatility forecasts. Notably, this research will help to promote the construction of clean energy development and energy transition pathways.

Evolution and future directions of Artificial Intelligence Generated Content (AIGC): A comprehensive review

Artificial Intelligence Generated Content (AIGC) has rapidly evolved, revolutionizing the creation of text, images, audio, and video content. Despite these advancements, research on the development process of AIGC technology remains scarce, necessitating a systematic discussion of its current state and future directions. So this paper delves into the significant advancements and foundational technologies driving AIGC, emphasizing the contributions of state-of-the-art models such as DALL-E 3 [1] and Sora [2]. We analyze the evolution of generative models from single-modal approaches to the current multimodal generative models. The paper further explores the application prospects of AIGC across various domains such as office work, art, education, and film, while addressing the existing limitations and challenges in the field. We propose potential improvement directions, including more efficient model architectures and enhanced multimodal capabilities. Emphasis is placed on the environmental impact of AIGC technologies and the need for sustainable practices. Our comprehensive review aims to provide researchers and professionals with a deeper understanding of AIGC, inspiring further exploration and innovation in this transformative domain.

A community effort to optimize sequence-based deep learning models of gene regulation.

A systematic evaluation of how model architectures and training strategies impact genomics model performance is needed. To address this gap, we held a DREAM Challenge where competitors trained models on a dataset of millions of random promoter DNA sequences and corresponding expression levels, experimentally determined in yeast. For a robust evaluation of the models, we designed a comprehensive suite of benchmarks encompassing various sequence types. All top-performing models used neural networks but diverged in architectures and training strategies. To dissect how architectural and training choices impact performance, we developed the Prix Fixe framework to divide models into modular building blocks. We tested all possible combinations for the top three models, further improving their performance. The DREAM Challenge models not only achieved state-of-the-art results on our comprehensive yeast dataset but also consistently surpassed existing benchmarks on Drosophila and human genomic datasets, demonstrating the progress that can be driven by gold-standard genomics datasets.

State-of-the-Art Results with the Fashion-MNIST Dataset

In September 2024, the Fashion-MNIST dataset will be 7 years old. Proposed as a replacement for the well-known MNIST dataset, it continues to be used to evaluate machine learning model architectures. This paper describes new results achieved with the Fashion-MNIST dataset using classical machine learning models and a relatively simple convolutional network. We present the state-of-the-art results obtained using the CNN-3-128 convolutional network and data augmentation. The developed CNN-3-128 model containing three convolutional layers achieved an accuracy of 99.65% in the Fashion-MNIST test image set. In addition, this paper presents the results of computational experiments demonstrating the dependence between the number of adjustable parameters of the convolutional network and the maximum acceptable classification quality, which allows us to optimise the computational cost of model training.

Advances in medical image analysis: A comprehensive survey of lung infection detection

AbstractThis research investigates advanced approaches in medical image analysis, specifically focusing on segmentation and classification techniques, as well as their integration into multi‐task architectures for lung infections. This research begins by explaining key architectural models used in segmentation and classification tasks. The study extends to the enhancement of these architectures through attention modules and conditional random fields. Relevant datasets and evaluation metrics, incorporating discussions on loss functions are also reviewed. This review encompasses recent advancements in single‐task and multi‐task models, highlighting innovations in semi‐supervised, self‐supervised, few‐shot, and zero‐shot learning techniques. Empirical analysis is conducted on both single‐task and multi‐task architectures, predominantly utilizing the U‐Net framework, and is applied across multiple datasets for segmentation and classification tasks. Results demonstrate the effectiveness of these models and provide insights into the strengths and limitations of different approaches. This research contributes to improved detection and diagnosis of lung infections by offering a comprehensive overview of current methodologies and their practical applications.

Non-blocking functional dependency discovery from data streams

With the proliferation of sensors and IoT technologies, there is an increasing need to analyze information from data streams that they produce dynamically. However, the volume and velocity of this data require algorithms that mine knowledge as data are read from streams. The capability of dynamically extracting functional dependencies (fds) from data streams would not only permit to assess and improve the quality of data, but also provide knowledge on the evolution of data correlations within streams, allowing to understand the relevance that each feature has in predicting unknown features. In this paper, we propose a new discovery algorithm, namely COD3, which allows to continuous discovery fds holding on a data stream, as the data are read from it. COD3 represents the first proposal to use a non-blocking architectural model for discovering fds from data streams. Furthermore, we present novel data structures and a validation method to handle dynamic discovery and reduce data load inbound streams. Experimental evaluations demonstrate its effectiveness on both adapted real-world datasets and real data streams, such as those from air quality sensors. Moreover, by integrating COD3 with Bleach, a well-known fd-based data stream cleansing framework, we demonstrate its effectiveness in a real-world use case.

An inductive bias from quantum mechanics: learning order effects with non-commuting measurements

There are two major approaches to building good machine learning algorithms: feeding lots of data into large models or picking a model class with an “inductive bias” that suits the structure of the data. When taking the second approach as a starting point to design quantum algorithms for machine learning, it is important to understand how mathematical structures in quantum mechanics can lead to useful inductive biases in quantum models. In this work, we bring a collection of theoretical evidence from the quantum cognition literature to the field of quantum machine learning to investigate how non-commutativity of quantum observables can help to learn data with “order effects,” such as the changes in human answering patterns when swapping the order of questions in a survey. We design a multi-task learning setting in which a generative quantum model consisting of sequential learnable measurements can be adapted to a given task — or question order — by changing the order of observables, and we provide artificial datasets inspired by human psychology to carry out our investigation. Our first experimental simulations show that in some cases the quantum model learns more non-commutativity as the amount of order effect present in the data is increased and that the quantum model can learn to generate better samples for unseen question orders when trained on others — both signs that the model architecture suits the task.

A Brief Review of Lightweighting Methods for Vision Transformers (ViT)

The Vision Transformer (ViT) has emerged as a powerful model in recent years, surpassing traditional Convolutional Neural Networks (CNNs) in various benchmarks. However, its large model architecture and high parameter count present significant challenges for mobile deployment. Consequently, the exceptional performance of ViT in computer vision tasks is often overshadowed by its difficulties in being deployed on mobile devices due to its large parameter size and high computational demands. This paper provides a comprehensive review of the literature on lightweight ViT models, focusing on model optimization strategies such as post-training modifications—quantization, pruning, and knowledge distillation—as well as architectural changes including hybrid CNN-Transformer, MLP-based, and sparse models. These strategies are aimed at improving efficiency for mobile platforms. The review aims to clarify current techniques for mobile ViT, guide future research, stimulate innovation, and contribute to the development of efficient ViT models for mobile environments.

Tree-based classification model for Long-COVID infection prediction with age stratification using data from the National COVID Cohort Collaborative.

We propose and validate a domain knowledge-driven classification model for diagnosing post-acute sequelae of SARS-CoV-2 infection (PASC), also known as Long COVID, using Electronic Health Records (EHRs) data. We developed a robust model that incorporates features strongly indicative of PASC or associated with the severity of COVID-19 symptoms as identified in our literature review. The XGBoost tree-based architecture was chosen for its ability to handle class-imbalanced data and its potential for high interpretability. Using the training data provided by the Long COVID Computation Challenge (L3C), which was a sample of the National COVID Cohort Collaborative (N3C), our models were fine-tuned and calibrated to optimize Area Under the Receiver Operating characteristic curve (AUROC) and the F1 score, following best practices for the class-imbalanced N3C data. Our age-stratified classification model demonstrated strong performance with an average 5-fold cross-validated AUROC of 0.844 and F1 score of 0.539 across the young adult, mid-aged, and older-aged populations in the training data. In an independent testing dataset, which was made available after the challenge was over, we achieved an overall AUROC score of 0.814 and F1 score of 0.545. The results demonstrated the utility of knowledge-driven feature engineering in a sparse EHR data and demographic stratification in model development to diagnose a complex and heterogeneously presenting condition like PASC. The model's architecture, mirroring natural clinician decision-making processes, contributed to its robustness and interpretability, which are crucial for clinical translatability. Further, the model's generalizability was evaluated over a new cross-sectional data as provided in the later stages of the L3C challenge. The study proposed and validated the effectiveness of age-stratified, tree-based classification models to diagnose PASC. Our approach highlights the potential of machine learning in addressing the diagnostic challenges posed by the heterogeneity of Long-COVID symptoms.