Learning Bias Research Articles

Bias in Machine Learning: A Literature Review

Bias could be defined as the tendency to be in favor or against a person or a group, thus promoting unfairness. In computer science, bias is called algorithmic or artificial intelligence (i.e., AI) and can be described as the tendency to showcase recurrent errors in a computer system, which result in “unfair” outcomes. Bias in the “outside world” and algorithmic bias are interconnected since many types of algorithmic bias originate from external factors. The enormous variety of different types of AI biases that have been identified in diverse domains highlights the need for classifying the said types of AI bias and providing a detailed overview of ways to identify and mitigate them. The different types of algorithmic bias that exist could be divided into categories based on the origin of the bias, since bias can occur during the different stages of the Machine Learning (i.e., ML) lifecycle. This manuscript is a literature study that provides a detailed survey regarding the different categories of bias and the corresponding approaches that have been proposed to identify and mitigate them. This study not only provides ready-to-use algorithms for identifying and mitigating bias, but also enhances the empirical knowledge of ML engineers to identify bias based on the similarity that their use cases have to other approaches that are presented in this manuscript. Based on the findings of this study, it is observed that some types of AI bias are better covered in the literature, both in terms of identification and mitigation, whilst others need to be studied more. The overall contribution of this research work is to provide a useful guideline for the identification and mitigation of bias that can be utilized by ML engineers and everyone who is interested in developing, evaluating and/or utilizing ML models.

Toward Explainable Affective Computing: A Review.

Affective computing has an unprecedented potential to change the way humans interact with technology. While the last decades have witnessed vast progress in the field, multimodal affective computing systems are generally black box by design. As affective systems start to be deployed in real-world scenarios, such as education or healthcare, a shift of focus toward improved transparency and interpretability is needed. In this context, how do we explain the output of affective computing models? and how to do so without limiting predictive performance? In this article, we review affective computing work from an explainable AI (XAI) perspective, collecting and synthesizing relevant papers into three major XAI approaches: premodel (applied before training), in-model (applied during training), and postmodel (applied after training). We present and discuss the most fundamental challenges in the field, namely, how to relate explanations back to multimodal and time-dependent data, how to integrate context and inductive biases into explanations using mechanisms such as attention, generative modeling, or graph-based methods, and how to capture intramodal and cross-modal interactions in post hoc explanations. While explainable affective computing is still nascent, existing methods are promising, contributing not only toward improved transparency but, in many cases, surpassing state-of-the-art results. Based on these findings, we explore directions for future research and discuss the importance of data-driven XAI and explanation goals, and explainee needs definition, as well as causability or the extent to which a given method leads to human understanding.

Physics-Informed Deep-Learning Models Improve Forecast Scalability, Reliability

_ This article, written by JPT Technology Editor Chris Carpenter, contains highlights of paper URTeC 4042557, “Physics-Informed Deep-Learning Models for Improving Shale and Tight Forecast Scalability and Reliability,” by Kainan Wang, SPE, Lichi Deng, and Yuzhe Cai, SPE, Chevron, et al. The paper has not been peer reviewed. _ In the complete paper, the authors present a workflow that combines probabilistic modeling and deep-learning models trained on an ensemble of physics models to improve scalability and reliability for shale and tight reservoir forecasting. Their approach is applied to synthetic cases and many wells in the Permian Basin. Through hindsight studies, these models have been demonstrated to generate realistic and diverse production curves, capture the physics of unconventional flow, quantify well-production-outlook uncertainty, and help interpretation of subsurface uncertainty. Introduction In the realm of unconventional asset development, scalable forecasting is a key component in forecast reliability. In recent years, data-driven machine-learning models and workflows have emerged as potent tools for predicting well performance, particularly in scenarios where wells share similar reservoir properties, completion designs, and operational conditions. A novel approach known as physics-informed machine learning (PIML) has gained prominence. These models leverage the strengths of machine learning while honoring the constraints imposed by physical laws. By incorporating observational, inductive, or learning bias, they bridge the gap between empirical data and fundamental physics. In this paper, the authors showcase the application of a PIML system tailored specifically for unconventional reservoirs. Their methodology involves the selection of an appropriate physics modeling framework, coupled with the development of multiple machine-learning models that augment the limited field data set. Data Acquisition and Generation Bottomhole pressure (BHP) data are key to reliable forecasting because they reflect constraints to well production. Downhole pressure gauges can provide measurements of BHP accurately, but these measurements are not always available. Numerical models can be used to correlate surface measurements to BHP with some accuracy, taking into consideration the well configuration and fluid properties, yet it is too time-consuming to develop such models for many wells in relation to the speed needed for unconventional reservoir development. An in-house machine-learning model was developed for BHP calculation to further augment the data set of gauge measurements and numerical modeling. Through a workflow that chooses the wells with higher-fidelity data as a training set, and careful calibration and iterations between incrementally adding wells of high-fidelity data to reduce uncertainty, the trained machine- learning model is able to predict BHP accurately for many more wells. Another type of engineering data important for reliable well-performance analysis and forecasting is reservoir-fluid description. In this study, the authors leveraged an internally developed pressure/volume/temperature (PVT) machine-learning model trained on proprietary PVT data, which was also shown to provide reliable predictions to key PVT properties such as saturation pressure, formation volume factor, and viscosity from limited inputs at the reservoir and separator level. These PVT models not only enhance scalability of any downstream forecasting models but also play a crucial role in physics-informed uncertainty analysis for these models.

Modeling the Time Evolution of Compact Binary Systems with Machine Learning

This work introduces advanced computational techniques for modeling the time evolution of compact binary systems using machine learning. The dynamics of compact binary systems, such as black holes and neutron stars, present significant nonlinear challenges due to the strong gravitational interactions and the requirement for precise numerical simulations. Traditional methods, like the post-Newtonian approximation, often require significant computational resources and face challenges in accuracy and efficiency. Here, we employed machine learning algorithms, including deep learning models like long short-term memory (LSTM) and temporal convolutional network (TCN), to predict the future evolution of these systems based on extensive simulation data. Our results demonstrate that employing both LSTM and TCN even as black-box predictors for sequence prediction can also significantly improve the prediction accuracy without physics-informed neural networks (as partial differential equation solvers with prior knowledge or inductive bias. By employing LSTM and TCN, we obtained R 2 values of 99.74% and 99.19% for the evolutionary orbits of the compact binaries data set, respectively. Our models demonstrate the ability to effectively capture the dynamics of the binaries, achieving high prediction performance with significantly reduced computational overhead by a factor of 40, compared to conventional numerical methods. This study paves the way for more effective and computationally scalable approaches to the understanding of gravitational phenomena and predictive modeling in gravitational-wave astronomy.

Utilizing Inherent Bias for Memory Efficient Continual Learning: A Simple and Robust Baseline

Learning from continuously evolving data is critical in real-world applications. This type of learning, known as Continual Learning (CL), aims to assimilate new information without compromising performance on prior knowledge. However, learning new information leads to a bias in the network towards recent observations, resulting in a phenomenon known as catastrophic forgetting. The complexity increases in Online Continual Learning (OCL) scenarios where models are allowed only a single pass over data. Existing OCL approaches that rely on replaying exemplar sets are not only memory-intensive when it comes to large-scale datasets but also raise security concerns. While recent dynamic network models address memory concerns, they often present computationally demanding, over-parameterized solutions with limited generalizability. To address this longstanding problem, we propose a novel OCL approach termed “Bias Robust online Continual Learning (BRCL).” BRCL retains all intermediate models generated. These models inherently exhibit a preference for recently learned classes. To leverage this property for enhanced performance, we devise a strategy we describe as ‘utilizing bias to counteract bias.’ This method involves the development of an Inference function that capitalizes on the inherent biases of each model towards the recent tasks. Furthermore, we integrate a model consolidation technique that aligns the first layers of these models, particularly focusing on similar feature representations. This process effectively reduces the memory requirement, ensuring a low memory footprint. Despite the simplicity of the methodology to guarantee expandability to various frameworks, extensive experiments reveal a notable performance edge over leading methods on key benchmarks, getting continual learning closer to matching offline training. (Source code will be made publicly available upon the publication of this paper.)

ASAP: Automatic Synthesis of Attack Prototypes, an online-learning, end-to-end approach

Zero-day attack detection and categorization is an open-research field where four main context factors need to be taken into account: novel or zero-day attacks (i) are unlabeled by definition, (ii) may correspond to out-of-distribution data, (iii) can arise concurrently, and (iv) distribution shifts in the feature space need online-learning. Given such constraints, the online detection and categorization of new cyber threats can be modeled as a heterogeneous collective anomaly detection problem, for which no online-learning solutions exist purely based on back-propagation. To this respect, this paper presents an online-learning, end-to-end back-propagation strategy for Automatically Synthesizing the potential signatures or Attack Prototypes of novel cyber threats (asap). The presented framework incorporates automatic feature engineering, operating over raw data from the OpenFlow monitoring API and raw bytes of traffic captures. In asap, specialized inductive biases enhance the training data efficiency and accommodate the inference machinery to resource-constrained scenarios such as the Internet of Things. Finally, the validity of this framework is demonstrated in a live training experiment comprising IoT traffic emulation 33To foster research in the field, the source-code of asap alongside instructions to reproduce the experiments are made publicly available among the set of SmartVille NID models in [1]..

A Tutorial on the Use of Physics-Informed Neural Networks to Compute the Spectrum of Quantum Systems

Quantum many-body systems are of great interest for many research areas, including physics, biology, and chemistry. However, their simulation is extremely challenging, due to the exponential growth of the Hilbert space with system size, making it exceedingly difficult to parameterize the wave functions of large systems by using exact methods. Neural networks and machine learning, in general, are a way to face this challenge. For instance, methods like tensor networks and neural quantum states are being investigated as promising tools to obtain the wave function of a quantum mechanical system. In this tutorial, we focus on a particularly promising class of deep learning algorithms. We explain how to construct a Physics-Informed Neural Network (PINN) able to solve the Schrödinger equation for a given potential, by finding its eigenvalues and eigenfunctions. This technique is unsupervised, and utilizes a novel computational method in a manner that is barely explored. PINNs are a deep learning method that exploit automatic differentiation to solve integro-differential equations in a mesh-free way. We show how to find both the ground and the excited states. The method discovers the states progressively by starting from the ground state. We explain how to introduce inductive biases in the loss to exploit further knowledge of the physical system. Such additional constraints allow for a faster and more accurate convergence. This technique can then be enhanced by a smart choice of collocation points in order to take advantage of the mesh-free nature of the PINN. The methods are made explicit by applying them to the infinite potential well and the particle in a ring, a challenging problem to be learned by an artificial intelligence agent due to the presence of complex-valued eigenfunctions and degenerate states

Monitoring surgical nociception using multisensor physiological models

Monitoring nociception, the flow of information associated with harmful stimuli through the nervous system even during unconsciousness, is critical for proper anesthesia care during surgery. Currently, this is done by tracking heart rate and blood pressure by eye. Monitoring objectively a patient's nociceptive state remains a challenge, causing drugs to often be over- or underdosed intraoperatively. Inefficient management of surgical nociception may lead to more complex postoperative pain management and side effects such as postoperative cognitive dysfunction, particularly in elderly patients. We collected a comprehensive and multisensor prospective observational dataset focused on surgical nociception (101 surgeries, 18,582 min, and 49,878 nociceptive stimuli), including annotations of all nociceptive stimuli occurring during surgery and medications administered. Using this dataset, we developed indices of autonomic nervous system activity based on physiologically and statistically rigorous point process representations of cardiac action potentials and sweat gland activity. Next, we constructed highly interpretable supervised and unsupervised models with appropriate inductive biases that quantify surgical nociception throughout surgery. Our models track nociceptive stimuli more accurately than existing nociception monitors. We also demonstrate that the characterizing signature of nociception learned by our models resembles the known physiology of the response to pain. Our work represents an important step toward objective multisensor physiology-based markers of surgical nociception. These markers are derived from an in-depth characterization of nociception as measured during surgery itself rather than using other experimental models as surrogates for surgical nociception.

Decode, Move and Speak! Self-supervised Learning of Speech Units, Gestures, and Sound Relationships Using Vocal Imitation

Abstract Speech learning encompasses mastering a complex motor system to produce speech sounds from articulatory gestures while simultaneously uncovering discrete units that provide entry to the linguistic system. Remarkably, children acquire these associations between speech sounds, articulatory gestures, and linguistic units in a weakly supervised manner, without the need for explicit labeling of auditory inputs or access to target articulatory gestures. This study uses self-supervised deep learning to investigate the respective roles of sounds, gestures, and linguistic units in speech acquisition and control. In a first experiment, we analyzed the quantized representations learned by vector-quantized variational autoencoders (VQ-VAE) from ground truth acoustic and articulatory data using ABX tests. We show an interesting complementarity between acoustic and articulatory modalities that may help in the discovery of phonemes. In a second experiment, we introduce a computational agent that repeats auditory speech inputs by controlling a virtual vocal apparatus. This agent integrates an articulatory synthesizer capable of reproducing diverse speech stimuli from interpretable parameters, along with two internal models implementing the articulatory-to-acoustic (forward) and acoustic-to-articulatory (inverse) mapping, respectively. Additionally, two inductive biases are used to regularize the ill-posed acoustic-to-articulatory inverse mapping. In line with the first experiment, we explore the complementarity between the auditory input and the articulatory parameters inferred by the agent. We also evaluate the impact of discretizing auditory inputs using VQ-VAE. While the majority of the agent’s productions are intelligible (according to perceptual evaluations), our analysis highlights inconsistencies in the underlying articulatory trajectories. In particular, we show that the agent’s productions only partially reproduce the complementarity between the auditory and articulatory modalities observed in humans.

A hierarchical reinforcement learning model explains individual differences in attentional set shifting

Attentional set shifting refers to the ease with which the focus of attention is directed and switched. Cognitive tasks, such as the widely used CANTAB IED, reveal great variation in set shifting ability in the general population, with notable impairments in those with psychiatric diagnoses. The attentional and learning processes underlying this cognitive ability and how they lead to the observed variation remain unknown. To directly test this, we used a modelling approach on two independent large-scale online general-population samples performing CANTAB IED, with one including additional psychiatric symptom assessment. We found a hierarchical model that learnt both feature values and dimension attention best explained the data and that compulsive symptoms were associated with slower learning and higher attentional bias to the first relevant stimulus dimension. These data showcase a new methodology to analyse data from the CANTAB IED task, as well as suggest a possible mechanistic explanation for the variation in set shifting performance, and its relationship to compulsive symptoms.

An illumination-guided dual attention vision transformer for low-light image enhancement

Existing Retinex-based low-light image enhancement methods often overlook corruptions hidden in darkness or pattern collapse caused by the lit-up process. Recent deep learning approaches suggest the use of U-shaped networks with Vision in Transformer (VIT) to address these issues. However, most VIT-based methods focus on channel modeling to reduce expensive computational costs, but in which the restored images suffer from spatial illumination inconsistencies, artifacts, and blurriness. To end for this, we propose a novel one-stage Retinex-based Illumination-Guided Dual transformer model (IGDFormer) to lit up low-light images. The model consists of an estimator and a restorer. The estimator generates a light-up feature map and a lit-up image through pure Convolutional Neural Networks (CNNs). The restorer denoises the lit-up image with a U-shaped network equipped with an Illumination-Guided Dual Attention Block (IGDAB). Specifically, IGDAB consists of cascaded channel attention and window attention that achieves cross-channel/spatial modeling. Channel attention alleviates inductive bias through the CNN-Transformer collaborative layer, and window attention introduces spatial domains knowledge by partitioning and shifting. In addition, the light-up features act as values guide the interaction modeling of non-local illumination intensities in both the channel and spatial domains. Extensive experiments were conducted on 5 low-light image enhancement benchmarks and 1 dark object detection benchmark, which demonstrate that the efficacy of our IGDFormer and its superiority in restoring spatial details compared to other state-of-the-art methods. The code is available at https://github.com/YanJieWen/IGDFormer-light-up-dark.

DSC-Net: Enhancing Blind Road Semantic Segmentation with Visual Sensor Using a Dual-Branch Swin-CNN Architecture.

In modern urban environments, visual sensors are crucial for enhancing the functionality of navigation systems, particularly for devices designed for visually impaired individuals. The high-resolution images captured by these sensors form the basis for understanding the surrounding environment and identifying key landmarks. However, the core challenge in the semantic segmentation of blind roads lies in the effective extraction of global context and edge features. Most existing methods rely on Convolutional Neural Networks (CNNs), whose inherent inductive biases limit their ability to capture global context and accurately detect discontinuous features such as gaps and obstructions in blind roads. To overcome these limitations, we introduce Dual-Branch Swin-CNN Net(DSC-Net), a new method that integrates the global modeling capabilities of the Swin-Transformer with the CNN-based U-Net architecture. This combination allows for the hierarchical extraction of both fine and coarse features. First, the Spatial Blending Module (SBM) mitigates blurring of target information caused by object occlusion to enhance accuracy. The hybrid attention module (HAM), embedded within the Inverted Residual Module (IRM), sharpens the detection of blind road boundaries, while the IRM improves the speed of network processing. In tests on a specialized dataset designed for blind road semantic segmentation in real-world scenarios, our method achieved an impressive mIoU of 97.72%. Additionally, it demonstrated exceptional performance on other public datasets.

POLYCORE: Polygon-based contour refinement for improved Intravascular Ultrasound Segmentation

Segmentation of the coronary vessel wall in intravascular ultrasound is a fundamental step in guiding coronary intervention. However, it is an challenging task, even for highly skilled cardiologists, due to image artefacts and shadowed regions caused by calcified plaque, guide wires and vessel side branches. Recently, dense-based neural networks have been applied to this task, however, they often fail to predict anatomically plausible contours in these low-signal areas. We propose a novel methodology called Polygon-based Contour Refiner (POLYCORE) that addresses topological error in dense-based segmentation networks using a relational inductive bias through higher-order connections between vertices to learn anatomically rational contours. Our approach remedies the over-smoothing phenomena common in polygon networks by introducing a new vector field refinement module which enables pixel-level detail to be added in an iterative process. POLYCORE is enhanced with augmented polygon aggregation which we show is more effective than typical dense-based test-time augmentation strategies. We achieve state-of-the-art results on two diverse datasets, observing particular improvements when segmenting the lumen structure and in topologically-challenging regions containing shadow artefacts. Our source code is available here: https://github.com/kitbransby/POLYCORE.

A novel dual-granularity lightweight transformer for vision tasks

Transformer-based networks have revolutionized visual tasks with their continuous innovation, leading to significant progress. However, the widespread adoption of Vision Transformers (ViT) is limited due to their high computational and parameter requirements, making them less feasible for resource-constrained mobile and edge computing devices. Moreover, existing lightweight ViTs exhibit limitations in capturing different granular features, extracting local features efficiently, and incorporating the inductive bias inherent in convolutional neural networks. These limitations somewhat impact the overall performance. To address these limitations, we propose an efficient ViT called Dual-Granularity Former (DGFormer). DGFormer mitigates these limitations by introducing two innovative modules: Dual-Granularity Attention (DG Attention) and Efficient Feed-Forward Network (Efficient FFN). In our experiments, on the image recognition task of ImageNet, DGFormer surpasses lightweight models such as PVTv2-B0 and Swin Transformer by 2.3% in terms of Top1 accuracy. On the object detection task of COCO, under RetinaNet detection framework, DGFormer outperforms PVTv2-B0 and Swin Transformer with increase of 0.5% and 2.4% in average precision (AP), respectively. Similarly, under Mask R-CNN detection framework, DGFormer exhibits improvement of 0.4% and 1.8% in AP compared to PVTv2-B0 and Swin Transformer, respectively. On the semantic segmentation task on the ADE20K, DGFormer achieves a substantial improvement of 2.0% and 2.5% in mean Intersection over Union (mIoU) over PVTv2-B0 and Swin Transformer, respectively. The code is open-source and available at: https://github.com/ISCLab-Bistu/DGFormer.git.

Overcoming learning bias via Prototypical Feature Compensation for source-free domain adaptation

The focus of Source-free Unsupervised Domain Adaptation (SFUDA) is to effectively transfer a well-trained model from the source domain to an unlabelled target domain. During the target domain adaptation, the source domain data is no longer accessible. Prevalent methodologies attempt to synchronize the data distributions between the source and target domains, utilizing pseudo-labels to impart categorical information, which has made some progress in improving the model’s performance. However, performance impairments persist due to the introduction of learning bias from the source model and the impact of noisy pseudo-labels generated for the target domain. In this research, we reveal that the central cause for feature misalignment during domain transition is the learning bias, which is generated by the discrepancy of information between source and target domain data. The source domain data may contain distinguishable features that do not appear on the target domain, which causes the pre-trained source model to fail to work during domain adaptation. To overcome the information discrepancy, we propose a Prototypical Feature Compensation (PFC) Network. The network extracts representative feature maps of the source domain. Then use them to minimize the discrepancy information in the target domain feature maps. This mechanism facilitates feature alignment across different domains, allowing the model to generate more accurate categorical data through pseudo-labelling. The experimental results and ablation studies demonstrate exceptional performance on three SFUDA datasets and provide evidence of the proposed PFC method’s ability to adjust the feature distribution of both source and target domain data, ensuring their overlap in the latent space.

Uncovering labeler bias in machine learning annotation tasks

AbstractAs artificial intelligence becomes increasingly pervasive, it is essential that we understand the implications of bias in machine learning. Many developers rely on crowd workers to generate and annotate datasets for machine learning applications. However, this step risks embedding training data with labeler bias, leading to biased decision-making in systems trained on these datasets. To characterize labeler bias, we created a face dataset and conducted two studies where labelers of different ethnicity and sex completed annotation tasks. In the first study, labelers annotated subjective characteristics of faces. In the second, they annotated images using bounding boxes. Our results demonstrate that labeler demographics significantly impact both subjective and accuracy-based annotations, indicating that collecting a diverse set of labelers may not be enough to solve the problem. We discuss the consequences of these findings for current machine learning practices to create fair and unbiased systems.

Semi-Supervised Remote Sensing Building Change Detection with Joint Perturbation and Feature Complementation

The timely updating of the spatial distribution of buildings is essential to understanding a city’s development. Deep learning methods have remarkable benefits in quickly and accurately recognizing these changes. Current semi-supervised change detection (SSCD) methods have effectively reduced the reliance on labeled data. However, these methods primarily focus on utilizing unlabeled data through various training strategies, neglecting the impact of pseudo-changes and learning bias in models. When dealing with limited labeled data, abundant low-quality pseudo-labels generated by poorly performing models can hinder effective performance improvement, leading to the incomplete recognition results of changes to buildings. To address this issue, we propose a feature multi-scale information interaction and complementation semi-supervised method based on consistency regularization (MSFG-SemiCD), which includes a multi-scale feature fusion-guided change detection network (MSFGNet) and a semi-supervised update method. Among them, the network facilitates the generation of multi-scale change features, integrates features, and captures multi-scale change targets through the temporal difference guidance module, the full-scale feature fusion module, and the depth feature guidance fusion module. Moreover, this enables the fusion and complementation of information between features, resulting in more complete change features. The semi-supervised update method employs a weak-to-strong consistency framework to achieve model parameter updates while maintaining perturbation invariance of unlabeled data at both input and encoder output features. Experimental results on the WHU-CD and LEVIR-CD datasets confirm the efficacy of the proposed method. There is a notable improvement in performance at both the 1% and 5% levels. The IOU in the WHU-CD dataset increased by 5.72% and 6.84%, respectively, while in the LEVIR-CD dataset, it improved by 18.44% and 5.52%, respectively.

Language structure reflects biases in pattern learning across domains and modalities.

A central goal for psychological science is the explanation of variation in human behaviour. In the domain of language, patterns of cross-linguistic variation have been extensively documented, but there has been vigorous debate over how to explain them. A particularly contentious question is whether constraints on linguistic variation are driven by properties of the human mind that are specific to language or domain-general. In this paper, we present four pattern-learning experiments (N = 306 English- and Italian-speaking adults) across domains (linguistic and nonlinguistic) and modalities (visual, auditory, and tactile) to show that the patterns that are more easily learned are precisely the ones that are found most frequently across languages. This supports a domain-general, cognitive explanation for cross-linguistic variation. However, we suggest that the general/specific dichotomy is ultimately misleading because language structure arises when domain- and modality-general biases meet domain-specific representations.

Transferable and data efficient metamodeling of storm water system nodal depths using auto-regressive graph neural networks

Storm water systems (SWSs) are essential infrastructure providing multiple services including environmental protection and flood prevention. Typically, utility companies rely on computer simulators to properly design, operate, and manage SWSs. However, multiple applications in SWSs are highly time-consuming. Researchers have resorted to cheaper-to-run models, i.e. metamodels, as alternatives of computationally expensive models. With the recent surge in artificial intelligence applications, machine learning has become a key approach for metamodelling urban water networks. Specifically, deep learning methods, such as feed-forward neural networks, have gained importance in this context. However, these methods require generating a sufficiently large database of examples and training their internal parameters. Both processes defeat the purpose of using a metamodel, i.e., saving time. To overcome this issue, this research focuses on the application of inductive biases and transfer learning for creating SWS metamodels which require less data and retain high performance when used elsewhere. In particular, this study proposes an auto-regressive graph neural network metamodel of the Storm Water Management Model (SWMM) from the Environmental Protection Agency (EPA) for estimating hydraulic heads. The results indicate that the proposed metamodel requires a smaller number of examples to reach high accuracy and speed-up, in comparison to fully connected neural networks. Furthermore, the metamodel shows transferability as it can be used to predict hydraulic heads with high accuracy on unseen parts of the network. This work presents a novel approach that benefits both urban drainage practitioners and water network modeling researchers. The proposed metamodel can help practitioners on the planning, operation, and maintenance of their systems by offering an efficient metamodel of SWMM for computationally intensive tasks like optimization and Monte Carlo analyses. Researchers can leverage the current metamodel’s structure for developing new surrogate model architectures tailored to their specific needs or start paving the way for more general foundation metamodels of urban drainage systems.

Physically-guided temporal diffusion transformer for long-term time series forecasting

Transformer has shown excellent performance in long-term time series forecasting because of its capability to capture long-term dependencies. However, existing Transformer-based approaches often overlook the unique characteristics inherent to time series, particularly multi-scale periodicity, which leads to a gap in inductive biases. To address this oversight, the temporal diffusion Transformer (TDT) was developed in this study to reveal the intrinsic evolution processes of time series. First, to uncover the connections among the periods of multi-periodic time series, the series are transformed into various types of patches using a multi-scale Patch method. Inspired by the principles of heat conduction, TDT conceptualizes the evolution of a time series as a diffusion process. TDT aims to achieve global consistency by minimizing energy constraints, which is accomplished through the iterative updating of patches. Finally, the results of these iterations across multiple periods are aggregated to form the TDT output. Compared to previous advanced models, TDT achieved state-of-the-art predictive performance in our experiments. In most datasets, TDT outperformed the baseline model by approximately 2% in terms of mean square error (MSE) and mean absolute error (MAE). Its effectiveness was further validated through ablation, efficiency, and hyperparameter analyses. TDT offers intuitive explanations by elucidating the diffusion process of time series patches throughout the iterative procedure.