General Machine Research Articles

Bridging Hydrological Ensemble Simulation and Learning Using Deep Neural Operators

AbstractEnsemble‐based simulation and learning (ESnL) has long been used in hydrology for parameter inference, but computational demands of process‐based ESnL can be quite high. To address this issue, we propose a deep neural operator learning approach. Neural operators are generic machine learning algorithms that can learn functional mappings between infinite‐dimensional spaces, providing a highly flexible tool for scientific machine learning. Our approach is built upon DeepONet, a specific deep neural operator, and is designed to address several common problems in hydrology, namely, model parameter estimation, prediction at ungaged locations, and uncertainty quantification. Here we demonstrate the effectiveness of our DeepONet‐based workflow using an existing large model ensemble created for an eastern U.S. watershed that is instrumented with 10 streamflow gages. Results suggest DeepONet achieves high efficiency in learning an ML surrogate model from the model ensemble, with the modified Kling‐Gupta Efficiency exceeding 0.9 on holdout test sets. Parameter inference, carried out using the trained DeepONet surrogate model and genetic algorithm, also yields robust results. Additionally, we formulate and train a separate DeepONet model for physics‐informed, seq‐to‐seq streamflow forecasting, which further reduces biases in the pre‐trained DeepONet surrogate model. While this study focuses primarily on a single watershed, our approach is general and may be extended to enable learning from model ensembles across multiple basins or models. Thus, this research represents a significant contribution to the application of hybrid machine learning in hydrology.

Hardware implementation of bearing fault diagnosis using empirical mode decomposition

ABSTRACT This paper presents a comprehensive methodology for developing a vibration measurement system, with a primary focus on monitoring rolling element bearings, widely recognised as pivotal components in generic machine vibration monitoring. Through the utilisation of sensors to capture vibration data signals, the system aims to meticulously analyse these signals to discern any indications of bearing health deterioration, thus facilitating the early identification of potential pre-breakdown conditions. The proposed system initiates the process by internally decomposing the three-axis vibration signal, enabling subsequent transmission of processed data to a remote location for the identification of specific faults in the roller element bearings. Installation of a three-axis accelerometer on the bearing housing, positioned radially, facilitates the capture of raw vibration data along one axial and two radial directions. Moreover, efforts to enhance computational efficiency involve cascading multiple identical modules in a serial pipeline structure, allowing for iterative loop computation of intrinsic mode functions within Empirical Mode Decomposition (EMD). Real-time computation is achieved through this serial pipeline structure, necessitating two distinct clock frequencies: one at 12.5 MHz for sampling and storing 3-axis vibration data in ping pong buffers, and another at 40 MHz for reading and processing the data signal retrieved from these buffers. Validation of the proposed method is demonstrated through simulation results obtained via synthesised Verilog code on Xilinx Artix-7 FPGA (XC7A35T-1CPG236C), affirming its efficacy in real-time applications.

Mesoscopic structure graphs for interpreting uncertainty in non-linear embeddings

Probabilistic-based non-linear dimensionality reduction (PB-NL-DR) methods, such as t-SNE and UMAP, are effective in unfolding complex high-dimensional manifolds, allowing users to explore and understand the structural patterns of data. However, due to the trade-off between global and local structure preservation and the randomness during computation, these methods may introduce false neighborhood relationships, known as distortion errors and misleading visualizations. To address this issue, we first conduct a detailed survey to illustrate the design space of prior layout enrichment visualizations for interpreting DR results, and then propose a node-link visualization technique, ManiGraph. This technique rethinks the neighborhood fidelity between the high- and low-dimensional spaces by constructing dynamic mesoscopic structure graphs and measuring region-adapted trustworthiness. ManiGraph also addresses the overplotting issue in scatterplot visualization for large-scale datasets and supports examining in unsupervised scenarios. We demonstrate the effectiveness of ManiGraph in different analytical cases, including generic machine learning using 3D toy data illustrations and fashion-MNIST, a computational biology study using a single-cell RNA sequencing dataset, and a deep learning-enabled colorectal cancer study with histopathology-MNIST.

Advancing the Prediction of MS/MS Spectra Using Machine Learning.

Tandem mass spectrometry (MS/MS) is an important tool for the identification of small molecules and metabolites where resultant spectra are most commonly identified by matching them with spectra in MS/MS reference libraries. While popular, this strategy is limited by the contents of existing reference libraries. In response to this limitation, various methods are being developed for the in silico generation of spectra to augment existing libraries. Recently, machine learning and deep learning techniques have been applied to predict spectra with greater speed and accuracy. Here, we investigate the challenges these algorithms face in achieving fast and accurate predictions on a wide range of small molecules. The challenges are often amplified by the use of generic machine learning benchmarking tactics, which lead to misleading accuracy scores. Curating data sets, only predicting spectra for sufficiently high collision energies, and working more closely with experimental mass spectrometrists are recommended strategies to improve overall prediction accuracy in this nuanced field.

Statistical Inference for Heterogeneous Treatment Effects Discovered by Generic Machine Learning in Randomized Experiments

Researchers are increasingly turning to machine learning (ML) algorithms to investigate causal heterogeneity in randomized experiments. Despite their promise, ML algorithms may fail to accurately ascertain heterogeneous treatment effects under practical settings with many covariates and small sample size. In addition, the quantification of estimation uncertainty remains a challenge. We develop a general approach to statistical inference for heterogeneous treatment effects discovered by a generic ML algorithm. We apply the Neyman’s repeated sampling framework to a common setting, in which researchers use an ML algorithm to estimate the conditional average treatment effect and then divide the sample into several groups based on the magnitude of the estimated effects. We show how to estimate the average treatment effect within each of these groups, and construct a valid confidence interval. In addition, we develop nonparametric tests of treatment effect homogeneity across groups, and rank-consistency of within-group average treatment effects. The validity of our methodology does not rely on the properties of ML algorithms because it is solely based on the randomization of treatment assignment and random sampling of units. Finally, we generalize our methodology to the cross-fitting procedure by accounting for the additional uncertainty induced by the random splitting of data.

Ensemble Conformalized Quantile Regression for Probabilistic Time Series Forecasting.

This article presents a novel probabilistic forecasting method called ensemble conformalized quantile regression (EnCQR). EnCQR constructs distribution-free and approximately marginally valid prediction intervals (PIs), which are suitable for nonstationary and heteroscedastic time series data. EnCQR can be applied on top of a generic forecasting model, including deep learning architectures. EnCQR exploits a bootstrap ensemble estimator, which enables the use of conformal predictors for time series by removing the requirement of data exchangeability. The ensemble learners are implemented as generic machine learning algorithms performing quantile regression (QR), which allow the length of the PIs to adapt to local variability in the data. In the experiments, we predict time series characterized by a different amount of heteroscedasticity. The results demonstrate that EnCQR outperforms models based only on QR or conformal prediction (CP), and it provides sharper, more informative, and valid PIs.

Monitoring human behaviour during pandemic — Attacks on healthcare personnel scenario

People exhibit all sorts of irrational behaviour during pandemics; occurrences during the Covid-19 pandemic also agree with this statement. Millions of people died during Covid-19 pandemic and people observed these deaths very closely. Therefore, it is highly probable that these deaths have a mental toll on people. One of the specific irrational incident types is - attacks on healthcare personnel (AoHP). These attacks include both verbal and physical abuse. Naturally, it is the duty of governing agencies to provide protection for these pandemic front-liners. In this work, we have explored the data related to AoHP incidents to fully understand the gravity of the situation. The current scenario is analysed through the utilization of data visualization techniques, thereby offering an insightful perspective. We have noticed that maximum cases are reported in India. In addition, civilians are mainly responsible for most occurrences. All things considered, we put forward an AoHP-type news detection mechanism, which is proposed through real experiments by applying several generic machine learning (ML) algorithms (Naive Bayes, Logistic Regression, Support Vector Machine, Extreme Gradient Boosting, Random Forest) and deep learning (DL) models (Artificial Neural Network, Long Short Term Memory, and Gated Recurrent Unit) over AoHP type news & reports for quick identification of occurrence, which would be helpful to alert the authorities. We have deployed four feature engineering techniques along with generic ML algorithms and pinpointed the most suitable one for our case. We declare Random Forest as the best fit for our purpose of classification, which achieved an F1-score of 97% in our experiments.

An attempt to augment performance of machine learning models in a pilot-scale urban wastewater treatment system

The challenge we face is striking a balance between ensuring the standards for wastewater treatment processes and minimizing energy consumption. As such, accurate predictive models become critical. With machine learning algorithms having the ability to discern relationships between inputs and outputs, broad interest has been shown in their applications in wastewater treatment. A key research focus is on effectively adjusting these generic machine learning algorithms to adapt to frequent fluctuations of water quality, inflow volume, and environmental variations. This study developed a Nonlinear Autoregressive neural network (NARX) model having an adaptive real-time updating module to predict the ammonium concentration in the treated effluent. The model has exhibited high predictive accuracy on the testing data, achieving an R2 value of 0.997. By incorporating an adaptive real-time updating module, the model's capability to predict effluent ammonium concentration on novel data sets has been significantly improved, with the R2 value for these predictions increasing from an initial 0.918 to a more refined range of 0.966 to 0.984. Our findings confirm that integrating real-time updating capabilities with machine learning algorithms substantially improves the stability and predictive performance of the models. These outcomes will prompt a broader integration of artificial intelligence in the domain of environmental engineering, which shall foster the implementation of more accurate and eco-friendly wastewater treatment strategies.

Identifying drivers of county-level industrial carbon intensity by a generic machine learning framework

Carbon intensity has been recognized as a measure of the relative change between carbon emissions and economic development in numerous policy documents at multiple levels. Counties as the basic governmental units for policy formulation and implementation remain largely unexplored in climate governance. This study makes an effort to fill in this gap by examining the dynamics and drivers of industrial carbon intensity (ICI) at the county level. Given the data limitations at both county and industrial levels, we develop a data-driven method that evaluates six commonly used machine learning algorithms to study county-level ICI accounted by statistical data. We find that energy consumption and labor scale are two consistent and significant drivers of ICI across all results, especially, energy consumption acts as the crucial driver. Accordingly, we divide the overall industrial sector into small and large sub-sectors to provide differentiated policy implications. Small sub-sectors with low energy and labor inputs prioritize production-driven ICI reduction strategies, because increasing energy consumption by introducing automated machinery leads to rapid economic growth and thereby negatively impacts ICI. For large sub-sectors with high energy and labor inputs, efficiency promotion through technological innovation is required for ICI reduction. As an example of scenario analysis, we show that an integrated intensity reduction scenario by reducing energy consumption and short-term increasing labor scale and industrial scale will make it possible for Manufacture of Furniture sub-sector in the studied county to achieve an ICI reduction of 68.7% by 2060 compared to the 2015 level. Overall, this study provides an integrated and generic framework for predicting ICI at the county level to overcome the context of data scarcity, laying a basis for making differentiated and applicable policies for ICI reduction at the sub-city level.

A Generic Machine Learning Model for Spatial Query Optimization based on Spatial Embeddings

Machine learning (ML) and deep learning (DL) techniques are increasingly applied to produce efficient query optimizers, in particular in regards to big data systems. The optimization of spatial operations is even more challenging due to the inherent complexity of such kind of operations, like spatial join or range query, and the peculiarities of spatial data. Although a few ML-based spatial query optimizers have been proposed in literature, their design limits their use, since each one is tailored for a specific collection of datasets, a specific operation, or a specific hardware setting. Changes to any of these will require building and training a completely new model which entails collecting a new very large training dataset to obtain a good model. This paper proposes a different approach which exploits the use of the novel notion of spatial embedding to overcome these limitations. In particular, a preliminary model is defined which captures the relevant features of spatial datasets, independently from the operation to be optimized and in an unsupervised manner. This model is trained with a large amount of both synthetic and real-world data, with the aim to produce meaningful spatial embeddings. The construction of an embedding model could be intended as a preliminary step for the optimization of many different spatial operations, so the cost of its building can be compensated during the subsequent construction of specific models. Indeed, for each considered spatial operation, a specific tailored model will be trained but by using spatial embeddings as input, so a very little amount of training data points is required for them. Three peculiar operations are considered as proof of concept in this paper: range query, self-join, and binary spatial join. Finally, a comparison with an alternative technique, known as transfer learning, is provided and the advantages of the proposed technique over it are highlighted.

The Value Added of Machine Learning to Causal Inference: Evidence from Revisited Studies

Abstract A new and rapidly growing econometric literature is making advances in the problem of using machine learning methods for causal inference questions. Yet, the empirical economics literature has not started to fully exploit the strengths of these modern methods. We revisit influential empirical studies with causal machine learning methods aiming to connect the econometric theory on these methods with empirical economics. We focus on the double machine learning, causal forest and generic machine learning methods, in the context of both average and heterogeneous treatment effects. We illustrate the implementation of these methods in a variety of settings and highlight the relevance and value added relative to traditional methods used in the original studies.

Generic Machine Learning Framework for Fully-Unsupervised Anomaly Detection with Contaminated Data

Anomaly detection (AD) tasks have been solved using machine learning algorithms in various domains and applications. The great majority of these algorithms use normal data to train a residual-based model, and assign anomaly scores to unseen samples based on their dissimilarity with the learned normal regime. The underlying assumption of these approaches is that anomaly-free data is available for training. This is, however, often not the case in real-world operational settings, where the training data may be contaminated with a certain fraction of abnormal samples. Training with contaminated data, in turn, inevitably leads to a deteriorated AD performance of the residual-based algorithms.
 In this paper we introduce a framework for a fully unsupervised refinement of contaminated training data for AD tasks. The framework is generic and can be applied to any residual-based machine learning model. We demonstrate the application of the framework to two public datasets of multivariate time series machine data from different application fields. We show its clear superiority over the naive approach of training with contaminated data without refinement. Moreover, we compare it to the ideal, unrealistic reference in which anomaly-free data would be available for training. Since the approach exploits information from the anomalies, and not only from the normal regime, it is comparable and often outperforms the ideal baseline as well.

Extrapolation validation (EV): a universal validation method for mitigating machine learning extrapolation risk

A generic machine learning model validation method named extrapolation validation (EV) has been proposed, which evaluates the trustworthiness of model predictions to mitigate the extrapolation risk before the model transitions to applications.

A generic machine learning model for CO2 equilibrium solubility into blended amine solutions

In this work, three machine learning methods were employed to predict carbon dioxide (CO2) equilibrium solubility in blended amine solutions consisting of ethanolamine (MEA), N,N-diethylethanolamine (DEEA) and methyldiethanolamine (MDEA). Three machine learning algorithms, Radial Basis Function Neural Network (RBFNN), Support Vector Machine Regression (SVR) and Extreme Gradient Boosting (XGBoost), were used to fit the experimental results. We found that the predicted values of the three models developed for the CO2 equilibrium solubility were in good agreement with the experimental results, and the comparison of the mean absolute percentage error (MAPE) and root mean square error (RMSE) results showed that XGBoost had the best prediction accuracy with the MAPE of less than 1%. Four different amine blends were then used to evaluate the expandability of the XGBoost model, namely MEA and 1-dimethylamino-2-propanol (1DMA2P), MDEA and piperazine (PZ), diethylenetriamine (DETA) and PZ, and MEA and triethanolamine (TEA); the MAPE between the experimental and predicted results were 0.30%, 0.91%, 0.86% and 1.63%, respectively. The results obtained showed that the XGBoost model has enormous potential for application in dealing with the CO2 equilibrium solubility in blended amine solutions.

Deep learning for automated fish grading

Fish is a staple food around the globe, and its quality is heavily dependent on freshness. The conventional method for evaluating the quality of fish is through a visual inspection of a sample. However, this approach heavily depends on human senses for precise assessment, leading to a susceptibility to variability in accuracy and efficiency. Moreover, the potential for safety to be compromised due to errors in the evaluation process makes it a less reliable method. This work presents two Neural Network (NN) architectures, FishNET-S and FishNET-T, to evaluate the quality of the Indian Sardinella and the Yellowfin Tuna, respectively, using RGB images captured from smartphone cameras. The FishNET-S is based on the VGG-16 with the introduction of a Block Attention Module (BAM) to drive the network towards learning the features related to fish quality evaluation from the eye region of the entire fish. In contrast, the FishNET-T architecture first performs a color decomposition based on hue, saturation, and intensity transformations before forwarding the hue and saturation components to the CNN in order to effectively identify grades through fish meat. Experimentally, FishNET-S has managed to obtain an accuracy of 84.1%, while FishNET-T yielded an accuracy of 68.3%. The comparison analysis carried out with the use of generic machine learning and state of the art deep learning models shows that the performance of the proposed novel architectures is dominant and unchallenged.

Context-aware Emotion Detection from Low-resource Urdu Language Using Deep Neural Network

Emotion detection (ED) plays a vital role in determining individual interest in any field. Humans use gestures, facial expressions, and voice pitch and choose words to describe their emotions. Significant work has been done to detect emotions from the textual data in English, French, Chinese, and other high-resource languages. However, emotion classification has not been well studied in low-resource languages (i.e., Urdu) due to the lack of labeled corpora. This article presents a publicly available Urdu Nastalique Emotions Dataset (UNED) of sentences and paragraphs annotated with different emotions and proposes a deep learning (DL)-based technique for classifying emotions in theUNEDcorpus. Our annotatedUNEDcorpus has six emotions for both paragraphs and sentences. We perform extensive experimentation to evaluate the quality of the corpus and further classify it using machine learning and DL approaches. Experimental results show that the developed DL-based model performs better than generic machine learning approaches with an F1 score of 85% on the UNED sentence-based corpus and 50% on the UNED paragraph-based corpus.

A case study on the evaluation of Korean-English legal translations by generic and custom neural machine translation engines

This paper addresses the quality of Korean-English legal translation outputs generated by three neural machine translation engines, namely NMT1, NMT2 and NMT3. NMT1 and NMT2 are widely available multilingual generic machine translation engines, whereas NMT3 is a custom machine translation engine for legal translation. In both automatic and human evaluations of the three engines’ English translations of Korean statutes, the custom engine outperformed the other two NMTs. BLEU and METEOR scores revealed that NMT3’s output was more similar to human reference translation. Human evaluation results confirmed NMT3’s strong performance, outperforming the generic engines in accuracy and fluency while underperforming in terminology. However, further analysis revealed that NMT3 was superior in handling terms comprising compound nouns and sentences over 25 words. Although the custom engine’s performance still requires substantial human involvement for quality translation, it showed a potential for the legal domain, demonstrating human parity in short sentences. The growing demand for speedy translation under budget constraints calls for further research on technology-based translation tools and human-computer interaction for legal translation.

Enhanced Machine Learning Sketches for Network Measurements

Network monitoring and management require accurate statistics of a variety of flow-level metrics such as flow sizes, top- <inline-formula><tex-math notation="LaTeX">$k$</tex-math></inline-formula> flows, and number of flows. Arguably, the current best technique to measure these metrics is sketches. While a significant amount of work has already been done on sketching techniques, there is still a lot of room for improvement because the accuracy of existing sketches varies with changing characteristics of network traffic. In this paper, we propose the idea of using machine learning to improve the accuracy of sketches, and propose a <i>generic machine learning framework</i> to reduce the dependence of accuracy of sketches on network traffic characteristics. We further present three case studies, where we applied our machine learning framework on sketches for measuring three flow-level network metrics, namely flow sizes, top- <inline-formula><tex-math notation="LaTeX">$k$</tex-math></inline-formula> flows, and number of flows. We implemented and extensively evaluated this framework for these three metrics using both real-world and synthetic traffic traces. To the best of our knowledge, this is the first work that uses machine learning to reduce the dependence of sketching techniques on the characteristics of network traffic. We have released all our traces and implementation codes at Github.

Physics-separating artificial neural networks for predicting sputtering and thin film deposition of AlN in Ar/N2 discharges on experimental timescales

Understanding and modeling plasma–surface interactions frame a multi-scale as well as multi-physics problem. Scale-bridging machine learning surface surrogate models have been demonstrated to perceive the fundamental atomic fidelity for the physical vapor deposition of pure metals. However, the immense computational cost of the data-generating simulations render a practical application with predictions on relevant timescales impracticable. This issue is resolved in this work for the sputter deposition of AlN in Ar/N2 discharges by developing a scheme that populates the parameter spaces effectively. Hybrid reactive molecular dynamics/time-stamped force-bias Monte Carlo simulations of randomized plasma-surface interactions/diffusion processes are used to setup a physics-separating artificial neural network. The application of this generic machine learning model to a specific experimental reference case study enables the systematic analysis of the particle flux emission as well as underlying system state (e.g. composition, density, point defect structure) evolution within process times of up to 45 min.

Objective Prediction of Next-Day's Affect Using Multimodal Physiological and Behavioral Data: Algorithm Development and Validation Study.

Affective states are important aspects of healthy functioning; as such, monitoring and understanding affect is necessary for the assessment and treatment of mood-based disorders. Recent advancements in wearable technologies have increased the use of such tools in detecting and accurately estimating mental states (eg, affect, mood, and stress), offering comprehensive and continuous monitoring of individuals over time. Previous attempts to model an individual's mental state relied on subjective measurements or the inclusion of only a few objective monitoring modalities (eg, smartphones). This study aims to investigate the capacity of monitoring affect using fully objective measurements. We conducted a comparatively long-term (12-month) study with a holistic sampling of participants' moods, including 20 affective states. Longitudinal physiological data (eg, sleep and heart rate), as well as daily assessments of affect, were collected using 3 modalities (ie, smartphone, watch, and ring) from 20 college students over a year. We examined the difference between the distributions of data collected from each modality along with the differences between their rates of missingness. Out of the 20 participants, 7 provided us with 200 or more days' worth of data, and we used this for our predictive modeling setup. Distributions of positive affect (PA) and negative affect (NA) among the 7 selected participants were observed. For predictive modeling, we assessed the performance of different machine learning models, including random forests (RFs), support vector machines (SVMs), multilayer perceptron (MLP), and K-nearest neighbor (KNN). We also investigated the capability of each modality in predicting mood and the most important features of PA and NA RF models. RF was the best-performing model in our analysis and performed mood and stress (nervousness) prediction with ~81% and ~72% accuracy, respectively. PA models resulted in better performance compared to NA. The order of the most important modalities in predicting PA and NA was the smart ring, phone, and watch, respectively. SHAP (Shapley Additive Explanations) analysis showed that sleep and activity-related features were the most impactful in predicting PA and NA. Generic machine learning-based affect prediction models, trained with population data, outperform existing methods, which use the individual's historical information. Our findings indicated that our mood prediction method outperformed the existing methods. Additionally, we found that sleep and activity level were the most important features for predicting next-day PA and NA, respectively.