Machine Learning Tasks Research Articles

Deep-ELA: Deep Exploratory Landscape Analysis with Self-Supervised Pretrained Transformers for Single- and Multi-Objective Continuous Optimization Problems.

In many recent works,the potential of Exploratory Landscape Analysis (ELA) features to numerically characterize single-objective continuous optimization problems has been demonstrated. These numerical features provide the input for all kinds of machine learning tasks in the domain of continuous optimization problems, ranging, i.a., from High-level Property Prediction to Automated Algorithm Selection and Automated Algorithm Configuration. Without ELA features, analyzing and understanding the characteristics of single-objective continuous optimization problems is - to the best of our knowledge - very limited. Yet, despite their usefulness, as demonstrated in several past works, ELA features suffer from several drawbacks. These include, in particular, (1.) a strong correlation between multiple features, as well as (2.) its very limited applicability to multiobjective continuous optimization problems. As a remedy, recent works proposed deep learning-based approaches as alternatives to ELA. In these works, among others point-cloud transformers were used to characterize an optimization problem's fitness landscape. However, these approaches require a large amount of labeled training data. Within this work, we propose a hybrid approach, Deep-ELA, which combines (the benefits of) deep learning and ELA features. We pre-trained four transformers on millions of randomly generated optimization problems to learn deep representations of the landscapes of continuous single- and multi-objective optimization problems. Our proposed framework can either be used out-of-the-box for analyzing single- and multiobjective continuous optimization problems, or subsequently fine-tuned to various tasks focusing on algorithm behavior and problem understanding.

A feature selection and scoring scheme for dimensionality reduction in a machine learning task

A feature selection and scoring scheme for dimensionality reduction in a machine learning task

KubePipe: a container-based high-level parallelization tool for scalable machine learning pipelines

As the complexity and scale of machine learning applications continue to grow, the need for efficient training methodologies becomes increasingly critical. Traditional training processes can be time-intensive, often limiting rapid development and deployment. In response to this challenge, we present KubePipe, a high-level tool that abstracts parallelism and containerization from the user, allowing non-expert users to leverage advanced parallel architectures without requiring deep knowledge of parallel computing or container orchestration. KubePipe enables the concurrent execution of multiple machine learning workflows within a Kubernetes cluster, optimizing computational resources and significantly reducing training times. By leveraging containerized environments, KubePipe ensures a high degree of modularity, scalability, and portability, making it adaptable to various machine learning frameworks and tasks. Our experimental results demonstrate substantial performance improvements when using KubePipe compared to conventional pipeline implementations. This paper explores the architecture and functionality of KubePipe, providing insights into its integration with existing machine learning systems and highlighting its potential to streamline the training process in high-performance computing environments.

CODE-ACCORD: A Corpus of building regulatory data for rule generation towards automatic compliance checking

Automatic Compliance Checking (ACC) within the Architecture, Engineering, and Construction (AEC) sector necessitates automating the interpretation of building regulations to achieve its full potential. Converting textual rules into machine-readable formats is challenging due to the complexities of natural language and the scarcity of resources for advanced Machine Learning (ML). Addressing these challenges, we introduce CODE-ACCORD, a dataset of 862 sentences from the building regulations of England and Finland. Only the self-contained sentences, which express complete rules without needing additional context, were considered as they are essential for ACC. Each sentence was manually annotated with entities and relations by a team of 12 annotators to facilitate machine-readable rule generation, followed by careful curation to ensure accuracy. The final dataset comprises 4,297 entities and 4,329 relations across various categories, serving as a robust ground truth. CODE-ACCORD supports a range of ML and Natural Language Processing (NLP) tasks, including text classification, entity recognition, and relation extraction. It enables applying recent trends, such as deep neural networks and large language models, to ACC.

Sustainability, Accuracy, Fairness, and Explainability (SAFE) Machine Learning in Quantitative Trading

The paper investigates the application of advanced machine learning (ML) methodologies, with a particular emphasis on state-of-the-art deep learning models, to predict financial market dynamics and maximize profitability through algorithmic trading strategies. The study compares the predictive capabilities and behavioral characteristics of traditional machine learning approaches, such as logistic regression and support vector machines, with those of highly sophisticated deep learning architectures, including Recurrent Neural Networks (RNNs), Long Short-Term Memory (LSTM) networks, and Gated Recurrent Units (GRUs). The findings underscore the fundamental distinctions between these methodologies, with deeply trained models exhibiting markedly different predictive behaviors and performance, particularly in capturing complex temporal patterns within financial data. A cornerstone of the paper is the introduction and rigorous analysis of a framework to evaluate models, by means of the SAFE framework (Sustainability, Accuracy, Fairness, and Explainability). The framework is designed to address the opacity of black-box ML models by systematically evaluating their behavior across a set of critical dimensions. It also demonstrates how models’ predictive outputs align with the observed data, thereby reinforcing their reliability and robustness. The paper leverages historical stock price data from International Business Machines Corporation (IBM). The dataset is partitioned into a training phase during which the models are calibrated, and a validation phase, used to evaluate the predictive performance of the generated trading signals. The study addresses two primary machine learning tasks: regression and classification. Classical models are utilized for classification tasks, with their outputs directly interpreted as trading signals, while advanced deep learning models are employed for regression, with predictions of future stock prices further processed into actionable trading strategies. To evaluate the effectiveness of each strategy, rigorous backtesting is conducted, incorporating visual representations such as equity curves to assess profitability and key risk metrics like maximum drawdown for risk management. Supplementary performance indicators, including hit rates and the incidence of false positions, are analyzed alongside the equity curves to provide a holistic assessment of each model’s performance. This comprehensive evaluation not only highlights the superiority of cutting-edge deep learning models in predicting financial market trends but also demonstrates the pivotal role of the SAFE framework in ensuring that machine learning models remain trustworthy, interpretable, and aligned with ethical considerations.

Self-Adaptive Quantum Kernel Principal Component Analysis for Compact Readout of Chemiresistive Sensor Arrays.

The rapid growth of Internet of Things (IoT) devices necessitates efficient data compression techniques to manage the vast amounts of data they generate. Chemiresistive sensor arrays (CSAs), a simple yet essential component in IoT systems, produce large datasets due to their simultaneous multi-sensor operations. Classical principal component analysis (cPCA), a widely used solution for dimensionality reduction, often struggles to preserve critical information in complex datasets. In this study, the self-adaptive quantum kernel (SAQK) PCA is introduced as a complementary approach to enhance information retention. The results show that SAQK PCA outperforms cPCA in various back end machine-learning tasks, particularly in low-dimensional scenarios where quantum bit resources are constrained. Although the overall improvement is modest in some cases, SAQK PCA proves especially effective in preserving group structures within low-dimensional data. These findings underscore the potential of noisy intermediate-scale quantum (NISQ) computers to transform data processing in real-world IoT applications by improving the efficiency and reliability of CSA data compression and readout, despite current qubit limitations.

Time Series Forecasting with LLMs: Understanding and Enhancing Model Capabilities

Large language models (LLMs) have been applied in many fields and have developed rapidly in recent years. As a classic machine learning task, time series forecasting has recently been boosted by LLMs. Recent works treat large language models as zero-shot time series reasoners without further fine-tuning, which achieves remarkable performance. However, some unexplored research problems exist when applying LLMs for time series forecasting under the zero-shot setting. For instance, the LLMs' preferences for the input time series are less understood. In this paper, by comparing LLMs with traditional time series forecasting models, we observe many interesting properties of LLMs in the context of time series forecasting. First, our study shows that LLMs perform well in predicting time series with clear patterns and trends but face challenges with datasets lacking periodicity. This observation can be explained by the ability of LLMs to recognize the underlying period within datasets, which is supported by our experiments. In addition, the input strategy is investigated, and it is found that incorporating external knowledge and adopting natural language paraphrases substantially improves the predictive performance of LLMs for time series. Our study contributes insight into LLMs' advantages and limitations in time series forecasting under different conditions.

Dissipation Alters Modes of Information Encoding in Small Quantum Reservoirs near Criticality.

Quantum reservoir computing (QRC) has emerged as a promising paradigm for harnessing near-term quantum devices to tackle temporal machine learning tasks. Yet, identifying the mechanisms that underlie enhanced performance remains challenging, particularly in many-body open systems where nonlinear interactions and dissipation intertwine in complex ways. Here, we investigate a minimal model of a driven-dissipative quantum reservoir described by two coupled Kerr-nonlinear oscillators, an experimentally realizable platform that features controllable coupling, intrinsic nonlinearity, and tunable photon loss. Using Partial Information Decomposition (PID), we examine how different dynamical regimes encode input drive signals in terms of redundancy (information shared by each oscillator) and synergy (information accessible only through their joint observation). Our key results show that, near a critical point marking a dynamical bifurcation, the system transitions from predominantly redundant to synergistic encoding. We further demonstrate that synergy amplifies short-term responsiveness, thereby enhancing immediate memory retention, whereas strong dissipation leads to more redundant encoding that supports long-term memory retention. These findings elucidate how the interplay of instability and dissipation shapes information processing in small quantum systems, providing a fine-grained, information-theoretic perspective for analyzing and designing QRC platforms.

Research Advanced in Sarcastic Detection based on Deep Learning

Sarcasm detection is an important task in natural language processing (NLP), which aims to identify ironic intentions in text or discourse. Unlike conventional narratives, satire often uses descriptions that are opposite to the literal meaning to convey humor or criticism, and it is widely used in various scenarios such as social media, comment sections, and forums. Accurately detecting satire is crucial for understanding users' true intentions, while remains an open issue due to the freedom of language expression. Compared to face-to-face communication with the extra tone, facial expressions, and body movements, or an article with background and contextual information, a single satirical comment can hardly provide information for the machine to make a correct judgment, which may mislead the model to recognize the emotions contained in the comment. This article aims to introduce the role of different features in machine learning and deep learning satire detection tasks based on current research in this field. In addition, this article discusses the problems of satire detection and looks forward to its future development direction

Adapting to the stream: an instance-attention GNN method for irregular multivariate time series data

Multivariate time series (MTS) data are vital for various applications, particularly in machine learning tasks. However, challenges such as sensor failures can result in irregular and misaligned data with missing values, thereby complicating their analysis. While recent advancements use graph neural networks (GNNs) to manage these Irregular Multivariate Time Series (IMTS) data, they generally require a reliable graph structure, either pre-existing or inferred from adequate data to properly capture node correlations. This poses a challenge in applications where IMTS data are often streamed and waiting for future data to estimate a suitable graph structure becomes impractical. To overcome this, we introduce a dynamic GNN model suited for streaming characteristics of IMTS data, incorporating an instance-attention mechanism that dynamically learns and updates graph edge weights for realtime analysis. We also tailor strategies for high-frequency and low-frequency data to enhance prediction accuracy. Empirical results on real-world datasets demonstrate the superiority of our proposed model in both classification and imputation tasks.

Synthetic satellite telemetry data for machine learning

AbstractFor many machine learning tasks, labeled data are crucial. Even though there are methods that can be trained with data with only few labels, most of the tasks require many labels. In satellite operations, a huge amount of data are generated by the telemetry parameters of a satellite that keep track of its status. Modern satellites collect telemetry data of thousands of parameters. For example, the GRACE Follow-On satellites, operated by the German Space Operations Center (GSOC) at the German Aerospace Center (DLR), define about 80,000 unique housekeeping parameters each. However, all these telemetry data lack a complete/holistic set of labels. These data are usually unpredictable, hard to reproduce, and very diverse. As a consequence, expert knowledge is necessary to label these data, e.g., with anomalies. Moreover, labeling data by hand can be very time-consuming and, therefore, expensive. To overcome these obstacles, we implemented a synthetic satellite telemetry data library that is able to (a) generate a large variety of telemetry-like data, (b) add a plethora of well-defined anomalies to these data, and (c) deliver the labels for these injected anomalies. With these data, we are now able to train, validate, and test our machine learning models. Furthermore, we can compare different models with reproducible data. Since satellite telemetry data are often strictly confidential, we can share these synthetic data easily with our research partners.

Boosting any learning algorithm with Statistically Enhanced Learning

Feature engineering is of critical importance in the field of Data Science. While any data scientist knows the importance of rigorously preparing data to obtain good performing models, only scarce literature formalizes its benefits. In this work, we present the method of Statistically Enhanced Learning (SEL), a formalization framework of existing feature engineering and extraction tasks in Machine Learning (ML). Contrary to existing approaches, predictors are not directly observed but obtained as statistical estimators. Our goal is to study SEL, aiming to establish a formalized framework and illustrate its improved performance by means of simulations as well as applications on practical use cases.

Towards data-driven electricity management: multi-region uniform data and knowledge graph

Due to growing population and technological advances, global electricity consumption is increasing. Although CO2 emissions are projected to plateau or slightly decrease by 2025 due to the adoption of clean energy sources, they are still not decreasing enough to mitigate climate change. The residential sector makes up 25% of global electricity consumption and has potential to improve efficiency and reduce CO2 footprint without sacrificing comfort. However, a lack of uniform consumption data at the household level spanning multiple regions hinders large-scale studies and robust multi-region model development. This paper introduces a multi-region dataset compiled from publicly available sources and presented in a uniform format. This data enables machine learning tasks such as disaggregation, demand forecasting, appliance ON/OFF classification, etc. Furthermore, we develop an RDF knowledge graph that characterizes the electricity consumption of the households and contextualizes it with household-related properties enabling semantic queries and interoperability with other open knowledge bases like Wikidata and DBpedia. This structured data can be utilized to inform various stakeholders towards data-driven policy and business development.

The Threat of Adversarial Attacks against Machine Learning in Network Security: A Survey

Machine learning models have made many decision support systems to be faster, more accurate and more efficient. However, applications of machine learning in network security face more disproportionate threat of active adversarial attacks compared to other domains. This is because machine learning applications in network security such as malware detection, intrusion detection, and spam filtering are by themselves adversarial in nature. In what could be considered an arm's race between attackers and defenders, adversaries constantly probe machine learning systems with inputs which are explicitly designed to bypass the system and induce a wrong prediction. In this survey, we first provide a taxonomy of machine learning techniques, tasks, and depth. We then introduce a classification of machine learning in network security applications. Next, we examine various adversarial attacks against machine learning in network security and introduce two classification approaches for adversarial attacks in network security. First, we classify adversarial attacks in network security based on a taxonomy of network security applications. Secondly, we categorize adversarial attacks in network security into a problem space vs. feature space dimensional classification model. We then analyze the various defenses against adversarial attacks on machine learning-based network security applications. We conclude by introducing an adversarial risk grid map and evaluate several existing adversarial attacks against machine learning in network security using the risk grid map. We also identify where each attack classification resides within the adversarial risk grid map.

Cholec80-Boxes: Bounding Box Labelling Data for Surgical Tools in Cholecystectomy Images

Surgical data analysis is crucial for developing and integrating context-aware systems (CAS) in advanced operating rooms. Automatic detection of surgical tools is an essential component in CAS, as it enables the recognition of surgical activities and understanding the contextual status of the procedure. Acquiring surgical data is challenging due to ethical constraints and the complexity of establishing data recording infrastructures. For machine learning tasks, there is also the large burden of data labelling. Although a relatively large dataset, namely the Cholec80, is publicly available, it is limited to the binary label data corresponding to the surgical tool presence. In this work, 15,691 frames from five videos from the dataset have been labelled with bounding boxes for surgical tool localisation. These newly labelled data support future research in developing and evaluating object detection models, particularly in the laparoscopic image data analysis domain.

Understanding fatigue and recovery mechanism in Hf0.5Zr0.5O2 capacitors for designing high endurance ferroelectric memory and neuromorphic hardware

Novel non-volatile memories are under intense investigation to revolutionize information processing for ultra energy-efficient implementation of artificial intelligence and machine learning tasks. Ferroelectric memories with ultra-low power and fast operation,...

Polynomial-based kernel reproduced gradient descent for stochastic optimization

Stochastic gradient descent (SGD) is widely applied to machine learning tasks. However, the existence of gradient variance degrades the convergence performance of SGD, which cannot be tackled well for tasks with one-time data sampling per sample. In this paper, a kernel reproduced gradient descent algorithm is proposed where the gradient of the risk function is estimated by utilizing the Reproducing Kernel Hilbert Space (RKHS) theory. Meanwhile, to ensure a low storage requirement, a kernel function linearization strategy is used to design the polynomial form of the kernel learning algorithm. It is shown that the error resulting from linearization decreases as the model parameter size increases, and the proposed polynomial-like kernel reproduced gradient descent (PKRGD) algorithm can converge faster than SGD. From experiments on models such as the latest image matching model, LightGlue, it is verified that the reproduced gradients have lower variances, and existing optimizers using the reproduced gradients can further accelerate model training.

ToSiM-IoT: Towards a Sustainable Optimisation of Machine Learning Tasks in Internet of Things

ToSiM-IoT: Towards a Sustainable Optimisation of Machine Learning Tasks in Internet of Things

Interpretable machine learning for time-to-event prediction in medicine and healthcare.

Time-to-event prediction, e.g. cancer survival analysis or hospital length of stay, is a highly prominent machine learning task in medical and healthcare applications. However, only a few interpretable machine learning methods comply with its challenges. To facilitate a comprehensive explanatory analysis of survival models, we formally introduce time-dependent feature effects and global feature importance explanations. We show how post-hoc interpretation methods allow for finding biases in AI systems predicting length of stay using a novel multi-modal dataset created from 1235 X-ray images with textual radiology reports annotated by human experts. Moreover, we evaluate cancer survival models beyond predictive performance to include the importance of multi-omics feature groups based on a large-scale benchmark comprising 11 datasets from The Cancer Genome Atlas (TCGA). Model developers can use the proposed methods to debug and improve machine learning algorithms, while physicians can discover disease biomarkers and assess their significance. We contribute open data and code resources to facilitate future work in the emerging research direction of explainable survival analysis.

ENSEMBLE OF SIMPLE SPIKING NEURAL NETWORKS AS A CONCEPT DRIFT DETECTOR

Context. This paper provides a new approach in concept drift detection using an ensemble of simple spiking neural networks. Such approach utilizes an event-based nature and built-in ability to learn spatio-temporal patterns of spiking neurons, while ensemble provides additional robustness and scalability. This can help solve an active problem of limited time and processing resources in tasks of online machine learning, especially in very strict environments like IoT which also benefit in other ways from the use of spiking computations. Objective. The aim of the work is the creation of an ensemble of simple spiking neural networks to act as a concept drift detector in the tasks of online data stream mining. Method. The proposed approach is primary based on the accumulative nature of spiking neural networks, especially Leaky Integrate-and-Fire neurons can be viewed as gated memory units, where membrane time constant Гm is a balance constant between remembering and forgetting information. A training algorithm is implemented that utilizes a shallow two-layer SNN, which takes features and labels of the data as an input layer and the second layer consists of a single neuron. This neuron’s activation implies that an abrupt drift has occurred. In addition to that, such model is used as a base model within the ensemble to improve robustness, accuracy and scalability. Results. An ensemble of shallow two-layer SNNs was implemented and trained to detect abrupt concept drift in the SEA data stream. The ensemble managed to improve accuracy significantly compared to a base model and achieved competitive results to modern state-of-the-art models. Conclusions. Results showcased the viability of the proposed solution, which not only provides a cheap and competitive solution for resource-restricted environments, but also open doors for further research of SNN’s ability to learn spatio-temporal patters in the data streams and other fields.