Data Analysis Tasks Research Articles

Application of machine reading comprehension techniques for named entity recognition in materials science

Materials science is an interdisciplinary field that studies the properties, structures, and behaviors of different materials. A large amount of scientific literature contains rich knowledge in the field of materials science, but manually analyzing these papers to find material-related data is a daunting task. In information processing, named entity recognition (NER) plays a crucial role as it can automatically extract entities in the field of materials science, which have significant value in tasks such as building knowledge graphs. The typically used sequence labeling methods for traditional named entity recognition in material science (MatNER) tasks often fail to fully utilize the semantic information in the dataset and cannot effectively extract nested entities. Herein, we proposed to convert the sequence labeling task into a machine reading comprehension (MRC) task. MRC method effectively can solve the challenge of extracting multiple overlapping entities by transforming it into the form of answering multiple independent questions. Moreover, the MRC framework allows for a more comprehensive understanding of the contextual information and semantic relationships within materials science literature, by integrating prior knowledge from queries. State-of-the-art (SOTA) performance was achieved on the Matscholar, BC4CHEMD, NLMChem, SOFC, and SOFC-Slot datasets, with F1-scores of 89.64%, 94.30%, 85.89%, 85.95%, and 71.73%, respectively in MRC approach. By effectively utilizing semantic information and extracting nested entities, this approach holds great significance for knowledge extraction and data analysis in the field of materials science, and thus accelerating the development of material science.Scientific contributionWe have developed an innovative NER method that enhances the efficiency and accuracy of automatic entity extraction in the field of materials science by transforming the sequence labeling task into a MRC task, this approach provides robust support for constructing knowledge graphs and other data analysis tasks.

Caching or not: An online cost optimization algorithm for geodistributed data analysis in cloud environments

With the wide application of big data technology, a large number of data geographically stored in data centers across various regions are generated everyday, waiting to be analyzed by big data tasks. Examples of such data analysis tasks include weather prediction and intelligent healthcare applications. Clouds are being used by more and more enterprises due to their nearly infinite resources, ease of scaling, and other characteristics. Organizations often rent StaaS (Storage-as-a-Service) storage products offered by cloud providers, such as OSS (Object Storage Service), for massive data storage, while building a big data cluster in cloud environments to analyze the collected data from various regions. However, when the data to be analyzed are not located in the rented cluster, how to efficiently and economically process the distributed input data stored in clouds becomes an urgent problem to be solved. A simple approach is to only cache frequently accessed data rather than all data from other regions into the cluster to reduce total traffic costs. However, it is generally very hard to predict future data access curves. Thus, a rash caching decision may incur more costs. To address this problem, in this paper we propose an online algorithm for guiding cloud users to make cost-effective caching decisions properly, while not requiring any future information. We prove theoretically that the competitive ratio of our online algorithm is less than 2. Finally we verify the effectiveness of our proposed algorithm through extensive experiments based on the real price of Alibaba’s public IaaS cloud products using both real-world Yahoo S2 data and synthesized datasets.

A Hybrid Deep Spatiotemporal Attention‐Based Model for Parkinson's Disease Diagnosis Using Resting State EEG Signals

ABSTRACTParkinson's disease (PD), a severe and progressive neurological illness, affects millions of individuals worldwide. For effective treatment and management of PD, an accurate and early diagnosis is crucial. This study presents a deep learning‐based model for the diagnosis of PD using a resting state electroencephalogram (EEG) signal. The objective of the study is to develop an automated model that can extract complex hidden nonlinear features from EEG and demonstrate its generalizability on unseen data. The model is designed using a hybrid model, consisting of a convolutional neural network (CNN), bidirectional gated recurrent unit (Bi‐GRU), and attention mechanism. The proposed method is evaluated on three public datasets (UC San Diego, PRED‐CT, and University of Iowa [UI] dataset), with one dataset used for training and the other two for evaluation. The proposed model demonstrated remarkable performance, attaining high accuracy scores of 99.4%, 84%, and 73.2% using UC San Diego, PRED‐CT, and UI datasets, respectively. These results justify the effectiveness and robustness of the proposed model across diverse datasets, highlighting its potential for versatile applications in data analysis and prediction tasks. Our proposed hybrid spatiotemporal attention‐based model has been developed with 10‐fold cross‐validation (CV) for UC San Diego dataset and 10‐fold CV and leave‐one‐out cross‐validation (LOOCV) strategies for PRED‐CT and UI datasets. Our results indicate that the proposed PD detection system is accurate and robust. The developed prototype can be used for other neurodegenerative diseases such as Alzheimer's disease, Huntington's disease, and so forth.

Multi-scale graph clustering network

Deep graph clustering, a fundamental yet formidable task in data analysis, aims to partition samples belonging to the same category into their respective clusters. Recently, significant advancements in graph self-supervised learning have been made through generative and contrastive learning methods. However, existing methods focus on directly aggregating neighboring node information during the feature extraction stage, thereby neglecting the crucial long-range correlations between nodes. Consequently, non-neighbor node information within the same category remains unexplored, leading to subpar performance in the clustering task. To address this issue, we propose a generative method named Multi-scale Graph Clustering Network (MGCN) to learn comprehensive and rich graph representations for deep graph clustering in the feature encoding stage. Specifically, we design a Multi-hop Adaptive Convolutional Module (MACM) integrated into MGCN, which effectively aggregates high-order neighbor node features in each layer of the network. Additionally, we develop an autoencoder to assist MACM in enhancing attribute information, which prevents the node's own features from being overshadowed in the multi-scale feature learning process. Experimental results demonstrate that our proposed MGCN method achieves significantly better clustering performance than existing methods on multiple public datasets.

The Renoir Dataflow Platform: Efficient Data Processing without Complexity

Today, data analysis drives the decision-making process in virtually every human activity. This demands for software platforms that offer simple programming abstractions to express data analysis tasks and that can execute them in an efficient and scalable way. State-of-the-art solutions range from low-level programming primitives, which give control to the developer about communication and resource usage, but require significant effort to develop and optimize new algorithms, to high-level platforms that hide most of the complexities of parallel and distributed processing, but often at the cost of reduced efficiency.To reconcile these requirements, we developed Renoir, a novel distributed data processing platform written in Rust. Renoir provides a high-level dataflow programming model as mainstream data processing systems. It supports static and streaming data, it enables data transformations, grouping, aggregation, iterative computations, and time-based analytics, and it provides all these features incurring in a low overhead.In this paper, we present the programming model and the implementation details of Renoir. We evaluate it under heterogeneous workloads. We compare it with state-of-the-art solutions for data analysis and high-performance computing, as well as alternative research products, which offer different programming abstractions and implementation strategies. Renoir programs are compact and easy to write: developers need not care about low-level concerns such as resource usage, data serialization, concurrency control, and communication. At the same time, Renoir consistently presents comparable or better performance than competing solutions, by a large margin in several scenarios.We conclude that Renoir offers a good tradeoff between simplicity and performance, allowing developers to easily express complex data analysis tasks and achieve high performance and scalability.

Synergizing human expertise and AI efficiency with language model for microscopy operation and automated experiment design * *Notice: This manuscript has been authored by UT-Battelle, LLC, under Contract No. DE-AC0500OR22725 with the U.S. Department of Energy. The United States Government retains and the publisher, by accepting the article for publication, acknowledges that the United States Government retains a non-exclusive, paid-up,

With the advent of large language models (LLMs), in both the open source and proprietary domains, attention is turning to how to exploit such artificial intelligence (AI) systems in assisting complex scientific tasks, such as material synthesis, characterization, analysis and discovery. Here, we explore the utility of LLMs, particularly ChatGPT4, in combination with application program interfaces (APIs) in tasks of experimental design, programming workflows, and data analysis in scanning probe microscopy, using both in-house developed APIs and APIs given by a commercial vendor for instrument control. We find that the LLM can be especially useful in converting ideations of experimental workflows to executable code on microscope APIs. Beyond code generation, we find that the GPT4 is capable of analyzing microscopy images in a generic sense. At the same time, we find that GPT4 suffers from an inability to extend beyond basic analyses for more in-depth technical experimental design. We argue that an LLM specifically fine-tuned for individual scientific domains can potentially be a better language interface for converting scientific ideations from human experts to executable workflows. Such a synergy between human expertise and LLM efficiency in experimentation can open new doors for accelerating scientific research, enabling effective experimental protocols sharing in the scientific community.

Design and Manufacturing of a Pin on Disk Tribometer

This paper presents the design and manufacturing of a pin on disk tribometer aimed at performing friction and wear testing as accurate as a professional tribometer at a lower cost. Since friction is an important part of our lives and its application varies greatly in size, environment and other factors, testing can be expensive. Tribometers are instruments where these conditions can be simulated experimentally. They are manufactured by various companies abroad and are imported to Turkey. For this reason, the cost of these instruments is very high. The design of the proposed tribometer prioritizes portability and low cost of manufacturing. The manufacturing process of the various parts are discussed in detail. The tribometer was manufactured in the ATILIM University manufacturing laboratories within a tight budget using CNC machining. The instrument has the full capability to adjust the load, testing radius, motor speed, and the duration of the test before starting the experiments. To handle the electronics and data gathering within the instrument, Arduino boards and software are utilized. Real-time data streaming is made possible for data analysis tasks both during and after the experiments. The paper also provides the results of experiments that are obtained from the newly developed tribometer and comparisons with a mass produced, professional tribometer. The experiments are conducted with the same type of samples and with the same parameters. The proposed tribometer provides researchers with easily accessible friction data that is within close range to what they would get from a professional tribometer.

Reservoir Sampling over Joins

Sampling over joins is a fundamental task in large-scale data analytics. Instead of computing the full join results, which could be massive, a uniform sample of the join results would suffice for many purposes, such as answering analytical queries or training machine learning models. In this paper, we study the problem of how to maintain a random sample over joins while the tuples are streaming in. Without the join, this problem can be solved by some simple and classical reservoir sampling algorithms. However, the join operator makes the problem significantly harder, as the join size can be polynomially larger than the input. We present a new algorithm for this problem that achieves a near-linear complexity. The key technical components are a generalized reservoir sampling algorithm that supports a predicate, and a dynamic index for sampling over joins. We also conduct extensive experiments on both graph and relational data over various join queries, and the experimental results demonstrate significant performance improvement over the state of the art.

Semantic-enabled architecture for auditable privacy-preserving data analysis

Small and medium-sized organisations face challenges in acquiring, storing and analysing personal data, particularly sensitive data (e.g., data of medical nature), due to data protection regulations, such as the GDPR in the EU, which stipulates high standards in data protection. Consequently, these organisations often refrain from collecting data centrally, which means losing the potential of data analytics and learning from aggregated user data. To enable organisations to leverage the full-potential of the collected personal data, two main technical challenges need to be addressed: (i) organisations must preserve the privacy of individual users and honour their consent, while (ii) being able to provide data and algorithmic governance, e.g., in the form of audit trails, to increase trust in the result and support reproducibility of the data analysis tasks performed on the collected data. Such an auditable, privacy-preserving data analysis is currently challenging to achieve, as existing methods and tools only offer partial solutions to this problem, e.g., data representation of audit trails and user consent, automatic checking of usage policies or data anonymisation. To the best of our knowledge, there exists no approach providing an integrated architecture for auditable, privacy-preserving data analysis. To address these gaps, as the main contribution of this paper, we propose the WellFort approach, a semantic-enabled architecture for auditable, privacy-preserving data analysis which provides secure storage for users’ sensitive data with explicit consent, and delivers a trusted, auditable analysis environment for executing data analytic processes in a privacy-preserving manner. Additional contributions include the adaptation of Semantic Web technologies as an integral part of the WellFort architecture, and the demonstration of the approach through a feasibility study with a prototype supporting use cases from the medical domain. Our evaluation shows that WellFort enables privacy preserving analysis of data, and collects sufficient information in an automated way to support its auditability at the same time.

An inversion-based clustering approach for complex clusters

BackgroundThe choice of an appropriate similarity measure plays a pivotal role in the effectiveness of clustering algorithms. However, many conventional measures rely solely on feature values to evaluate the similarity between objects to be clustered. Furthermore, the assumption of feature independence, while valid in certain scenarios, does not hold true for all real-world problems. Hence, considering alternative similarity measures that account for inter-dependencies among features can enhance the effectiveness of clustering in various applications.MethodsIn this paper, we present the Inv measure, a novel similarity measure founded on the concept of inversion. The Inv measure considers the significance of features, the values of all object features, and the feature values of other objects, leading to a comprehensive and precise evaluation of similarity. To assess the performance of our proposed clustering approach that incorporates the Inv measure, we evaluate it on simulated data using the adjusted Rand index.ResultsThe simulation results strongly indicate that inversion-based clustering outperforms other methods in scenarios where clusters are complex, i.e., apparently highly overlapped. This showcases the practicality and effectiveness of the proposed approach, making it a valuable choice for applications that involve complex clusters across various domains.ConclusionsThe inversion-based clustering approach may hold significant value in the healthcare industry, offering possible benefits in tasks like hospital ranking, treatment improvement, and high-risk patient identification. In social media analysis, it may prove valuable for trend detection, sentiment analysis, and user profiling. E-commerce may be able to utilize the approach for product recommendation and customer segmentation. The manufacturing sector may benefit from improved quality control, process optimization, and predictive maintenance. Additionally, the approach may be applied to traffic management and fleet optimization in the transportation domain. Its versatility and effectiveness make it a promising solution for diverse fields, providing valuable insights and optimization opportunities for complex and dynamic data analysis tasks.

Data-Driven Strategies for Complex System Forecasts: The Role of Textual Big Data and State-Space Transformers in Decision Support

In this research, an innovative state space-based Transformer model is proposed to address the challenges of complex system prediction tasks. By integrating state space theory, the model aims to enhance the capability to capture dynamic changes in complex data, thereby improving the accuracy and robustness of prediction tasks. Extensive experimental validations were conducted on three representative tasks, including legal case judgment, legal case translation, and financial data analysis to assess the performance and application potential of the model. The experimental results demonstrate significant performance improvements of the proposed model over traditional Transformer models and other advanced variants such as Bidirectional Encoder Representation from Transformers (BERT) and Finsformer across all evaluated tasks. Specifically, in the task of legal case judgment, the proposed model exhibited a precision of 0.93, a recall of 0.90, and an accuracy of 0.91, significantly surpassing the traditional Transformer model (with precision of 0.78, recall of 0.73, accuracy of 0.76) and performances of other comparative models. In the task of legal case translation, the precision of the proposed model reached 0.95, with a recall of 0.91 and an accuracy of 0.93, also outperforming other models. Likewise, in the task of financial data analysis, the proposed model also demonstrated excellent performance, with a precision of 0.94, recall of 0.90, and accuracy of 0.92. The state space-based Transformer model proposed not only theoretically expands the research boundaries of deep learning models in complex system prediction but also validates its efficiency and broad application prospects through experiments. These achievements provide new insights and directions for future research and development of deep learning models, especially in tasks requiring the understanding and prediction of complex system dynamics.

Multi-Scale Fusion Siamese Network Based on Three-Branch Attention Mechanism for High-Resolution Remote Sensing Image Change Detection

Remote sensing image change detection (CD) is an important means in remote sensing data analysis tasks, which can help us understand the surface changes in high-resolution (HR) remote sensing images. Traditional pixel-based and object-based methods are only suitable for low- and medium-resolution images, and are still challenging for complex texture features and detailed image detail processing in HR images. At present, the method based on deep learning has problems such as inconsistent fusion and difficult model training in the combination of the difference feature information of the deep and shallow layers and the attention mechanism, which leads to errors in the distinction between the changing region and the invariant region, edge detection and small target detection. In order to solve the above problems of inconsistent fusions of feature information aggregation and attention mechanisms, and indistinguishable change areas, we propose a multi-scale feature fusion Siamese network based on attention mechanism (ABMFNet). To tackle the issues of inconsistent fusion and alignment difficulties when integrating multi-scale fusion and attention mechanisms, we introduce the attention-based multi-scale feature fusion module (AMFFM). This module not only addresses insufficient feature fusion and connection between different-scale feature layers, but also enables the model to automatically learn and prioritize important features or regions in the image. Additionally, we design the cross-scale fusion module (CFM) and the difference feature enhancement pyramid structure (DEFPN) to assist the AMFFM module in integrating differential information effectively. These modules bridge the spatial disparity between low-level and high-level features, ensuring efficient connection and fusion of spatial difference information. Furthermore, we enhance the representation and inference speed of the feature pyramid by incorporating a feature enhancement module (FEM) into DEFPN. Finally, the BICD dataset proposed by the laboratory and public datasets LEVIR-CD and BCDD are compared and tested. We use F1 score and MIoU values as evaluation metrics. For AMBMFNet, the F1 scores on the three datasets are 77.69%, 81.57%, and 77.91%, respectively, while the MIoU values are 84.65%, 85.84%, and 84.54%, respectively. The experimental results show that ABMFNet has better effectiveness and robustness.

Knowledge-integrated autoencoder model

Data encoding is a common and central operation in most data analysis tasks. The performance of other models downstream in the computational process highly depends on the quality of data encoding. One of the most powerful ways to encode data is using the neural network AutoEncoder (AE) architecture. However, the developers of AE cannot easily influence the produced embedding space, as it is usually treated as a black box technique. This means the embedding space is uncontrollable and does not necessarily possess the properties desired for downstream tasks. This paper introduces a novel approach for developing AE models that can integrate external knowledge sources into the learning process, possibly leading to more accurate results. The proposed Knowledge-integrated AutoEncoder (KiAE) model can leverage domain-specific information to make sure the desired distance and neighborhood properties between samples are preservative in the embedding space. The proposed model is evaluated on three large-scale datasets from three scientific fields and is compared to nine existing encoding models. The results demonstrate that the KiAE model effectively captures the underlying structures and relationships between the input data and external knowledge, meaning it generates a more useful representation. This leads to outperforming the rest of the models in terms of reconstruction accuracy.

A Study on Collaborative Visual Data Analysis in Augmented Reality with Asymmetric Display Types.

Collaboration is a key aspect of immersive visual data analysis. Due to its inherent benefit of seeing co-located collaborators, augmented reality is often useful in such collaborative scenarios. However, to enable the augmentation of the real environment, there are different types of technology available. While there are constant developments in specific devices, each of these device types provide different premises for collaborative visual data analysis. In our work we combine handheld, optical see-through and video see-through displays to explore and understand the impact of these different device types in collaborative immersive analytics. We conducted a mixed-methods collaborative user study where groups of three performed a shared data analysis task in augmented reality with each user working on a different device, to explore differences in collaborative behaviour, user experience and usage patterns. Both quantitative and qualitative data revealed differences in user experience and usage patterns. For collaboration, the different display types influenced how well participants could participate in the collaborative data analysis, nevertheless, there was no measurable effect in verbal communication.

Evaluating Diffusion Models for the Automation of Ultrasonic Nondestructive Evaluation Data Analysis

We develop decision support and automation for the task of ultrasonic non-destructive evaluation data analysis. First, we develop a probabilistic model for the task and then implement the model as a series of neural networks based on Conditional Score-Based Diffusion and Denoising Diffusion Probabilistic Model architectures. We use the neural networks to generate estimates for peak amplitude response time of flight and perform a series of tests probing their behavior, capacity, and characteristics in terms of the probabilistic model. We train the neural networks on a series of datasets constructed from ultrasonic non-destructive evaluation data acquired during an inspection at a nuclear power generation facility. We modulate the partition classifying nominal and anomalous data in the dataset and observe that the probabilistic model predicts trends in neural network model performance, thereby demonstrating a principled basis for explainability. We improve on previous related work as our methods are self-supervised and require no data annotation or pre-processing, and we train on a per-dataset basis, meaning we do not rely on out-of-distribution generalization. The capacity of the probabilistic model to predict trends in neural network performance, as well as the quality of the estimates sampled from the neural networks, support the development of a technical justification for usage of the method in safety-critical contexts such as nuclear applications. The method may provide a basis or template for extension into similar non-destructive evaluation tasks in other industrial contexts.

HE-Gaston algorithm for frequent subgraph mining with hadoop framework

Graph mining contributes a key role in data mining and as the size of the data increases, it becomes complicated. Identifying the interesting subgraphs in the graph is a commonly researched issue, where the subgraphs denote the commonly occurring pattern exhibiting a particular structure. Frequent Subgraph Mining (FSM) is an important task for exploratory data analysis on graph data.Though many techniques are proposed for FSM, the large dimension of the data makes FSM complex. This research proposes a novel technique for performing FSM in a Hadoop framework. Here, FSM is carried out using the proposed Holoentropy Gaston algorithm (HE-Gaston), which is developed by incorporating the Holoentropy support measure instead of the Recurrent support measure in the Recurrent-Gaston (R-Gaston) technique. Here, the weblog files are considered for FSM and are fed to the Spark framework, which encompasses a master and several slaves. The slave nodes generate the frequent subgraphs based on the Holoentropy support measure and the generated frequent subgraphs are applied to the master node which produces the final frequent subgraphs by utilizing an aggregate Holoentropy support measure. Further, the HE-Gaston shows that it recorded execution time of 43 ms, memory of 2.161 MB, and number of structures mined as 117.

Dynamic Adaptive Resource Allocation for Edge Computing in Big Data Analytics Using GBDT, DQN, and GA Algorithms

Edge computing in big data refers to processing and analysing data closer to its source, reducing latency and bandwidth usage. It leverages devices at the network edge to perform computations, making real-time analytics feasible. This distributed approach improves efficiency and enables faster decision-making, critical for applications like IoT, autonomous vehicles, and healthcare. The research proposes an innovative approach that harnesses three machine learning algorithms Gradient Boosting Decision Trees (GBDT), Deep Q-Network (DQN), and Genetic Algorithm (GA) to enable dynamic adaptive resource allocation within edge computing environments tailored for big data analytics. GBDT enhances classification accuracy by sequentially refining predictions through decision trees, accommodating heterogeneous data types and yielding high prediction accuracy crucial for dynamic edge environments. The GA evaluates resource allocation strategies represented as chromosomes within a population, selecting promising solutions as parents for the next generation and generating diverse offspring through crossover and mutation operations to discover optimal solutions. DQN facilitates intelligent resource allocation by iteratively refining Q-values based on experiences gathered during interactions with the environment, utilizing a neural network to determine optimal actions for a given state, thereby enhancing performance and efficiency in edge computing environments. This integrated approach ensures flexible resource allocation and fortified capabilities for big data analytics within edge computing environments. The research underscores GBDT as the most promising algorithm for resource allocation in edge computing environments, owing to its exceptional performance in resource utilization, scalability, and accuracy. This nuanced understanding of algorithmic behaviour in dynamic settings offers invaluable insights for optimizing resource allocation strategies, thereby enhancing the efficiency and effectiveness of edge computing systems in handling big data analytics tasks.

Визуализация в задачах реконструкции данных

Many application tasks of multidimensional data analysis which describe the state of real physical or other systems face with difficulties. This is a consequence of the low-quality source data, including missing values, the probability of errors or unreliability of measurements. Incomplete data can become an obstacle for research using many modern informational methods. The current work examines the potential and capabilities of visual analytics tools for preliminary preparation, correction or complete analysis of primary data volumes. A promising area of application of the approach discussed in the study is the targeted use of visualization capabilities as a data analysis tool. The implementation of specialized visual metaphors is used to solve problems of processing and interpreting data, the sources of which are cyberphysical systems of different complexity levels. Such systems operate in an autonomous or partially controlled mode. A characteristic feature of these systems is the presence of a large number of sensors that collect various types of data. Such data differ in the capacity of the corresponding information channels, their speed and reliability. Examples of such cyberphysical systems are unmanned aerial vehicles (UAVs), robotic stations, and multimodal monitoring systems. These systems can function in conditions where it is difficult to obtain objective observation experience (deep-sea robots). The effective use of data collected by cyberphysical monitoring systems is a condition for solving a large number of application and research tasks.

Performance Analysis of Tree-Based and Deep Learning Algorithms for Developing Distributed Secure Systems in IoT: A Comparative Study

Notably, IoT device utilization has experienced a substantial wave recently, and ensuring these devices' privacy and security has become a critical concern. ML-based security approaches are promising for IoT network protection against security concerns. This study provides a proximate analysis of tree-based and deep-learning algorithms for securing IoT domains. Specifically, we evaluate Decision Tree, RandomForest, XGBoost, Catboost, Extreme Tree, Light GMB, Adaptive Boosting, CNN, LSTM, MLP, GRU, and Autoencoder on four publicly available datasets - IoT23, CICID2017, EdgeIIoT, BotnetIoT and Contiki OS and Cooja simulation were used to generate a dataset featuring various RPL attacks. To assess the performance of a model, we measure its accuracy, precision, recall, and F1-score metrics. Our discoveries indicate that tree-based algorithms outperform deep learning algorithms regarding training time, memory usage, and interpretability while gaining comparable or even better detection accurateness. Conversely, deep-learning algorithms exhibit higher detection rates for rare or previously unseen attacks; their proficiency in detecting complex patterns and relationships within a given dataset has demonstrated remarkable efficacy in data analysis and classification tasks. We conclude that both tree-based and deep learning algorithms have their strengths and weaknesses, and in the IoT environment, one should base the choice of the algorithm on requirements and constraints. Our research shows hybrid approaches combining algorithm strengths can establish secure, distributed IoT systems

Mathematical programming for simultaneous feature selection and outlier detection under l1 norm

The goal of simultaneous feature selection and outlier detection is to determine a sparse linear regression vector by fitting a dataset possibly affected by the presence of outliers.The problem is well-known in the literature. In its basic version it covers a wide range of tasks in data analysis. Simultaneously performing feature selection and outlier detection strongly improves the application potential of regression models in more general settings, where data governance is a concern. To trigger this potential, flexible training models are needed, with more parameters under control of decision makers.The use of mathematical programming, although pertinent, is scarce in this context and mostly focusing on the least-squares setting. Instead we consider the least absolute deviation criterion, proposing two mixed-integer linear programs, one adapted from existing studies, and the other obtained from a disjunctive programming argument. We show theoretically and computationally that the disjunctive-based formulation is better in terms of both continuous relaxation quality and integer optimality convergence.We experimentally benchmark against existing methodologies from the literature. We identify the characteristics of contamination patterns, in which mathematical programming is better than state-of-the-art algorithms in combining prediction quality, sparsity and robustness against outliers. Additionally, the mathematical programming approaches allow the decision maker to directly control parameters like the number of features or outliers to tolerate, those based on least absolute deviations performing best. On real world datasets, where privacy is a concern, our approach compares well to state-of-the-art methods in terms of accuracy, being at the same time more flexible.