Data Processing Flow Research Articles

Design and implementation of breast cancer prediction system based on machine learning

With the increasing incidence of breast cancer worldwide, early diagnosis and treatment of breast cancer has become the key to improving patient survival and quality of life. As a powerful data analysis tool, machine learning is increasingly widely used in the medical field, especially in disease prediction and assisted diagnosis. This paper aims to design and implement a machine learning-based breast cancer prediction system to improve the early diagnosis rate of breast cancer and reduce medical costs. Through an in-depth analysis of the global incidence of breast cancer and the current application of machine learning in the medical field, this study clarified the importance of breast cancer prediction and the problems existing in the existing prediction system. This paper further discusses the theoretical basis of machine learning in breast cancer detection, evaluates the advantages and disadvantages of commonly used machine learning algorithms, and reviews the latest research progress in this field at home and abroad. In the part of system design and implementation, the architecture design, data flow and processing process of the prediction system, as well as the method of data preprocessing and feature selection are introduced in detail. In addition, this paper also constructs a machine learning model suitable for breast cancer prediction, and carries out systematic implementation and testing through the actual development environment. In the discussion section, the applicability of machine learning model in breast cancer prediction is analyzed, the causes of model inefficiency are discussed, and the corresponding solutions are proposed. Finally, the paper summarizes the content of the full text, points out the limitations of the research, and puts forward the direction of future research. The results of this study not only provide a new technical means for the early diagnosis of breast cancer, but also provide valuable experience for the application of machine learning in the medical field.

Sparse seismic data regularization in shot and trace domains using a residual block autoencoder based on the fast Fourier transform

The increasing use of sparse acquisitions in seismic data acquisition offers advantages in cost and time savings. However, it results in irregularly sampled seismic data, adversely impacting the quality of the final images. In this paper, we develop the residual block fast Fourier transform-convolutional autoencoder (ResFFT-CAE) network, a convolutional neural network with residual blocks based on the Fourier transform. Incorporating residual blocks allows the network to extract high- and low-frequency features from seismic data. The high-frequency features capture detailed information, whereas the low-frequency features integrate the overall data structure, facilitating superior recovery of irregularly sampled seismic data in the trace and shot domains. We evaluate the performance of the ResFFT-CAE network on the synthetic and field data. On synthetic data, we compare the ResFFT-CAE network with the compressive sensing method using the curvelet transform. On field data, we conduct comparisons with other neural networks, such as the CAE and U-Net. The results demonstrate that the ResFFT-CAE network consistently outperforms other approaches in all scenarios. It produces images of superior quality, characterized by lower residuals and reduced distortions. Furthermore, when evaluating model generalization, tests using models trained on synthetic data also exhibit promising results. In conclusion, the ResFFT-CAE network indicates great promise as a highly efficient tool for regularizing irregularly sampled seismic data. Its excellent performance suggests potential applications in the preconditioning of seismic data analysis and processing flows.

AutonoMS: Automated Ion Mobility Metabolomic Fingerprinting.

Automation is dramatically changing the nature of laboratory life science. Robotic lab hardware that can perform manual operations with greater speed, endurance, and reproducibility opens an avenue for faster scientific discovery with less time spent on laborious repetitive tasks. A major bottleneck remains in integrating cutting-edge laboratory equipment into automated workflows, notably specialized analytical equipment, which is designed for human usage. Here we present AutonoMS, a platform for automatically running, processing, and analyzing high-throughput mass spectrometry experiments. AutonoMS is currently written around an ion mobility mass spectrometry (IM-MS) platform and can be adapted to additional analytical instruments and data processing flows. AutonoMS enables automated software agent-controlled end-to-end measurement and analysis runs from experimental specification files that can be produced by human users or upstream software processes. We demonstrate the use and abilities of AutonoMS in a high-throughput flow-injection ion mobility configuration with 5 s sample analysis time, processing robotically prepared chemical standards and cultured yeast samples in targeted and untargeted metabolomics applications. The platform exhibited consistency, reliability, and ease of use while eliminating the need for human intervention in the process of sample injection, data processing, and analysis. The platform paves the way toward a more fully automated mass spectrometry analysis and ultimately closed-loop laboratory workflows involving automated experimentation and analysis coupled to AI-driven experimentation utilizing cutting-edge analytical instrumentation. AutonoMS documentation is available at https://autonoms.readthedocs.io.

The influence of current results in a event­oriented data collection system

The article discusses synchronous and asynchronous procedures in the implementation of the microservices management algorithm in an adaptive system for processing large data flows when collecting information on the main event-oriented approach in the implementation of the architecture of a software system that processes information in real-time. This approach is important when processing large volumes of data from heterogeneous information sources, especially when the task is to minimize the total processing time of large data streams and reduce the total number of calls to data sources. The proposed approach makes it possible, based on its adaptability, to manage the selection of the composition and the number of calls of microservices to the sources, by the events that occur during the collection of information. Thus, it is possible to determine the choice of sources based on the assessment of the effectiveness of obtaining relevant information from them. This is especially important when processing large data flows from heterogeneous information sources when the task is to minimize the total time of collection and processing of large data flows. In turn, this poses the task of minimizing the number of requests to information sources to obtain a sufficient number of data units that are relevant to the search query. The creation of effective big data processing systems requires constant development of approaches to the architecture of building software applications. The event-oriented microservice architecture of the system makes it possible to adapt the operation of the system to the loads on individual microservices and the efficiency of their work by forming and responding to relevant events based on the analysis of relevant events that occur during the collection and initial processing of the received data. Depending on the specific task, it is possible to use both a synchronous and an asynchronous microservice management algorithm. The article provides an analysis of the effectiveness of obtaining relevant data depending on the degree of consideration in the evaluation of results and the formation of events, both the depth of the previous history of the results of requests and the size of their quantitative impact. The use of event-oriented microservice architecture can be especially effective when developing various information and analytical systems that need, according to user requests, to access various sources of information, analyze their data for relevance according to the request, and process large data streams.

An efficient DDoS detection method based on packet grouping via online data flow processing

An efficient DDoS detection method based on packet grouping via online data flow processing

APEIRON: A Framework for High Level Programming of Dataflow Applications on Multi-FPGA Systems

High Energy Physics (HEP) Trigger and Data Acquisition systems (TDAQs) need ever increasing throughput and real-time data analytics capabilities either to improve particle identification accuracy and further suppress background events in trigger systems or to perform an efficient online data reduction for trigger-less ones. As for the requirements imposed by HEP TDAQs applications in the class of real-time dataflow processing, FPGA devices are a good fit inasmuch they can not only provide adequate compute, memory and I/O resources but also a smooth programming experience thanks to the availability of High-Level Synthesis (HLS) tools. The main motivation for the design and development of the APEIRON framework is that the currently available HLS tools do not natively support the deployment of applications over multiple FPGA devices, which severely chokes the scalability of problems that this approach could tackle. To overcome this limitation, we envisioned APEIRON as an extension of the Xilinx Vitis framework able to support a network of FPGA devices interconnected by a lowlatency direct network as the reference execution platform. Developers can define scalable applications, using a dataflow programming model inspired by Kahn Process Networks, that can be efficiently deployed on a multi-FPGAs system: the APEIRON communication IPs allow low-latency communication between processing tasks deployed on FPGAs, even if they are hosted on different computing nodes. Thanks to the use of HLS tools in the workflow, processing tasks are described in C++ as HLS kernels, while communication between tasks is expressed through a lightweight C++ API based on non-blocking send() and blocking receive() operations.

Assistance for The Use of The Microfinance Application for Managers of Bumdes Jati Unggul

Abstract. This service program aims to increase the technical capacity of managers of Jati Unggul BUMDes (Partner) Financial Service Unit in using the Microfinance Application. This microfinance application was previously called the LKM Application which was a product of the IbM service team in 2014. Due to changes in business processes from PERUMDES to BUMDES, the LKM Application underwent improvements and metamorphosed into a Microfinance Application. The methods for implementing this service program were conducting surveys and observations, preparing training and mentoring materials, training and assisting in using the Microfinance Application, and reporting on the results of the service. The results and discussion of this service program are as follows: 1) The managers of the Jati Unggul BUMDES financial services unit need to strengthen their knowledge and technical skills in using the Microfinance Application for handling customer service, recording transactions, and preparing financial reports. 2) Training materials were prepared according to the needs and data process flow which includes; data entry, settings, account mapping, services, reports, and user policies. 3) Training was carried out for 2 effective days with the concept of problem-based learning. 4) Technical assistance was carried out for 12 effective days directly in customer service practices and preparation of financial reports using the Microfinance Application. 5) After participating in training and mentoring, the managers of the Jati Unggul BUMDES financial services unit gained the technical ability to use microfinance applications in service activities, recording transactions, and preparing financial reports.

Problematizing feedback loops: ‘on’, ‘with’, and ‘beyond’ analytics dashboards in MOOCs

ABSTRACT This paper examines the enactment of feedback in Massive Open Online Courses (MOOCs), focusing on analytics dashboards. Building on scholarship that recognizes data practices as entangled and ‘messy’, the paper problematizes the model of the feedback loop that assumes that analytics dashboards ‘feed back’ data to instructors and/or learners through a singular flow of data processing. By setting out an empirical study that focuses on four MOOCs, in two universities and on two platforms, the paper maps where, when, and how design teams, instructors, and learners are involved in the enactment of feedback through and beyond analytics dashboards. The findings draw on visualizations that highlight complex relations among people and technologies, which include multiple ‘loops’. The paper concludes with questioning the need to capture feedback in a singular loop and suggests prioritizing continued attention to the roles and responsibilities of people – educational designers, instructors, and learners – in MOOCs.

AN INTERACTION MODEL OF RELAY PROTECTION MEASURING TRANSDUCERS WITH COMPUTING SYSTEMS VIA IIOT TECHNOLOGY

"The object of this research is the interaction model between measuring transducers and computing systems, particularly focusing on its application within the framework of Industrial Internet of Things (IIoT) and Cloud technologies. The imperative driving this study is the increasing demand for precise and efficient data acquisition, protection and processing in various industrial sectors. The research aims to address this need by elucidating an interaction model that optimizes measuring computing systems through IIoT and Cloud technologies. As a result of our research, a comprehensive interaction model that encompasses the entire data flow process was established. This model integrates data acquisition, analytics, real-time communication protocols, and secure data transmission methodologies. It offers a structured framework for acquiring, transmitting, and analyzing data with high accuracy and efficiency. The significance of our results lies in their capacity to resolve critical challenges in data acquisition and processing of the electrical power data, as well as monitoring and protection against electrical faults. These results hold practical value under diverse industrial conditions, including manufacturing, energy production and energy distribution. Keywords: relay protection, open architecture, encryption, data processing, data visualization, internet of things "

Consistency Constraints for Mapping Dataflow Graphs to Hybrid Dataflow/von Neumann Architectures

Dataflow process networks (DPNs) provide a convenient model of computation that is often used to model system behavior in model-based designs. With fixed sets of nodes, they are also used as dataflow graphs as an intermediate program representation by compilers to uncover instruction-level parallelism of sequential programs. Many recent processor architectures, which are still von Neumann architectures, also use dataflow computing to increase their exploitation of instruction-level parallelism by exposing their datapaths so that the compiler can take care of the allocation of processing units (PUs), the execution schedules of instructions on the PUs, and the communication of intermediate values between PUs. If the communication paths are buffered, these architectures can be abstracted into a DPN architecture whose PUs and interconnection network are DPN nodes. In this article, we introduce a DPN abstraction of hybrid dataflow/von Neumann architectures and consider the mapping of the nodes of a given dataflow graph to the PUs of such a DPN architecture such that there are no conflicts due to the mapping of different nodes to the same PU. We express the allocation and scheduling constraints in terms of propositional logic for the original dataflow graph and for a modified version of the dataflow graph that simplifies the constraints by introducing levels using copy nodes, such that all nodes receive inputs only from nodes of the previous level. We also formulate equisatisfiable SMT constraints using integer variables to reason directly about the parallel runtime. On this basis, we further present alternative SAT constraints that explicitly encode concurrency, and discuss variants of the constraints for a better understanding of the same.

A comprehensive modeling approach for the task mapping problem in heterogeneous systems with dataflow processing units

SummaryWe introduce a new model for the task mapping problem to aid in the systematic design of algorithms for heterogeneous systems including, but not limited to, CPUs, GPUs, and FPGAs. A special focus is set on the communication between the devices, its influence on parallel execution, as well as on device‐specific differences regarding parallelizability and streamability. We give a comprehensive description on how a given task mapping can be abstractly evaluated including mappings to dataflow‐based hardware accelerators. We show how this model can be utilized in different system design phases and present two novel mixed‐integer linear programs to demonstrate the usage of the model, showing significant improvements compared to pure CPU mapping for randomly generated task graphs. To the best of our knowledge, we present the first ILP for task mapping that considers pipelining effects when streaming tasks on an FPGA.

Feature selection for image classification using ESSO

Feature selection is one of the major components of the data processing flow that correctly selects real-time entities for categorization. It is utilized in numerous research fields, including the machine learning approach. Feature selection and classification are beneficial to the processing of biomedical data in high-importance, high-dimensional datasets. Due to numerous obstacles in research, the current implementations are inadequate for predicting classification accuracy. To handle the classification challenge, we require dedicated neural network and classification model approaches. Initial features are selected using a method called "fuzzy c-means clustering with rough set theory" and are subsequently classified using "support vector machines." The learning machines' accuracy is seriously harmed by irrelevant and redundant features. The curse of dimensionality makes it difficult to locate data clusters in high-dimensional space. Data in the irrelevant dimensions may cause a lot of noise when the dimensionality grows. In addition, the current approach has a significant flaw: it is extremely time-consuming. The proposed solution used efficient feature subset selection in high-dimensional data to fix these problems. To begin with, our recommended method used Enhanced Social Spider Optimization (ESSO) computation, in which the standard Social Spider Optimization is improved with the aid of optimal radial based calculation to select the best highlights. When categorizing data, the Optimal Radial Basis Function Neural Network (ORBFNN) is used. Methods for calculating Artificial Bee Colony (ABC) are used to streamline RBFNN's sufficiency in characterizing smaller scale show information.

A generic framework for federated CDEs applied to Issue Management

This paper analyses the requirements for managing interoperable building data in a federated Common Data Environment (CDE). We discuss the need for generic (meta)data storage patterns, semantic query interfaces, decentral authentication, data aggregation, and adaptation and prove that their combination is feasible with current-day technologies. We illustrate the mechanisms of such federated CDE by considering the topic of digital Issue Management, one of the primary functions of a CDE. In an exemplary data flow process, we show how generic (federated, Semantic Web-based) data patterns for Issue Management can be aggregated and restructured to match existing industry standards like buildingSMART’s BIM Collaboration Format (BCF) API. Finally, we show the methodology is compatible with current-day practice by implementing this process in a proof of concept. The main contribution of this research is a generic, federated framework for project-related, interdisciplinary collaboration for CDEs.

Inverse Q-Filtering as a Tool for Seismic Resolution Enhancement: A Case Study from the Carpathian Foredeep

In this study, we analyze the quality factor- Q derived from the Vertical Seismic Profiling (VSP) and sonic log data that was performed in the DS-2 borehole located in the Carpathian Foredeep. We use the obtained quality factor distributions in the inverse Q-filtering (IQF) procedure during active seismic data processing. Another very important and, unfortunately, subjective in nature parameter is the gain function applied to seismic data while performing the IQF. To calculate Q from VSP data we use an iterative approach to the spectral ratio method where we use several shallowest receivers as reference. We observe high variability of the obtained Q values, depending on the reference receiver, however, depth distribution of Q was consistent regardless To increase reliability of the obtained values, an additional method of Q measurements is incorporated, i.e., acoustic travel time dispersion method. We show that incorporating inverse Q-filtering to active seismic data processing flow allows to enhance the vertical resolution of 2D seismic data.

Information systems for automating work flows: a comparative analysis

Workflow is a sequence of repeated and controlled steps aimed at completing a specific task or work. The urgency of the task of optimizing such works contributed to the development of methods and means of operations research to optimize such processes for the needs of various subject areas. Information technologies to support such workflow are workflow engines that enable faster automation, ensure compliance with norms and standards, formalize business processes, improve communication, etc. The workflow management systems are divided into the following categories: automation using robotics, intelligent integration platforms, intelligent business process management systems, open-source engines, the cloud-supported, as well as data flow processing systems. For a more detailed analysis, it is advisable to choose the following engines and services: built-in, cloud-oriented, and those to support both scenarios (jBPM, Camunda, Zeebe, Amazon Step Functions). It is appropriate to define evaluation criteria and compare such workflow automation systems to form further recommendations regarding their selection and application. Such criteria are development activity, stability and history of commercial use, versioning support, standards support, support for timers and asynchronous execution, support for human-oriented and manual tasks, integration with other solutions, monitoring and logging, scaling, cloud support, the possibility of scanning in private infrastructure, the presence of a visual interface, the convenience of local development and testing, open source code, the necessary programming for implementation and cost. A comparison of advantages and disadvantages can be used to decide on a workflow automation system.

An Application Designed for Guiding the Coordinated Charging of Electric Vehicles

Guiding the coordinated charging of electric vehicles can alleviate the load fluctuation of power systems within a local area caused by uncoordinated charging of electric vehicles and greatly reduce the cost of power system operation. This will become an inevitable development trend of future energy system transformation. In this paper, a new mobile application is built to realize the dynamic adjustment of electric vehicle charging prices according to the change in weather conditions to guide the coordinated charging of electric vehicles. After systematically introducing the structure and data flow process of the application, we simulate the fluctuation of charging prices under various weather conditions using the electricity load data of North China and verify the good performance of the application. We believe that this application can help power systems to achieve low-carbon transformation by adopting a new dynamic time-of-use pricing charging model.

Optimizing Data Processing for Nanodiamond Based Relaxometry

AbstractThe nitrogen‐vacancy (NV) center in diamond is a powerful and versatile quantum sensor for diverse quantities. In particular, relaxometry (or T1), can be used to detect magnetic noise at the nanoscale. For experiments with single NV centers the analysis of the data is well established. However, due to relatively low brightness and reproducibility it is beneficial for biological experiments to use ensembles. While increasing the number of NV centers in a nanodiamond leads to more signal, a standardized method to extract information from relaxometry experiments is still missing. This article uses T1 relaxation curves acquired at different concentrations of gadolinium ions to calibrate and optimize the entire data processing flow, from the acquired raw data to the extracted T1. In particular, a bootstrap is used to derive a signal to noise ratio (SNR) that can be quantitatively compared from one method to another. At first, T1 curves are extracted from photoluminescence pulses. This work compares integrating their signal through an optimized window as performed conventionally, to fitting a known function on it. Fitting the decaying T1 curves leads to the relevant T1 value. This work compares here the three most commonly used fit models that are, single, bi, and stretched exponential. This work finally investigates the effect of the bootstrap itself on the precision of the result as well as the use of a rolling window to increase time resolution.

Multi-step histogram based outlier scores for unsupervised anomaly detection: ArcelorMittal engineering dataset case of study

Anomaly detection is the task of detecting samples that behave differently from the rest of the data or that include abnormal values. Unsupervised anomaly detection is the most common scenario, which implies that the algorithms cannot train with a labeled input and do not know the anomaly behavior beforehand. Histogram-based methods are one of the most approaches in unsupervised anomaly detection, remarking a good performance and a low runtime. Despite the good performance, histogram-based anomaly detectors are not capable of processing data flows while updating their knowledge and cannot deal with a high amount of samples.In this paper, we propose a new histogram-based approach for addressing the aforementioned problems by introducing the ability to update the information inside a histogram. We have applied these strategies to design a new algorithm called Multi-step Histogram Based Outlier Scores (MHBOS), including five new histogram update mechanisms. The results have shown the performance and validity of MHBOS as well as the proposed strategies in terms of performance and computing times.

BIM and IoT data fusion: The data process model perspective

Digitalization in the architecture, engineering, and construction (AEC) industry has highlighted the process of gathering and combining data from numerous sources. This paper uses data fusion from information science to investigate how data from information systems like Building Information Modeling (BIM) and the Internet of Things (IoT) could be coupled to enable a data-driven AEC. However, given the large amount of data, the technological diversity, the heterogeneous data schema, and hierarchical data abstraction, BIM and IoT data fusion are not addressed systematically. This study aims to develop a BIM and IoT data fusion framework to facilitate the appropriate data process flow for various applications. To investigate the research topic, an integrative review was conducted. Specifically, a general and tailored four-step data fusion process model was proposed as the review’s guiding framework. Critical evaluations were incorporated into a road map to provide a comprehensive perspective of BIM and IoT data fusion. In addition, based on the findings, a two-level and interconnected data fusion framework was proposed, which illustrates the data process flow and differentiates the data abstraction level, data processing technologies and purpose of the fusion. Furthermore, the fusion challenges and future trends are also highlighted.

RescueSNN: enabling reliable executions on spiking neural network accelerators under permanent faults.

To maximize the performance and energy efficiency of Spiking Neural Network (SNN) processing on resource-constrained embedded systems, specialized hardware accelerators/chips are employed. However, these SNN chips may suffer from permanent faults which can affect the functionality of weight memory and neuron behavior, thereby causing potentially significant accuracy degradation and system malfunctioning. Such permanent faults may come from manufacturing defects during the fabrication process, and/or from device/transistor damages (e.g., due to wear out) during the run-time operation. However, the impact of permanent faults in SNN chips and the respective mitigation techniques have not been thoroughly investigated yet. Toward this, we propose RescueSNN, a novel methodology to mitigate permanent faults in the compute engine of SNN chips without requiring additional retraining, thereby significantly cutting down the design time and retraining costs, while maintaining the throughput and quality. The key ideas of our RescueSNN methodology are (1) analyzing the characteristics of SNN under permanent faults; (2) leveraging this analysis to improve the SNN fault-tolerance through effective fault-aware mapping (FAM); and (3) devising lightweight hardware enhancements to support FAM. Our FAM technique leverages the fault map of SNN compute engine for (i) minimizing weight corruption when mapping weight bits on the faulty memory cells, and (ii) selectively employing faulty neurons that do not cause significant accuracy degradation to maintain accuracy and throughput, while considering the SNN operations and processing dataflow. The experimental results show that our RescueSNN improves accuracy by up to 80% while maintaining the throughput reduction below 25% in high fault rate (e.g., 0.5 of the potential fault locations), as compared to running SNNs on the faulty chip without mitigation. In this manner, the embedded systems that employ RescueSNN-enhanced chips can efficiently ensure reliable executions against permanent faults during their operational lifetime.