High-performance Data Analysis Research Articles

Industrial mechanical equipment fault detection and high-performance data analysis technology based on the Internet of Things

In view of the problems of low detection accuracy, long detection time, and inability to monitor fault data in real time in the fault detection of traditional machinery and equipment, this paper studies the identification and fault detection of industrial machinery based on the Internet of Things (IoT) technology. By using Internet of Things technology to build a mechanical equipment fault detection system, Internet of Things technology can better build diagnostic and early warning modules for the system, so as to achieve the goal of improving the accuracy of equipment fault detection, shortening equipment fault detection time, and remotely monitoring equipment. The fault detection system studied in this paper has an accuracy rate of more than 93.4% to detect different types of fault. The use of Internet of Things technology is conducive to improving the accuracy of mechanical equipment fault detection and realizing real-time monitoring of equipment data.

Immersive and Interactive 3D Visualization of Large-Scale Geoscientific Data

Abstract Virtual reality (VR) technology has great potential in supporting planetary scientists and the field of geosciences by offering immersive and interactive experiences for data exploration and analysis. This paper shows the opportunities presented by the use of VR technology in the geosciences domain while also identifying the open challenges associated with this integration. Our focus is on highlighting the several benefits that VR brings, including stereo vision, head tracking, and collaborative capabilities. These features can foster knowledge exchange and interdisciplinary research. Nevertheless, the adoption of VR presents certain challenges that need to be addressed. These include maintaining high refresh rates, handling large heterogeneous datasets, and striking a balance between visualization fidelity and performance. Fortunately, significant advancements have been made in high-performance data analysis, progressive data streaming, and real-time visualization, enabling interactive exploration of large-scale datasets within VR environments. To support a broader adoption among domain experts, we propose a visualization approach that scales both with display and with data size. As such, the system can be used to interactively explore large-scale datasets in immersive environments like CAVEs or powerwalls, on HMDs, or on traditional desktop setups. This approach allows for seamless transitions between desktop and VR experiences, leveraging immersive environments for collaboration and outreach activities. In this paper, we present an architectural framework, hardware environments, use cases, lessons learned, and the future potential of VR in this field. Through this scalable system, we anticipate transformative advancements in scientific exploration, collaboration, and knowledge dissemination.

Development and Design of an Intelligent Financial Asset Management System Based on Big Data Analysis and Kubernetes

The rise of deep learning in the financial field has led to the integration of artificial intelligence and investment, providing users with intelligent investment decisions. However, the data volume of financial market continues to expand, traditional data processing methods can no longer meet the needs of efficiency and accuracy. This article focuses on deep reinforcement learning algorithms and delves into key issues such as stock price prediction, investment portfolios, and algorithmic trading. By comparing and analyzing the experimental results, not only was the performance of the model evaluated, but also the actual effect of the algorithm output was deeply explored. At the same time, drawing on Kubernetes container orchestration and microservice technology, a high concurrency and high-performance distributed financial data analysis system was constructed. This system not only meets the needs of users for real-time data analysis and deep learning, but also provides more reasonable investment suggestions for users. The contribution of this article lies in introducing deep reinforcement learning to solve nonlinear data problems in the financial field, proposing intelligent asset management methods, and designing a feasible intelligent financial asset management system, providing new ideas and practical experience for the further development of financial data analysis platforms.

Institutional Repository Keyword Analysis with Web Crawler

This study aims at investigating procedures of semantic and linguistic extraction of keywords from metadata of documents indexed in the Institutional Repository Unesp. For that purpose, a web crawler was developed, that collected 325.181 keywords from authors, in all fields of knowledge, from February 28th, 2013 to November 10th, 2021. The preparation of the collection, extraction and analysis environment used the Python programming language, composed of three program libraries: library requests, which allows manipulation of hyperlinks of webpages visited through web crawler; BeautifulSoup library, used to extract HTML data through webpage analysis; and Pandas library, which has an open code (free software) and stands for providing tools for high performance data manipulation and analysis. The final listing consisted of 273,485 keywords, which represents 15.9% of the listing initially collected. Results indicated that the most recurring problem was the duplication of keywords, with 51,696 duplicated keywords, representing indicators of inconsistencies in the search for documents. It is concluded that the refinement of keywords assigned by authors eliminates the incorporation of a set of symbols that do not represent the authors’ keywords with the same spelling, but with upper/lower case variations or lexical variations indexing different documents.

Seasonal variation, sources, and risk assessment of PM2.5-bound PAHs in Nantong, China: A pre- and post-COVID-19 case study

Between 2018 and 2022, a total of 231 PM2.5 samples were collected in Nantong, China, and analyzed online for 16 US EPA PAHs utilizing a variety of analytical methodologies, including high-performance liquid chromatography data analysis, principal component analysis, and characteristic ratio analysis. The seasonal variation and pollution characteristics of 16 PAHs in PM2.5 over a long period in Nantong, China, were analyzed, and correlations and traceability changes with a variety of emission sources were established. Additionally, health assessment models for BaP equivalent concentration, daily exposure dose, lifetime excess risk of cancer, and life expectancy loss were constructed for various ages and genders. The results indicated that the total number of days with BaP concentrations less than the national limit standard of 1.0 ng/m3 was 60.17% (139/231), with rates of 25.40%, 35.71%, and 94.05% from 2019 to 2021, respectively; the total concentration range of 16 PAHs demonstrated a downward trend, but pollution was relatively severe in winter. The seasonal variation showed winter > autumn > spring > summer. The detection rates of 4 and 5 rings were generally high and fluctuated between dropping and increasing. Principal factor analysis and characteristic ratio traceability analysis indicated that PAH pollution is primarily caused by local coal-fired sources and traffic emissions, with a new trend of biomass combustion. Furthermore, BaP remains the primary contributor to carcinogenic factors, and the health risk is higher in females than in males, and in adults than in children. A low level of PAHs in the air may have health benefits. Traffic control and flow restrictions, as well as production restrictions, are all part of Nantong’s COVID-19 pandemic prevention and control policies. These policies contributed to PAH risk prevention and control, as well as pollution reduction.

BrowVis: Visualizing Large Graphs in the Browser

A recent stream of research focuses on building high-performance data analysis and management systems that run completely in the browser. Indeed, today personal devices offer non-trivial amount of computing power, while the latest Web browsers provide powerful JavaScript engines. On the other hand, the use of visualization to present and analyze networks is taking a leading role in conveying information and knowledge to users that operate in multiple domains. In this scenario, the aim of our research is to explore the scalability limits of a system that executes the full graph visualization pipeline entirely in the browser. In this paper, we present BrowVis, a self-contained system to compute interactive visualizations of large graphs in the browser. Experiments show that, on a common laptop, BrowVis can visualize graphs with thousands of elements in seconds, as well as graphs with millions of elements in minutes. Once the initial visualization has been computed, BrowVis makes it possible to interactively explore the represented graph by following a details-on-demand paradigm. The use of BrowVis in practice is demonstrated by a case study dealing with a real-world scientific collaboration network.

Testing and Evaluation of the Electric Drive System on the Vehicle Level

In order to obtain the performance of the electric drive system (EDS) on the vehicle level correctly and effectively, the authors carried out research on the testing and evaluation technology. Firstly, the typical control strategy and its influence and limitations regarding the EDS performance were discussed in detail. Secondly, a test system on four-wheel dynamometers with a high-performance data acquisition and analysis system was introduced. A dedicated test was performed. Further analysis of the results was introduced. It is proved that the method introduced here is feasible and effective, which is beneficial to benchmarking and evaluation of the EDS used in electric vehicles.

PyGNA: a unified framework for geneset network analysis

BackgroundGene and protein interaction experiments provide unique opportunities to study the molecular wiring of a cell. Integrating high-throughput functional genomics data with this information can help identifying networks associated with complex diseases and phenotypes.ResultsHere we introduce an integrated statistical framework to test network properties of single and multiple genesets under different interaction models. We implemented this framework as an open-source software, called Python Geneset Network Analysis (PyGNA). Our software is designed for easy integration into existing analysis pipelines and to generate high quality figures and reports. We also developed PyGNA to take advantage of multi-core systems to generate calibrated null distributions on large datasets. We then present the results of extensive benchmarking of the tests implemented in PyGNA and a use case inspired by RNA sequencing data analysis, showing how PyGNA can be easily integrated to study biological networks. PyGNA is available at http://github.com/stracquadaniolab/pygna and can be easily installed using the PyPi or Anaconda package managers, and Docker.ConclusionsWe present a tool for network-aware geneset analysis. PyGNA can either be readily used and easily integrated into existing high-performance data analysis pipelines or as a Python package to implement new tests and analyses. With the increasing availability of population-scale omic data, PyGNA provides a viable approach for large scale geneset network analysis.

Improved Norfloxacin degradation by urea precipitation Ti/SnO2–Sb anode under photo-electro catalysis and kinetics investigation by BP-neural-network-physical modeling

The photo-electro catalysis has emerged as efficient and sustainable degradation method for antibiotics, where metal oxide anode plays a critical role. Exploring novel preparation method for anodes catalysis to achieve larger active sites and diverse oxidants production can directly enhance the degradation performance. Therefore, in this work, a new urea precipitation method for anode formation was studied. Both novel urea precipitation Ti/SnO2–Sb anode (TSSA-U) and traditional electro-deposition Ti/SnO2–Sb anode (TSSA-E) were prepared for Norfloxacin degradation by photo-electro catalysis in saline water. A unique tubular Ti/SnO2–Sb units formatted by urea precipitation resulted in higher porosity. This led to dominating advantage for TSSA-U on photo/electrochemical activity and degradation performances by individual photo/electro catalysis, compared to TSSA-E. However, this gap of degradation performances shrank when synergistic effect of photo-electro catalysis involved. The highest TOC removal ratio of 91.1% was obtained from TSSA-U under photo-electro catalysis. Besides, a novel BP-neural-network-physical modeling (BP-ANN-P) was developed for analysis. According to this modeling, both direct (adsorption-degradation, radiation, etc.) and indirect (mainly •Cl) routes contributed significantly in degradation work by TSSA-U, where indirect route shared 41.8–90.1% of total degradation ratio. An increasing of current density (from 5 to 25 mA cm−2) enhanced the kinetics for both routes. Indirect route preferred pH = 3–7 with direct route enhanced by neutral condition. Moreover, indirect route also showed better adaptability with higher initial NOR loadings.Both urea precipitation and BP-ANN-P have shown their value for high performance material formation and data analysis, respectively. Notably, higher catalysis performance and better analysis connect to low energy cost, shorter running time and more efficient judgement and selection, which meets the requirement of cleaner production and environmental sustainability.

Machine learning driven smart electric power systems: Current trends and new perspectives

The current power systems are undergoing a rapid transition towards their more active, flexible, and intelligent counterpart smart grid, which brings about tremendous challenges in many domains, e.g., integration of various distributed renewable energy sources, cyberspace security, demand-side management, and decision-making of system planning and operation. The fulfillment of advanced functionalities in the smart grid firmly relies on the underlying information and communication infrastructure, and the efficient handling of a massive amount of data generated from various sources, e.g., smart meters, phasor measurement units, and various forms of sensors. In this paper, a comprehensive survey of over 200 recent publications is conducted to review the state-of-the-art practices and proposals of machine learning techniques and discuss the trend in a wide range of smart grid application domains. This study demonstrates the increasing interest and rapid expansion in the use of machine learning techniques to successfully address the technical challenges of the smart grid from various aspects. It is also revealed that some issues still remain open and worth further research efforts, such as the high-performance data processing and analysis for intelligent decision-making in large-scale complex multi-energy systems, lightweight machine learning-based solutions, and so forth. Moreover, the future perspectives of utilizing advanced computing and communication technologies, e.g., edge computing, ubiquitous internet of things and 5G wireless networks, in the smart grid are also highlighted. To the best of our knowledge, this is the first review of machine learning-driven solutions covering almost all the smart grid application domains. Machine learning will be one of the major drivers of future smart electric power systems, and this study can provide a preliminary foundation for further exploration and development of related knowledge and insights.

Software Platform for European XFEL: Towards Online Experimental Data Analysis

Large amount of data being generated at large scale facilities like European X-ray Free- Electron Laser (XFEL) requires new approaches for data processing and analysis. One of the most computationally challenging experiments at an XFEL is single-particle structure determination. In this paper we propose a new design for an integrated software platform which combines well-established techniques for XFEL data analysis with High Performance Data Analysis (HPDA) methods. In our software platform we use streaming data analysis algorithms with high performance computing solutions. This approach should allow analysis of the experimental dataflow in quasi-online regime.

Framework of Intelligent Analysis and Mining for Power Big Data

With the continuous development of smart grid, how to make full use of and analyze these data and get valuable information quickly is the key points which need to be solve in the big data processing. This paper constructs the framework of intelligent and efficient mining for power big data, analysis technology, high performance data analysis technology and the parallel analysis framework for power big data. Several service system described in detail the key techniques of intelligent analysis of large data mining and electricity from the power of big data characteristics.

Towards on-the-fly Cryo-Electron Microscopy Data Processing by High Performance Data Analysis

In this paper we present an approach to the on-the-fly Cryo-EM data acquisition and analysis using High Performance Data Analysis concept. An analysis of bottlenecks in the currently used data processing is presented, and the new iterative scheme for near real time implementation of the Cryo-EM data processing is discussed. The on-the-fly approach is proposed that will considerably reduce the time required for the protein (or other biological objects, e.g. viruses) 3D structure determination and allow eliminating manual interventions in the data processing.

A cloud/edge computing streaming system for network traffic monitoring and threat detection

The unyielding trend of increasing cyber threats has made cyber security paramount in protecting personal and private intellectual property. To provide a highly secured network environment, network threat detection systems must handle real-time big data from varied places in enterprise networks. In this paper, we introduce a streaming-based threat detection system that can rapidly analyse highly intensive network traffic data in real-time, utilising streaming-based clustering algorithms to detect abnormal network activities. The developed system integrates the high-performance data analysis capabilities of Flume, Spark and Hadoop into a cloud-computing environment to provide network monitoring and intrusion detection. Our performance evaluation validates that the developed system can cope with a significant volume of streaming data in a high detection accuracy and good system performance. We further extend our system for edge computing and discuss some key challenges, as well as some potential solutions, aiming to improve the scalability of our system.

A Data Quality Strategy to Enable FAIR, Programmatic Access across Large, Diverse Data Collections for High Performance Data Analysis

To ensure seamless, programmatic access to data for High Performance Computing (HPC) and analysis across multiple research domains, it is vital to have a methodology for standardization of both data and services. At the Australian National Computational Infrastructure (NCI) we have developed a Data Quality Strategy (DQS) that currently provides processes for: (1) Consistency of data structures needed for a High Performance Data (HPD) platform; (2) Quality Control (QC) through compliance with recognized community standards; (3) Benchmarking cases of operational performance tests; and (4) Quality Assurance (QA) of data through demonstrated functionality and performance across common platforms, tools and services. By implementing the NCI DQS, we have seen progressive improvement in the quality and usefulness of the datasets across the different subject domains, and demonstrated the ease by which modern programmatic methods can be used to access the data, either in situ or via web services, and for uses ranging from traditional analysis methods through to emerging machine learning techniques. To help increase data re-usability by broader communities, particularly in high performance environments, the DQS is also used to identify the need for any extensions to the relevant international standards for interoperability and/or programmatic access.

A Data Analysis Framework for Earth System Simulation within an <i>In-Situ</i> Infrastructure

This paper presents a generic procedure to implement a scalable and high performance data analysis framework for large-scale scientific simulation within an in-situ infrastructure. It demonstrates a unique capability for global Earth system simulations using advanced computing technologies (i.e., automated code analysis and instrumentation), in-situ infrastructure (i.e., ADIOS) and big data analysis engines (i.e., SciKit-learn). This paper also includes a useful case that analyzes a globe Earth System simulations with the integration of scalable in-situ infrastructure and advanced data processing package. The in-situ data analysis framework can provides new insights on scientific discoveries in multiscale modeling paradigms.

High Performance Data Analysis via Coordinated Caches

With the second run period of the LHC, high energy physics collaborations will have to face increasing computing infrastructural needs. Opportunistic resources are expected to absorb many computationally expensive tasks, such as Monte Carlo event simulation. This leaves dedicated HEP infrastructure with an increased load of analysis tasks that in turn will need to process an increased volume of data. In addition to storage capacities, a key factor for future computing infrastructure is therefore input bandwidth available per core. Modern data analysis infrastructure relies on one of two paradigms: data is kept on dedicated storage and accessed via network or distributed over all compute nodes and accessed locally. Dedicated storage allows data volume to grow independently of processing capacities, whereas local access allows processing capacities to scale linearly. However, with the growing data volume and processing requirements, HEP will require both of these features. For enabling adequate user analyses in the future, the KIT CMS group is merging both paradigms: popular data is spread over a local disk layer on compute nodes, while any data is available from an arbitrarily sized background storage. This concept is implemented as a pool of distributed caches, which are loosely coordinated by a central service. A Tier 3 prototype cluster is currently being set up for performant user analyses of both local and remote data.

Tier 3 batch system data locality via managed caches

Modern data processing increasingly relies on data locality for performance and scalability, whereas the common HEP approaches aim for uniform resource pools with minimal locality, recently even across site boundaries. To combine advantages of both, the High- Performance Data Analysis (HPDA) Tier 3 concept opportunistically establishes data locality via coordinated caches.In accordance with HEP Tier 3 activities, the design incorporates two major assumptions: First, only a fraction of data is accessed regularly and thus the deciding factor for overall throughput. Second, data access may fallback to non-local, making permanent local data availability an inefficient resource usage strategy. Based on this, the HPDA design generically extends available storage hierarchies into the batch system. Using the batch system itself for scheduling file locality, an array of independent caches on the worker nodes is dynamically populated with high-profile data. Cache state information is exposed to the batch system both for managing caches and scheduling jobs. As a result, users directly work with a regular, adequately sized storage system. However, their automated batch processes are presented with local replications of data whenever possible.

OpenMSI: A High-Performance Web-Based Platform for Mass Spectrometry Imaging

Mass spectrometry imaging (MSI) enables researchers to directly probe endogenous molecules directly within the architecture of the biological matrix. Unfortunately, efficient access, management, and analysis of the data generated by MSI approaches remain major challenges to this rapidly developing field. Despite the availability of numerous dedicated file formats and software packages, it is a widely held viewpoint that the biggest challenge is simply opening, sharing, and analyzing a file without loss of information. Here we present OpenMSI, a software framework and platform that addresses these challenges via an advanced, high-performance, extensible file format and Web API for remote data access (http://openmsi.nersc.gov). The OpenMSI file format supports storage of raw MSI data, metadata, and derived analyses in a single, self-describing format based on HDF5 and is supported by a large range of analysis software (e.g., Matlab and R) and programming languages (e.g., C++, Fortran, and Python). Careful optimization of the storage layout of MSI data sets using chunking, compression, and data replication accelerates common, selective data access operations while minimizing data storage requirements and are critical enablers of rapid data I/O. The OpenMSI file format has shown to provide >2000-fold improvement for image access operations, enabling spectrum and image retrieval in less than 0.3 s across the Internet even for 50 GB MSI data sets. To make remote high-performance compute resources accessible for analysis and to facilitate data sharing and collaboration, we describe an easy-to-use yet powerful Web API, enabling fast and convenient access to MSI data, metadata, and derived analysis results stored remotely to facilitate high-performance data analysis and enable implementation of Web based data sharing, visualization, and analysis.

A computational framework for spatially explicit agroecosystem modeling: Application to regional simulation

Site-based agroecosystem model has been applied at regional and state level to enable comprehensive analyses of environmental sustainability of food and biofuel production. However, spatially explicit ecosystem simulations over large landscape present computational challenges. This paper presents a framework to support spatially explicit agroecosystem modeling and data analysis over large landscape, which includes four major phases of agroecosystem simulation: simulation data preparation, site-based simulation on high performance computers, data management and data analysis. Then, a case study on a regional intensive modeling area (RIMA) was presented as an application to demonstrate the system implementation and capability.