Big Data Analytics Platform Research Articles

Power consumption state evaluation of important power customers based on AHP‐TOPSIS algorithm

AbstractIn the construction of a new power system, the identification and evaluation of power consumption status of power customers will become an important basis for them to participate in the emerging businesses such as demand response and virtual power plants. In order to ensure the power safety of important power customers, a new evaluation of power consumption status of important power customers based on the AHP (analytic hierarchy process)‐TOPSIS (technique for order preference by similarity to an ideal solution) algorithm is proposed by fully mining and applying the power big data. Firstly, a power consumption big data analysis platform based on the Hadoop architecture is built to provide a high‐performance platform support for big data analysis. Secondly, nine evaluation indexes are constructed from the three dimensions of voltage, load and synthesis, which objectively and scientifically describes the power consumption status of important power customers. Finally, the AHP‐TOPSIS algorithm is used to evaluate and analyse the voltage, load and comprehensive indicators respectively, thus, obtaining the evaluation values of three kinds of indicators. The power consumption status scores of important power customers are determined by the variable weight weighted summation. The rationality and feasibility of the method and algorithm are proved by example analysis and field verification. This method helps to promote the transformation from post fault emergency repair to warning beforehand. It has the multiple effects of ensuring safe power consumption, supporting accurate patrolling and active emergency repair serving.

PIONEER big data platform for prostate cancer: lessons for advancing future real-world evidence research.

Prostate Cancer Diagnosis and Treatment Enhancement through the Power of Big Data in Europe (PIONEER) is a European network of excellence for big data in prostate cancer. PIONEER brings together 34 private and public stakeholders from 9 countries in one multidisciplinary research consortium with the aim of positively transforming the field of prostate cancer clinical care by answering pressing questions related to prostate cancer screening, diagnosis and treatment. PIONEER has developed a unique state-of-the-art big data analytic platform by integrating existing data sources from patients withprostate cancer. PIONEER leveraged this platform to address prioritized research questions, filling knowledge gaps in the characterization, management and core outcomes of prostate cancer across the different disease stages. The network has benefited from sustained patient and stakeholder involvement and engagement, but many challenges remain when using real-world data for big data projects. To continue to advance prostate cancer care, data need to be available, suitable methodologies should be selected and mechanisms for knowledge sharing must be in place. Now acting as the prostate cancer arm of the European Association of Urology's new endeavour, UroEvidenceHub, PIONEER maintains its goal of maximizing the potential of big data to improve prostate cancer care.

Enhancing the English natural language processing dictionary using natural language processing++

<p><span lang="EN-US">Every natural language-based project requires the use of an English dictionary. But the current English dictionaries are not updated as the English language is constantly evolving. The English dictionary used for natural language processing (NLP) projects needs to be enhanced by adding more words and phrases. This helps in improving the accuracy of NLP applications such as machine translation, performance of text analysis, recognition, and part of speech (POS) tagging. Several approaches are proposed in this direction, this paper develops and demonstrates enhancement of the English dictionary using a more versatile and robust programming language known as NLP++, a plugin to distributed big data analytics platforms such as HPCC systems. The unique features of NLP++ language is the enabler for realization of the proposed approach. This paper also discusses key NLP techniques, dictionary refinements analysis using NLP and NLP++. The results show that the proposed approach using NLP++ has significantly improved the accuracy and comprehensiveness of the English dictionary.</span></p>

Studying the Intelligent Power Distribution and Consumption of the Power Grid and its Cost Structure Using Big Data Analysis

Based on the analysis of big data, this paper studies the impact of user behavior response on the cost structure of the microgrid system in the power grid system. The article first conducts in-depth research on distributed power generation and energy storage systems, focusing on the principles and output characteristics of smart power distribution and utilization in power grids, researches smart power distribution and utilization systems in power grids, and makes a more comprehensive discussion of the current situation. Secondly, a big data analysis platform was built, and distributed storage and computing were studied. The platform was used to perform distributed storage and regulation of electricity consumption data, and the electricity consumption information data was divided into the important load, controllable load and transferable load, constructed a microgrid system model based on electricity consumption behavior response, and analyzed a calculation example. After that, a micro-grid system was simulated based on HOMER software, and the optimal capacity configuration of the system was performed. Under this configuration, the micro-grid system has the highest economic efficiency. At the same time, a demand-side load control system was built. Introduced distributed power as a controllable load, integrated new energy access and load control technology, coordinated the contradiction between the power grid and distributed power, and completed a cost-benefit analysis. Finally, for demand-side management electricity price response, peak-valley time-of-use price, the most important implementation method, according to the cost structure theory, the peak-valley period is divided according to the membership function in fuzzy mathematics, and the user's response model to peak-valley time-of-use price is established. The experiment uses the original data to simulate, find the user response model parameters based on the load transfer rate, and complete the comparative analysis of the effect under the peak-valley time-of-use electricity price, which is of great significance to the implementation and improvement of the peak-valley time-of-use electricity price project. The analysis results of the calculation examples show that the method constructed in this paper can effectively realize the power quality analysis of the distribution network in the big data environment. The research results provide technical support for the management of the rural grid voltage deviation of the power company, and lay the foundation for improving the safe operation and management of the power grid.

Human Resource Management in the Transformative Digital Era

Digitalization in human resource management helps companies to modernize HR functions and gives them a competitive edge. At the same time, it requires a change in work style and require changes in demand for HR competencies. Study this aims to introduce the phenomenon of digitalization in literature, explore its current benefits and key risks, and analyze the competencies and role of HR professionals. Innovative strategies that can be applied in human resource management (HRM) to enhance organizational performance in the digital era. With rapid changes in technology and the business environment, organizations need to develop innovative approaches in managing their human resources. This research method is a qualitative research method with a descriptive approach. This research operationally examines various sources of literature relevant to the context of the above research. The data taken, identified in the following order: (1) data collection (2) data sorting (3) data analysis (4) conclusion making. As for data analysis, there is a predetermined sequence in accordance with the empirical steps taken, namely as follows: (1) Examination of data (2) suspected data findings, (3) Data confirmation (4) Diagnosis, (5) Action. The results obtained show that HR professionals tend to be a little reluctant to adopt technology the. The results of this research confirm the importance of digitalization in recent years for human resources and increasing demand for digital skills. Every organization should develop its digital strategy to improve productivity. In the present scenario, DHRM is considered a more significant and growing research topic as well as an important topic for a business organization. With the use of DHR practice and through social media, the internet, AI, and other technology organization can maintain their performance and employees' quality standard for the smooth running of the organization. In exploring "Strategic Human Resource Planning in the Era of Digital Transformation" through the Systematic Literature Review method, this research reveals important findings. Technology integration, especially through artificial intelligence, big data analysis and technology platforms, is proven to be a deep foundation in responsive and adaptive HR planning in the era of digital transformation. The success of this planning also depends heavily on the involvement of key stakeholders, particularly upper management and business units, who contribute to the identification of skills relevant to technological change.

Local outlier factor for anomaly detection in HPCC systems

Local Outlier Factor (LOF) is an unsupervised anomaly detection algorithm that finds anomalies by assessing the local density of a data point relative to its neighborhood. Anomaly detection is the process of finding anomalies in datasets. Anomalies in real-time datasets may indicate critical events like bank frauds, data compromise, network threats, etc. This paper deals with the implementation of the LOF algorithm in the HPCC Systems platform, which is an open-source distributed computing platform for big data analytics. Improved LOF is also proposed which efficiently detects anomalies in datasets rich in duplicates. The impact of varying hyperparameters on the performance of LOF is examined in HPCC Systems. This paper examines the performance of LOF with other algorithms like COF, LoOP, and kNN over several datasets in the HPCC Systems. Additionally, the efficacy of LOF is evaluated across big-data frameworks such as Spark, Hadoop, and HPCC Systems, by comparing their runtime performances.

Big data processing and analysis platform based on deep neural network model

Users are increasingly turning to big data processing systems to extract valuable information from massive datasets as the field of big data grows. Data analytics platforms are used by e-commerce enterprises to improve product suggestions and model business processes. In order to meet the needs of large-scale data center operation and maintenance management, Internet companies often use Flink to process log data. This paper takes the big data processing and analysis platforms built by Internet financial companies and large banks as examples, and implants a stock prediction model based on Deep Neural Network (DNN). In this context, this paper completes the following work: 1) The research status of big data processing and analysis platforms at home and abroad is introduced. 2) Drawing on the modular design idea, the commercial bank big data platform is designed and the functions of each sub-module are introduced. Then the basic principle and structure of Convolutional Neural Networks (CNN) are expounded. 3) The optimal parameters of Convolutional Neural Networks are selected through experiments, and then the trained model is used for experiments. It can be seen that the stock prediction model proposed in this article has a higher prediction accuracy compared to existing models, which also verifies the validity of the proposed model. Input the data and compare the obtained results with the actual results, and finally show that the model in this paper has a good performance on stock prediction.

Data analytics for predicting disease outbreaks: A review of models and tools

The burgeoning field of data analytics has emerged as a pivotal force in the realm of public health, particularly in the context of predicting and mitigating disease outbreaks. This comprehensive review delves into the diverse landscape of models and tools employed in data analytics for disease outbreak prediction. With a focus on synthesizing existing knowledge, the paper aims to provide a nuanced understanding of the strengths, limitations, and future directions within this dynamic field. The review begins with an exploration of various models utilized for disease outbreak prediction, ranging from statistical approaches to machine learning models and epidemiological frameworks. Each model category is scrutinized for its efficacy in capturing the complexities inherent in infectious disease dynamics. Simultaneously, the paper investigates the array of tools and technologies leveraged in disease outbreak prediction, encompassing Geographic Information Systems (GIS), data visualization tools, and big data analytics platforms. A critical aspect of the review lies in the examination of diverse data sources contributing to predictive analytics. Epidemiological data, environmental factors, and the burgeoning influence of social media and web data are dissected for their roles in enhancing the accuracy and timeliness of outbreak predictions. Amidst the promises of data analytics, the paper navigates the challenges inherent in predicting disease outbreaks. Issues of data quality and availability, model complexity, interpretability, and ethical considerations are dissected, providing a holistic view of the hurdles that practitioners encounter. Drawing upon case studies and real-world applications, the review showcases instances where data analytics has proven successful in predicting disease outbreaks, shedding light on both triumphs and setbacks. The implications for public health, lessons learned from challenges, and the evolving role of data analytics in shaping global health preparedness are thoroughly discussed. As the paper unfolds, it illuminates future trends and innovations in the field, foreseeing the integration of advanced technologies, global collaboration for information sharing, and the adaptation of predictive analytics for emerging diseases. The review culminates in a comprehensive conclusion, summarizing key findings and emphasizing the potential transformative impact of data analytics on the landscape of disease outbreak prediction.

텍스트마이닝 기반 데이터분석을 통한 전문대학 교양교과에 대한 사회적 담론과 현황이해

Objectives The purposes of this study, we seek to gain a macroscopic understanding of the social discourse and current status of liberal arts subjects in colleges in order to improve the quality and strengthen the liberal arts education in colleges.
 Methods To this end, ‘Big data’ was collected using the keywords of ‘junior college+liberal arts’ subjects appearing on the portal site, and data analysis was conducted using the liberal arts subject schedule of junior colleges in the Chungcheong and Gangwon regions as ‘retained data’. Keyword analysis, TF-IDF weight analysis, and network centrality analysis were performed in the big data analysis platform(TEXTOM), and CONCOR analysis was performed using the UCINET6.0 program.
 Results First, as a result of keyword analysis, keywords such as credits, graduation, and completion appeared in portal-collected big data, and in the case of retained data, keywords such as life, English, ability, and understanding were mainly found. Second, keywords with high TF-IDF values a ppeared in the order of ‘credits’, ‘bachelor’s degree’, ‘degree’, and ‘graduation’ in the case of portal-collected big data, and there appeared to be no difference in the case of retained data. Third, as a result of keyword centrality analysis, it was found that in the case of portal big data, ‘credit’ was out of the ranking in closeness centrality, and in the case of retained data, ‘English’ was found to be out of the ranking. Fourth, the clusters formed through CONCOR analysis were found to form two clusters for portal big data and four clusters for retained data.
 Conclusions Big data on liberal arts subjects at colleges showed that social discourse is being formed at a very formal level, such as basic materials related to bachelor's degrees, and the data shows that the nature of liberal arts subjects based on vocational fundamentals is still strong in the composition of subjects.

SHERPA-Cloud: An open-source online model to simulate air quality management policies in Europe

In this manuscript we present the SHERPA-Cloud online simplified air quality model, which is an evolution of the previous offline SHERPA model. SHERPA-cloud is a meta-model of a full Chemical Transport Model, and can be used to screen how policies would impact air quality. The main advantages of SHERPA-cloud (in comparison to the previous model version) are its speed (5 time faster than the offline version), its accessibility (permanent online availability) and its easier and more cost-effective maintenance (automatic updates can be shared more easily, and tool developments are facilitated). From the IT infrastructure and technologies points of view, the SHERPA-Cloud is based on an in-house cloud system called BDAP (Big Data Analytic Platform) which provides a ’JupyterLab’ service and tools to create dashboards using ’Voilà’ technology. SHERPA-Cloud is a useful easy-to-use tool in the hands of policy makers, citizens, NGOs and industries, that allows for interactive and participative policy developments. Indeed, it allows to quickly quantify changes in air quality resulting from the implementation of emission reduction strategies, in terms of both geographical or sectoral policies.

Construction of Big Data Analysis Platform for College Students’ Sports Training Driven by Wireless Communication Network

College students’ physical training is an important aspect of their studies and lives. In light of the existing issues with the accuracy of the physical training evaluation index for college students and other issues. Based on the theory of wireless communication networks, model definition and stability analysis methods are used to optimize the original model. The optimization model of a wireless communication network is obtained by considering the data output control theory, trained by college students. This model can analyze the big data of college students’ physical training and construct the corresponding data platform. Through calculations, the rules for data changes under different indicators can be obtained. Finally, the accuracy of the optimized model is verified by comparing it with the original model. Related studies show that the variable parameters in the control system include model parameters, network upper limit, and corresponding control system parameters. On the whole, the model parameters exhibit a consistent trend of change, while the upper limit of the network displays a characteristic three-stage linear pattern. The control system first increases rapidly and then exhibits a gradual decline in exponential function type. The control system and the switching system can be switched using model calculations. The two different change curves exhibit obvious symmetry characteristics when influenced by the output times. It shows that the two change systems have opposite effects on the model, and the overall volatility is relatively high. The wireless communication network model can analyze and build a platform for collecting college students’ sports training data. According to the model calculation, it can be seen that the changes in different indexes exhibit clear linear characteristics. Among them, the concepts of sports, personalized training, and basic training show linearly increasing changes. However, the training structure and content exhibit a typical linear decline. Finally, the advantages of the wireless communication network model are illustrated with the experimental data. This study can provide guidance and ideas for the construction of a data platform for college students’ physical training.

Analysis and Application of Big Data Feature Extraction Based on Improved k-means Algorithm

This paper addresses the challenges modelled by collecting and storing large volumes of big data, focusing on mitigating data errors. The primary goal is to propose and evaluate an enhanced K-means algorithm for big data applications. This research also aims to design an extensive energy data system to demonstrate the improved algorithm's practical utility in monitoring power equipment. The research begins with an in-depth analysis of the traditional K-means algorithm, culminating in the proposal of an improved version. Subsequently, the study outlines developing a comprehensive, extensive energy data system, encompassing architectural aspects such as data storage, mechanical layers, and data access structures. The research also involves the development of a power big data analysis platform, incorporating the improved algorithm for clustering and analyzing power equipment monitoring data. Experimental results reveal that the proposed improved K-means algorithm outperforms the traditional version, with significantly improved accuracy and reduced classification errors, achieving an error rate of less than one. The improved K-means algorithm showcased remarkable enhancements, achieving a meagre misclassification rate of just 0.08% while substantially boosting accuracy levels, consistently exceeding 95% across all datasets. Moreover, the power big data system developed in this study to meet practical requirements while enhancing storage and processing efficiency effectively.

Instance segmentation on distributed deep learning big data cluster

AbstractDistributed deep learning is a promising approach for training and deploying large and complex deep learning models. This paper presents a comprehensive workflow for deploying and optimizing the YOLACT instance segmentation model as on big data clusters. OpenVINO, a toolkit known for its high-speed data processing and ability to optimize deep learning models for deployment on a variety of devices, was used to optimize the YOLACT model. The model is then run on a big data cluster using BigDL, a distributed deep learning library for Apache Spark. BigDL provides a high-level programming interface for defining and training deep neural networks, making it suitable for large-scale deep learning applications. In distributed deep learning, input data is divided and distributed across multiple machines for parallel processing. This approach offers several advantages, including the ability to handle very large data that can be stored in a distributed manner, scalability to decrease processing time by increasing the number of workers, and fault tolerance. The proposed workflow was evaluated on virtual machines and Azure Databricks, a cloud-based platform for big data analytics. The results indicated that the workflow can scale to large datasets and deliver high performance on Azure Databricks. This study explores the benefits and challenges of using distributed deep learning on big data clusters for instance segmentation. Popular distributed deep learning frameworks are discussed, and BigDL is chosen. Overall, this study highlights the practicality of distributed deep learning for deploying and scaling sophisticated deep learning models on big data clusters.

Selecting the foremost big data tool to optimize YouTube data in dynamic Fermatean fuzzy knowledge.

Big data pertains to extensive and intricate compilations of information that necessitate the implementation of proficient and cost-effective evaluation and analysis tools to derive insights and support decision making. The Fermatean fuzzy set theory possesses remarkable capability in capturing imprecision due to its capacity to accommodate complex and ambiguous problem descriptions. This paper presents the study of the concepts of dynamic ordered weighted aggregation operators in the context of Fermatean fuzzy environment. In numerous practical decision making scenarios, the term "dynamic" frequently denotes the capability of obtaining decision-relevant data at various time intervals. In this study, we introduce two novel aggregation operators: Fermatean fuzzy dynamic ordered weighted averaging and geometric operators. We investigate the attributes of these operators in detail, offering a comprehensive description of their salient features. We present a step-by-step mathematical algorithm for decision making scenarios in the context of proposed methodologies. In addition, we highlight the significance of these approaches by presenting the solution to the decision making problem and determining the most effective big data analytics platform for YouTube data analysis. Finally, we perform a thorough comparative analysis to assess the effectiveness of the suggested approaches in comparison to a variety of existing techniques.

Design of Accounting Information Big Data Analysis Platform Based on Cloud Computing

Design of Accounting Information Big Data Analysis Platform Based on Cloud Computing

Healthcare Big Data Analysis with Artificial Neural Network for Cardiac Disease Prediction

The generation of a huge volume of structured, semi-structured and unstructured real-time health monitoring data and its storage in the form of electronic health records (EHRs) need to be processed and analyzed intelligently to provide timely healthcare. A big data analytic platform is an alternative to the traditional warehouse paradigms for the processing, analysis and storage of the tremendous volume of healthcare data. However, the manual analysis of these voluminous, multi-variate patients data is tedious and error-prone. Therefore, an intelligent solution method is highly essential to perform multiple correlation analyses for disease diagnosis and prediction. In this paper, first, a structural framework is proposed to process the huge volume of cardiological big data generated from the hospital and patients. Then, an intelligent analytical model for the cardiological big data analysis is proposed by combining the concept of artificial neural network (ANN) and particle swarm optimization (PSO) to predict the abnormalities in the cardiac health of a person. In the proposed cardiac disease prediction model, an extensive electrocardiogram (ECG) data analysis method is developed to identify the probable normal and abnormal cardiac feature points. Simulation results show the effects of a number of attributes for improving the accuracy of the cardiac disease prediction and data processing time in the cloud with an increase in the number of the cardiac patients.

An Improved Big Data Analytics Architecture for Intruder Classification Using Machine Learning

The approval of retrieving information on the Internet originates several network securities matters. Intrusion recognition is a critical study in network security to spot unauthorized admission or occurrences on protected networks. Intrusion detection has a fully-fledged reputation in the current era. Research emphasizes several datasets to upsurge system precision and lessen the false-positive proportion. This article proposes a new intrusion detection system using big data analytics and deep learning to address some of the misuse and irregularity detection limitations. The proposed method could identify any odd activities in a network to recognize malicious or unauthorized action and permit a response during a confidentiality break. The proposed system utilizes the big data analytics platform based on parallel and distributed mechanisms. The parallel and distributed platforms improve the training time along with the accuracy. The experimentation appropriately classifies the information as either normal or abnormal. The proposed system has a recognition proportion of 96.11% that pointedly expands overall recognition accuracy related to existing strategies.

Research on the Application of Big Data Analysis Platform in the Enterprise Management Optimization

As companies attempt to make analysis an important data support component of daily decision-making, big data analysis technology is rapidly expanding across all industries. Although there are many software tools and libraries available to assist analysts and software engineers in developing solutions, enterprises are looking for reliable analysis platforms that can meet their specific goals and requirements. In order to minimize costs, such platforms also need to coexist with existing IT infrastructure and reuse the knowledge and resources already accumulated within the organization. To meet these requirements, this article proposes the Data Analysis Solution Engineering (DASE) framework - a knowledge driven approach supported by semantic web technology, for the design and development of requirements engineering and new data analysis platforms. It includes capturing data analysis platform requirements through a knowledge base, and enterprises learning how to use this data analysis platform to analyze all daily production data involved in the engineering data analysis platform. This article analyzes the DASE framework through knowledge modeling, requirement modeling, data architecture modeling, and platform design modeling, and demonstrates how it promotes knowledge and requirement driven data analysis platform engineering. The resulting data analysis platform is considered user-friendly, easy to maintain, and flexible in handling changes in requirements. This work contributes to the knowledge system of knowledge driven requirements engineering and data analysis platform engineering by providing customized models and reference architectures for different analytical application fields.

Optimized Feature Learning for Anti-Inflammatory Peptide Prediction Using Parallel Distributed Computing

With recent advancements in computational biology, high throughput Next-Generation Sequencing (NGS) has become a de facto standard technology for gene expression studies, including DNAs, RNAs, and proteins; however, it generates several millions of sequences in a single run. Moreover, the raw sequencing datasets are increasing exponentially, doubling in size every 18 months, leading to a big data issue in computational biology. Moreover, inflammatory illnesses and boosting immune function have recently attracted a lot of attention, yet accurate recognition of Anti-Inflammatory Peptides (AIPs) through a biological process is time-consuming as therapeutic agents for inflammatory-related diseases. Similarly, precise classification of these AIPs is challenging for traditional technology and conventional machine learning algorithms. Parallel and distributed computing models and deep neural networks have become major computing platforms for big data analytics now required in computational biology. This study proposes an efficient high-throughput anti-inflammatory peptide predictor based on a parallel deep neural network model. The model performance is extensively evaluated regarding performance measurement parameters such as accuracy, efficiency, scalability, and speedup in sequential and distributed environments. The encoding sequence data were balanced using the SMOTETomek approach, resulting in a high-accuracy performance. The parallel deep neural network demonstrated high speed up and scalability compared to other traditional classification algorithms study’s outcome could promote a parallel-based model for predicting anti-Inflammatory Peptides.

Big data methods and news communication innovation from theoretical definition to operational route

Abstract To make news communication better adapt to the era of big data, it is necessary to innovate news communication. Accordingly, this paper constructs a big data analysis platform for news communication innovation methods based on the regression analysis model in big data analysis, determines the parameters and coefficients of the regression model with the help of the maximum likelihood estimation method, and determines the form of the regression model. Based on the platform, the main paths of news communication innovation are analyzed, standardized regression coefficients are calculated, and the main methods of news communication innovation are changing news communication forms and news communication media. To verify that the above methods can make news communication innovative, we designed a visualization information platform to change the news communication format and launched a simulation experiment to analyze the effect of news communication under different media. The results showed that after the visualization of the news communication format, the number of viewers of news programs increased from 2.45 million to 3.65 million, an increase of 1.2 million. The conversion rate of audience behavior of news programs increased from the original 63% to 87%, an increase of 24%. From the above results, it is clear that news communication under the big data approach has improved from a theoretical definition to an operational route.