Big Data Mining Technology Research Articles

Quality consistency analysis for complex assembly process based on Bayesian networks

In recent years, assembly process has been rapidly pushing the envelope in building complex, optimized products, by taking advantage of big data mining and machine intelligence technologies. Along with such developments, it has become increasingly necessary to strictly control quality consistency level of the assembly activities. However, due to the difficulty of modelling, quality consistency improvement of complex assembly process has always been a challenge for academia and industry. This paper attempts to describe the assembly process by using Bayesian networks and to provide an effective data-based scheme for improving quality consistency. First of all, considering the nonlinear and coupling characteristics, we introduce the maximum information coefficient and network deconvolution method to detect the direct associations in the assembly process and get an undirected graph. Secondly, in order to obtain a complete Bayesian network, we derive the directions of edges based on the independence between causal variable distribution and function mechanism. Thus, the key influence factors on quality consistency and the latent causal mechanism can be analyzed. This data-based scheme is applied to power consistency analysis of a real diesel engine production line, where the effectiveness is further demonstrated from real industrial data. As a result, this study provides theoretical support and technical guarantee for the improvement of assembly process, quality control, product quality and enterprise benefit.

Optimization of Air-Coal Distribution System for 1000MW Thermal Power Unit Based on Data Mining

At present, the operation adjustment of the power distribution system of the power station boiler mainly relies on the experience of the operators and the optimization and adjustment test, neither of them can be economically and flexibly adapted to the current situation of complex boiler combustion conditions and strict environmental protection indicators. To reduce the NOx generated by pulverized coal combustion in the furnace, this paper effectively combines the big data mining technology with the boiler combustion system based on the best ratio of coal-air. Taking the double-cut circular boiler designed by a 1000MW thermal power unit as an example, we use Apriori algorithm to mine the association rules in historical operation data and develop software to guide the operation of the operator. By calculating, the NOx content of the SCR inlet after adopting the optimization scheme is reduced by about 17%. Combined with the “cost-benefit analysis of denitration technology” spectrum, the denitration cost is reduced by about 200, 000 yuan per month.

Core‐sets: An updated survey

AbstractIn optimization or machine learning problems we are given a set of items, usually points in some metric space, and the goal is to minimize or maximize an objective function over some space of candidate solutions. For example, in clustering problems, the input is a set of points in some metric space, and a common goal is to compute a set of centers in some other space (points, lines) that will minimize the sum of distances to these points. In database queries, we may need to compute such a some for a specific query set of k centers. However, traditional algorithms cannot handle modern systems that require parallel real‐time computations of infinite distributed streams from sensors such as GPS, audio or video that arrive to a cloud, or networks of weaker devices such as smartphones or robots. Core‐set is a “small data” summarization of the input “big data,” where every possible query has approximately the same answer on both data sets. Generic techniques enable efficient coreset maintenance of streaming, distributed and dynamic data. Traditional algorithms can then be applied on these coresets to maintain the approximated optimal solutions. The challenge is to design coresets with provable tradeoff between their size and approximation error. This survey summarizes such constructions in a retrospective way, that aims to unified and simplify the state‐of‐the‐art.This article is categorized under Algorithmic Development > Structure Discovery Fundamental Concepts of Data and Knowledge > Big Data Mining Technologies > Machine Learning Algorithmic Development > Scalable Statistical Methods

Research on High Voltage Transmission and Transfer Engineering and Environmental Protection of Electromagnetic Radiation Based on Large Data Mining Technology

The definition of electromagnetic radiation is that when the source current or charge changes with time, a part of electromagnetic energy enters the surrounding space, which is the radiation of electromagnetic energy. By explaining the mechanism of electromagnetic radiation in high-voltage power transmission and transformation project and the influence and harm of electromagnetic radiation pollution in high-voltage power transmission and transformation project, the standard and management of electromagnetic radiation environmental protection in high-voltage power transmission and transformation project are explained. At the same time, the degree of electromagnetic radiation produced by high-voltage power transmission and transformation projects is also described, which is helpful to correctly understand the electromagnetic radiation problems of high-voltage power transmission and transformation projects and to promote the coordinated development of economy and environmental protection. Applying big data mining technology to predict the probability of failure of microcomputer protection equipment in substation, and developing an effective electromagnetic interference suppression strategy by analyzing the generated model to improve the electromagnetic environment in the substation. Combined with the actual work experience of environmental impact assessment of power transmission and transformation engineering, analyze the status of electromagnetic pollution and public participation, find problems and make recommendations.

Discovering the tourists' behaviors and perceptions in a tourism destination by analyzing photos' visual content with a computer deep learning model: The case of Beijing

Visual content analysis of tourist photos is an effective way to excavate tourist behavior and explore tourists' cognition in the tourism destination. With the development of computer deep learning and big data mining technology, identifying the content of massive numbers of tourist photos by Artificial Intelligence (AI) approaches breaks through the limitations of manual approaches of identifying photos' visual information, e.g. small sample size, complex identification process and results deviation. In this study, 35,356 Flickr tourists' photos in Beijing were identified into 103 scenes by computer deep learning technology. Comparison through statistical analysis for behaviors and perceptions of tourists from different continents and countries was conducted. Tourists' cognitive maps with different perceptual themes were visualized according to photos' geographical information by ArcGIS. The field of how to apply AI technology into tourism destination research was explored and extended by this trial study.

Medidor de agua para sistema inteligente de control de agua potable

Objectives: Develop an electronic prototype that respects the environment, through the use of microcontroller cards such as Arduino, Nodemcu, IoT concept and cloud for the control and measurement of water consumption. Objectives specific: Process the data provided by sensors and meters distributed in the main water supplies of the university, as well as the public and private sectors and combine them with information, such as consumption patterns, to build a sophisticated image of how the water network is behaving . through the concept of IoT, data mining, big data and cloud. Measure and evaluate the impacts of the results obtained by the smart meter and that provide the applications for the correct decision making. Objectives Methodologycs: Strengthen the research line of the work team, which sets the standard for us as part of an academic body, of proposing projects based on Internet of Things, Big-Data and data mining technologies, using as a platform the potential of the microcontroller boards (Photon, Arduino, etc.), to enter the new industrial model 4.0 - environment. Create an academic body that carries out research activities through the use of different research methods for the solution to different problems within the institution. Contribution The water resource is essential for life, however, society rarely thinks about the different ways of use that is generally given, or the many activities of daily life in which it is present, and how our life It would change if its availability were near the end of its life cycle. The present project arises from the need to have a better control in the water consumption registers mainly in the Technological University of the North of Guanajuato which is the place where this research originates and from these registers allow an analysis of the data of consumption with greater accuracy and the best decision making.

Fuzzy trees and forests—Review

AbstractData classification and regression are commonly encountered data analysis problems. Many researchers created multiple tools to deal with these issues. Fuzzy clustering, fuzzy decision trees, and ensemble classifiers such as fuzzy forests are popular tools used for this kind of problems. We would like to describe some interesting, more or less popular, solutions which belong to mentioned areas to show the way they deal with data classification and regression problems. This paper is divided into four parts. In the first part we present the issue of fuzzy clustering, which is one of the most important aspects of fuzzy trees which base on clusters. Some methods of splitting objects into clusters using fuzzy logic are described there. The second part describes different fuzzy decision trees. The way these trees can deal with classification and regression problems is presented. In the third part the issue of forests—ensemble classifiers which consist of fuzzy trees—is described. The last part treats about the way of performing weighted decision making in fuzzy forests.This article is categorized under: Fundamental Concepts of Data and Knowledge > Big Data Mining Technologies > Classification Technologies > Prediction Technologies > Machine Learning

Intelligent Transportation Decision Analysis System Based on Big Data Mining

In the big data era, the intelligent transportation field also has begun to use big data technology to mine and analyze the massive traffic data, from which to extract valuable data information, for people to travel to provide convenience and for the relevant traffic decisions to provide an important basis. This paper mainly describes the intelligent transportation and big data mining, and analyzes the application of big data and data mining technology in intelligent transportation decision analysis system, which provides a new way to improve traffic congestion and relieve traffic pressure in big data era.

Social Media Big Data Mining and Spatio-Temporal Analysis on Public Emotions for Disaster Mitigation

Social media contains a lot of geographic information and has been one of the more important data sources for hazard mitigation. Compared with the traditional means of disaster-related geographic information collection methods, social media has the characteristics of real-time information provision and low cost. Due to the development of big data mining technologies, it is now easier to extract useful disaster-related geographic information from social media big data. Additionally, many researchers have used related technology to study social media for disaster mitigation. However, few researchers have considered the extraction of public emotions (especially fine-grained emotions) as an attribute of disaster-related geographic information to aid in disaster mitigation. Combined with the powerful spatio-temporal analysis capabilities of geographical information systems (GISs), the public emotional information contained in social media could help us to understand disasters in more detail than can be obtained from traditional methods. However, the social media data is quite complex and fragmented, both in terms of format and semantics, especially for Chinese social media. Therefore, a more efficient algorithm is needed. In this paper, we consider the earthquake that happened in Ya’an, China in 2013 as a case study and introduce the deep learning method to extract fine-grained public emotional information from Chinese social media big data to assist in disaster analysis. By combining this with other geographic information data (such population density distribution data, POI (point of interest) data, etc.), we can further assist in the assessment of affected populations, explore emotional movement law, and optimize disaster mitigation strategies.

Analysis and Visualization Implementation of Medical Big Data Resource Sharing Mechanism Based on Deep Learning

With the development of information technology, the informationization of the medical industry is also constantly developing rapidly, and medical data is growing exponentially. In the context of “Big Data +”, people began to study the application of data visualization to medical data. Data visualization can make full use of the human sensory vision system to guide users through data analysis and present information hidden behind the data in an intuitive and easy-to-use manner. This paper first introduces the workflow of DBN, a deep learning algorithm, and summarizes the computational characteristics of the algorithm. The classification function is translated into an assembler using an instruction set-based assembly language, and the program is evaluated for performance. Secondly, based on the Hadoop ecosystem, this paper analyzes the BDMISS system for big data medical information resource sharing. Based on the system's requirements and functional positioning, from the medical information collection and sharing, data mining and knowledge management level, the big data medical service system is constructed. Based on the semantic network and ontology theory, big data mining technology and the design of “medical cloud”, the resource sharing mechanism is analyzed. Based on the Spring MVC framework, using Echarts, HCharts and other data visualization technology, according to the design of specific modules, the visualization and display of medical data is realized, which has certain promotion effect on the research and development of medical big data visualization analysis.

Architecture of a decision support system based on big data for monitoring type 2 diabetics

Type 2 diabetes is one of chronic diseases that require continuous and real-time monitoring to prevent the occurrence of complications. First, the doctor must have information about the patient's daily life (vital signs, stress, sedentary lifestyle, physical activities, nutrition, etc.). Secondly, the prescribed treatment must be evaluated each time to test the validity of the diagnosis. To achieve this goal, a decision support system based on big data mining technology must be designed in order to have a centralised knowledge of diabetics. This system will improve the quality of monitoring and treatment from the different data collected. Thus, this paper presents an architecture of a decision support system allowing doctors to monitor the health status of their patients, based on data collected from different resources, in order to enrich the knowledge database and prescribe new treatments based on similar cases and experiences of doctors and patients belonging to this system.

Big data processing tools: An experimental performance evaluation

Big Data is currently a hot topic of research and development across several business areas mainly due to recent innovations in information and communication technologies. One of the main challenges of Big Data relates to how one should efficiently handle massive volumes of complex data. Due to the notorious complexity of the data that can be collected from multiple sources, usually motivated by increasing data volumes gathered at high velocity, efficient processing mechanisms are needed for data analysis purposes. Motivated by the rapid growth in technology, development of tools, and frameworks for Big Data, there is much discussion about Big Data querying tools and, specifically, those that are more appropriated for specific analytical needs. This paper describes and evaluates the following popular Big Data processing tools: Drill, HAWQ, Hive, Impala, Presto, and Spark. An experimental evaluation using the Transaction Processing Council (TPC‐H) benchmark is presented and discussed, highlighting the performance of each tool, according to different workloads and query types.This article is categorized under: Technologies > Computer Architectures for Data Mining Fundamental Concepts of Data and Knowledge > Big Data Mining Technologies > Data Preprocessing Application Areas > Data Mining Software Tools

Transforming big data into smart data: An insight on the use of the k‐nearest neighbors algorithm to obtain quality data

The k‐nearest neighbors algorithm is characterized as a simple yet effective data mining technique. The main drawback of this technique appears when massive amounts of data—likely to contain noise and imperfections—are involved, turning this algorithm into an imprecise and especially inefficient technique. These disadvantages have been subject of research for many years, and among others approaches, data preprocessing techniques such as instance reduction or missing values imputation have targeted these weaknesses. As a result, these issues have turned out as strengths and the k‐nearest neighbors rule has become a core algorithm to identify and correct imperfect data, removing noisy and redundant samples, or imputing missing values, transforming Big Data into Smart Data—which is data of sufficient quality to expect a good outcome from any data mining algorithm. The role of this smart data gleaning algorithm in a supervised learning context are investigated. This includes a brief overview of Smart Data, current and future trends for the k‐nearest neighbor algorithm in the Big Data context, and the existing data preprocessing techniques based on this algorithm. We present the emerging big data‐ready versions of these algorithms and develop some new methods to cope with Big Data. We carry out a thorough experimental analysis in a series of big datasets that provide guidelines as to how to use the k‐nearest neighbor algorithm to obtain Smart/Quality Data for a high‐quality data mining process. Moreover, multiple Spark Packages have been developed including all the Smart Data algorithms analyzed.This article is categorized under: Technologies > Data Preprocessing Fundamental Concepts of Data and Knowledge > Big Data Mining Technologies > Classification

Mining electronic health records to identify influential predictors associated with hospital admission of patients with dementia: an artificial intelligence approach

BackgroundEarly prediction of the outcomes of dementia is important and challenging. This study aimed to identify influential predictors from primary care electronic health records that can robustly predict whether patients with dementia will be admitted to hospital or remain under GP care. MethodsHealth records of patients with dementia were collected from general practice (GP) and hospital data in Wales between 1980 and 2015. These records were linked at individual patient level via the Secure Anonymised Information Linkage databank. The GP records of each patient were selected 1 year before diagnosis up to hospital admission. An artificial intelligence technique, neural network with entropy regularisation (a multilayer feedforward neural network whose weights between input layer and the first hidden layer were regularised by an entropy metric into the fitness function during training process) was used to automatically identify the most influential predictors from initial GP read codes, sex, and age. 10-fold cross validation was used to assess the predictive performance of the identified signals. Findings52·5 million individual records of 59 298 patients (20 674 men, 38 624 women) with dementia were used. 30 178 were admitted to hospital and 29 120 remained with GP care. More men were admitted to hospital than stayed with GP care (11 233 vs 9441), whereas more women stayed with GP care than were admitted (19 679 vs 18 945). From the 54 649 initial event codes, the ten most important signals identified for admission for dementia were two diagnostic events (nightmares, essential hypertension), five medication events (betahistine dihydrochloride, ibuprofen gel, simvastatin, influenza vaccine, calcium carbonate and colecalciferol chewable tablets), and three procedural events (third party encounter, social group 3—skilled, blood glucose raised). They performed significantly above chance to predict admission to hospital with sensitivity of 0·758 (95% CI 0·731–0·785), specificity 0·759 (0·71–0·808), precision 0·766 (0·735–0·797), and negative predictive value 0·751 (0·741–0·761). Linear regression with all raw features yielded values of 0·286 (0·26–0·313), 0·804 (0·792–0·816), 0·487 (0·463–0·511), and 0·633 (0·615–0·651), respectively, and with ten identified features yielded values of 0·684 (0·679–0·691), 0·712 (0·705–0·718), 0·747 (0·738–0·754), and 0·644 (0·64–0·651). InterpretationOutperforming traditional methods, the artificial intelligence technique provides an effective means of identifying influential clinical signals to predict hospital admission of patients with dementia significantly above chance. FundingHealth Data Research UK Wales and Northern Ireland Site, National Centre for Population Health and Wellbeing Research (CA02), Major Project of National Social Science Foundation of China (16ZDA0092), Guangxi University Digital ASEAN Cloud Big Data Security and Mining Technology Innovation Team.

Key Technologies Research of Urban Power Grid AVC System with High Load Density Based on Big Data

In recent years, with the rapid expansion of urban power grids, the automatic voltage control (AVC) system develops some new characteristics. Based on the technology of big data analysis and mining, this paper expounds an AVC system for urban power network with high load density based on the integrated dispatch and operation mode. This paper introduces several different application modes, and shows the operation characteristics of the multi-point distribution mode and the integrated control mode. The big data mining technology is used to discovering the hidden troubles and defects in AVC system. This paper also presents some other AVC key technologies such as security check of remote control, self-adaption analysis method and so on.

Intelligent Traffic Decision Analysis System Based on Big Data Mining

The urban intelligent traffic decision analysis system is a system designed to solve urban road traffic problems, improve traffic safety and comfort, and improve the level of traffic services. The application of big data mining technology can improve the ability of various types of road traffic information analysis and processing, expand the scope of application, and provide decision-making reference for the application of intelligent transportation systems, so as to ensure the smoothness and safety of roads. This article will introduce the concept of big data mining technology and intelligent transportation system, and analyze the architecture of urban traffic intelligent system based on big data technology and the design and implementation of mining system.

Toward efficient business behavior prediction using location‐based social networks

Understanding business behavior in a city requires acquiring huge amount of data coming from diverse field studies. The growing use of mobile devices in social media provides massive data transactions that can replace such data acquired by field studies. Location‐based social networks' (LBSNs') data can be exploited in urban analysis for economic reasons. In this research, the spatial correlation of business turnouts for venues registered in LBSNs is studied for business behavior predictions. A novel similarity embedded (SE)‐spatial interpolation technique is proposed for business turnouts' predictions. The proposed technique utilizes diverse features provided by LBSNs in the prediction process to improve prediction performance. Moreover, a local filter is introduced to avoid local extreme involvements in the prediction process issuing better prediction results. To test the proposed techniques, experimental case study is implemented for predicting business behavior of venues registered in Foursquare in Texas. The proposed SE‐spatial interpolation has shown better prediction accuracy than classical spatial interpolation predictions. The additional integration of the local filter shows further alleviated prediction errors. Furthermore, this study extends the work for the efficient application of the proposed prediction technique in big datasets. An iterative nearest neighbors first search method is designed for accelerating the execution time of the prediction technique implementation regardless the dataset size. The proposed method was tested over several size datasets. The test results show accelerated execution time for the proposed method when compared with the classical implementation execution time.This article is categorized under: Algorithmic Development > Spatial and Temporal Data Mining Fundamental Concepts of Data and Knowledge > Big Data Mining Technologies > Prediction

A nowcasting model for the prediction of typhoon tracks based on a long short term memory neural network

It is of vital importance to reduce injuries and economic losses by accurate forecasts of typhoon tracks. A huge amount of typhoon observations have been accumulated by the meteorological department, however, they are yet to be adequately utilized. It is an effective method to employ machine learning to perform forecasts. A long short term memory (LSTM) neural network is trained based on the typhoon observations during 1949–2011 in China’s Mainland, combined with big data and data mining technologies, and a forecast model based on machine learning for the prediction of typhoon tracks is developed. The results show that the employed algorithm produces desirable 6–24 h nowcasting of typhoon tracks with an improved precision.

Cloud‐based data streams optimization

Many modern applications of sensor networks and transaction analysis require real‐time processing of their stream data sets. These data streams vary continuously over time. Current stream processing approaches focus on only one of the two optimization perspectives, proposing optimization techniques for data streams processing regardless of the processing environment or improving the processing environment only. In this paper, a brief survey of recent approaches to data streams processing coming from the two optimizations perspectives is proposed; their shortcomings are presented as well. Then, a proposal to an innovative and integrative framework is developed; it is referred to as the continuous query optimization based on multiple plans (CQOMP) for data streams over the cloud environment. CQOMP combines the two optimization perspectives and provides an optimized stream clusters processing using multiple split query plans. Each plan is constructed for a cluster of data that has nearest characteristics and it processes streams tuples over the cloud. We also propose a novel algorithm called the optimized multiple plans (OMP) for processing data streams clusters on Cloud Computing. The OMP algorithm efficiently divides data streams and generates optimized multiple split plans. Each plan is for processing a group of data streams on the cloud. We present the experimental results of the OMP solution compared to the alternative state‐of‐the‐art data stream approaches. The experiments show the efficiency and the scalability of the combined OMP algorithm on different cloud environments, the real Amazon cloud environment, and the simulated windows azure cloud environment.This article is categorized under: Fundamental Concepts of Data and Knowledge > Big Data Mining Technologies > Classification Technologies > Data Preprocessing Technologies > Structure Discovery and Clustering

Research on the Application of Persona in Book Recommendation System

Currently, there still exists a host of problems in the book recommendation system, such as low accuracy, weak correlation and poor pertinence. Aiming to unravel these problems, this paper based on the theory of big data and data mining technology, through analyzing internet user behavior and the “5C” model of personal credit evaluation, combined with joint impact weight calculation method, which involves user grade, borrowing credit, book friend recommendation degree, book friend recommended adoption degree, borrowing frequency, borrowing number, and borrowing time interval. User activity and credit are also taken into account in the process of establishing user tagging system so as to build classified book recommendation service. This method is of universal meaning to the book recommendation service of smart campus with user as the core under big data environment.