Development Of Big Data Technology Research Articles

Development of Fault Detection Systems: The Hadoop Ecosystem Implementation

Semiconductor manufacturing relies on a long and complex production line for wafer fabrication. In a modern semiconductor fabrication plant (fab), a wafer manufacturing line is composed of more than 1,000 manufacturing tools and an overhead hoist system for delivering wafer lots among the tools. Novel technologies, such as Internet of Things (IoT) and software engineering technologies, are adopted in semiconductor manufacturing such that large amounts of data can be collected from manufacturing tools. Also, these tools and the overhead hoist system should cooperate well with each other. Due to the data-processing capacity limitation in a current fault detection and classification (FDC) system, the diagnosis efficiency is limited such that some undesired events cannot be detected in time, leading to significant economic loss. With the booming development of big data technology, this work conducts a study on a new FDC framework based on a Hadoop ecosystem to deal with the data-processing limitation and improve diagnosis efficiency. Also, a migration path is presented such that the current FDC system can be smoothly migrated to a Hadoop ecosystem-based one without shutting down a wafer fabrication line. Experimental results show that the proposed FDC framework can run safely and stably.

Visualization of data analysis platform Taking QQ music recommendation system as an example

With the rapid development of big data technology, people's demand for personalized music recommendation systems is growing more and more urgent. However, the current music recommendation system still has some problems, such as inaccurate recommendations and too slow recommendation speeds, as well as cold starts and data sparsity caused by massive data. In order to design and implement a music recommendation system for the recommendation system storage caused by the continuous increase of data, insufficient storage, and computing power, this paper improved the QQ music recommendation system based on the collaborative filtering recommendation algorithm of the offline data warehouse technology project. After testing, the music recommendation system designed in this paper has good stability, scalability, and efficiency.

Research on big data privacy protection technology

The extensive application of big data technology makes data burst with unprecedented value and vitality. However, due to the large amount of data, many data sources, and complex data access relationships, the current development of privacy protection technology is seriously lagging behind the development of big data technology, which restricts the application and promotion of big data. At present, it is urgent to sort out the development status of big data privacy protection technology, so as to provide a reference for the research and breakthrough of key issues of big data privacy protection. This paper analyzes k-anonymity and differential privacy protection, and points out that a protection method that can reduce the cost of high-dimensional data processing and ignore the background knowledge requirements of attackers is urgently needed for privacy protection in big data scenarios. At the same time, this paper also puts forward suggestions for the application and future development of these technologies.

Feature fusion improves performance and interpretability of machine learning models in identifying soil pollution of potentially contaminated sites

Owing to the rapid development of big data technology, use of machine learning methods to identify soil pollution of potentially contaminated sites (PCS) at regional scales and in different industries has become a research hot spot. However, due to the difficulty in obtaining key indexes of site pollution sources and pathways, current methods have problems such as low accuracy of model predictions and insufficient scientific basis. In this study, we collected the environmental data of 199 PCS in 6 typical industries involving heavy metal and organic pollution. Then, 21 indexes based on basic information, potential for pollution from product and raw material, pollution control level, and migration capacity of soil pollutants were used to established the soil pollution identification index system. We fused the original indexes into the new feature subset with 11 indexes through the method of consolidation calculation. The new feature subset was then used to train machine learning models of random forest (RF), support vector machine (SVM), and multilayer perceptron (MLP), and tested to determine whether it improved the accuracy and precision of soil pollination identification models. The results of correlation analysis showed that the four new indexes created by feature fusion have the correlation with soil pollution is similar to the original indexes. The accuracies and precisions of three machine learning models trained on the new feature subset were 67.4%− 72.9% and 72.0%− 74.7%, which were 2.1%− 2.5% and 0.3%− 5.7% higher than these of the models trained on original indexes, respectively. When the PCS were divided into typical heavy metal and organic pollution sites according to the enterprise industries, the accuracy of the model trained on the two datasets for identifying soil heavy metal and organic pollution were significantly improve to approximately 80%. Owing to the imbalance in positive and negative samples in the prediction of soil organic pollution, the precisions of soil organic pollution identification models were 58%− 72.5%, which were significantly lower than their accuracies. According to the factors analysis based on the model interpretability of SHAP, most of the indexes of basic information, potential for pollution from product and raw material, and pollution control level had different degrees of impact on soil pollution. However, the indexes of migration capacity of soil pollutants had the least effect in the classification task of soil pollution identification of PCS. Among the indexes, traces of soil pollution, industrial utilization years/start-up time, pollution control risk scores and enterprise scale having the greatest effects on soil pollution with the mean SHAP values of 0.17–0.36, which reflected their contribution rate on soil pollution and could help to optimize the current index scoring of the technical regulation for identifying site soil pollution. This study provides a new technical method to identify soil pollution based on big data and machine learning methods, in addition to providing a reference and scientific basis for environmental management and soil pollution control of PCS.

Management path optimization of accordion music course education based on big data technology analysis

Abstract With the continuous development of information technology, especially the rapid development of big data technology, the coverage and effectiveness of information application and dissemination have increased substantially. With the current innovation and development of music management education mode, the traditional accordion course management mode also changed, this paper makes full use of big data technology to optimize the accordion course education management path, mainly through the construction of an accordion score database and retrieval system, the closer the data mining algorithm calculation result is to 1, it means that the data mining result is more similar to the actual result. The experimental analysis result is equal to 1, which indicates that the data mining result of this study is similar to the actual result, which is conducive to the intelligent management under the analysis of big data technology for music course education, improving students’ learning interest and passion, improving the teaching quality of learning, and facilitating the teaching management work of university leaders. And it can be further based on the big data era, grasp the trend of the big data era, and understand the level, learning needs, and personal characteristics of students, analysis can be seen that accordion course education management based on big data can make different learning plans for different levels of students, which is conducive to improving management quality and efficiency at the data level.

HQDCNet: Hybrid Quantum Dilated Convolution Neural Network for detecting covid-19 in the context of Big Data Analytics.

Medical care services are changing to address problems with the development of big data frameworks as a result of the widespread use of big data analytics. Covid illness has recently been one of the leading causes of death in people. Since then, related input chest X-ray image for diagnosing COVID illness have been enhanced by diagnostic tools. Big data technological breakthroughs provide a fantastic option for reducing contagious Covid disease. To increase the model's confidence, it is necessary to integrate a large number of training sets, however handling the data may be difficult. With the development of big data technology, a unique method to identify and categorise covid illness is now found in this research. In order to manage incoming big data, a massive volume of chest x-ray images is gathered and analysed using a distributed computing server built on the Hadoop framework. In order to group identical groups in the input x-ray images, which in turn segments the dominating portions of an image, the fuzzy empowered weighted k-means algorithm is then employed. A hybrid quantum dilated convolution neural network is suggested to classify various kinds of covid instances, and a Black Widow-based Moth Flame is also shown to improve the performance of the classifier pattern. The performance analysis of COVID-19 detection makes use of the COVID-19 radiography dataset. The suggested HQDCNet approach has an accuracy of 99.01. The experimental results are evaluated in Python using performance metrics such as accuracy, precision, recall, f-measure, and loss function.

RETRACTED ARTICLE: Design of NoSQL database in oral English teaching based on 5G network and AI recognition

With the continuous development of science and technology, we have fully entered the information age, people's entertainment life is becoming more and more abundant, and the Internet has also provided people with a lot of convenience. The advent of the Internet age means that more and more various kinds of data are appearing, and the situation is becoming more and more abundant. The use of traditional relational databases can no longer store these data, nor can it be queried. With the continuous development of voice technology, database technology based on NoSQL has become a research hotspot. The availability of NoSQL databases is very high, the scalability is also very high, and the efficiency is very high when processing data. Based on the development of 5G networks and the development of big data technology, this research proposes a brand-new business architecture. This architecture can use the network to store data on the basis of massive data. At the same time, we also described the business scenario, providing a new idea for more intelligent education services. We use this model for teaching in schools, and we can transmit some spoken language resources to the school through the Internet for students to use for learning. Nowadays, the application range of AI technology and intelligent technology has become more and more extensive, and these technologies have also been applied in the education field. We can apply this brand-new technology in teaching to promote the development of teaching and improve students’ enthusiasm and learning effect.

Acquisition Method of User Requirements for Complex Products Based on Data Mining

The vigorous development of big data technology has changed the traditional user requirement acquisition mode of the manufacturing industry. Based on data mining, manufacturing enterprises have the innovation ability to respond quickly to market changes and user requirements. However, in the stage of complex product innovation design, a large amount of design data has not been effectively used, and there are some problems of low efficiency and lack of objectivity of user survey. Therefore, this paper proposes an acquisition method of user requirements based on patent data mining. By constructing a patent data knowledge base, this method combines the Latent Dirichlet Allocation topic model and a K-means algorithm to cluster patent text data to realize the mining of key functional requirements of products. Then, the importance of demand is determined by rough set theory, and the rationality of demand is verified by user importance performance analysis. In this paper, the proposed method is explained and verified by mining the machine tool patent data in CNKI. The results show that this method can effectively improve the efficiency and accuracy of user requirements acquisition, expand the innovative design approach of existing machine tool products, and be applied to other complex product fields with strong versatility.

Research on the problems and reform strategies of government statistics based on big data technology

Abstract In the past two years, with the development of big data technology and the revolutionary innovations it has caused within various industries, there has been a boom in the use of big data to transform and optimize government workflow within the field of public administration. In this paper, we build a theoretical framework through government statistics, focusing on the four elements of government statistics. The decision tree algorithm is used to analyze the indicators of the government statistical work problem, mainly from three aspects: grassroots statistical institutions, government revenue, and government forestry construction statistics. The accuracy of the decision tree algorithm is controlled between 83% and 91%, with a variation range value of 8%, which is more accurate than the other two algorithms. The decision tree algorithm helps improve the efficiency and quality of government statistics, thus contributing to the processing of statistical work problems and reform strategies. This study is important for deepening the reform of the Chinese government administration system, improving the construction of a service-oriented government, ensuring the quality of statistical information, and guiding the benign development of statistical work.

A Multiperspective Intervention Path Study of Employment Expectancy in the Era of Big Data

The increasingly large employment group in colleges and universities has brought huge employment pressure to the society. Therefore, improving the level of professional ability of college students, providing students with personalized career development direction planning, and helping students to establish a correct concept of career selection are feasible ways to solve the structural contradiction of college students’ employment. The arrival of big data era provides new development opportunities for employment work in colleges and universities. Big data drives the high-quality development of precise employment in colleges and universities, which is inevitably required to reach the precise requirements of work development, information mastering, platform docking, information pushing, and help work. The traditional employment guidance work is backward in means and poor in timeliness, which leads to poor intervention effect on students’ employment expectation. With the deepening of digital campus construction and the development of big data technology, massive educational data has been accumulated but not reasonably utilized. Therefore, in this paper, we take the massive student data accumulated in digital campus as the research object, based on the research method of convolutional neural network, to explore the hidden personalized information of students behind it, predict their future career development direction, and provide scientific and technological support for the work of university education and teaching. Thus, the architecture classical DenseNet was improvised to avoid gradient disappearance and guarantee the classification accuracy, thereby targeting to reduce the number of connections. The study also proposed a 2-DenseNet model, wherein the attention module was embedded into 2-DenseNet followed by the conduction of training and validation on the university employment dataset. The experimental results show that the method proposed in this paper could predict the appropriate career development direction timely and effectively based on multiple perspectives of students’ comprehensive quality. This would enable the students to plan and adjust their employment expectations leading to disseminating of efficient employment guidance in colleges and universities.

Evaluation and prediction of the safety management efficiency of coal enterprises based on a DEA-BP neural network

With the development of big data technology, big data have gradually become the main driving force of safety management. To effectively predict and evaluate the changes in the safety management efficiency of coal enterprises under big data technology, the model-driven DEA method and the data-driven BP neural network method are combined. A DEA-BP neural network hybrid-driven model, in which safety management efficiency is considered, is constructed for coal enterprise evaluation and prediction. The data of 20 coal enterprises from 2015 to 2020 are used for empirical research. The results indicate that the static analysis shows a fluctuating upwards trend in the level of safety management of coal enterprises. The dynamic analysis shows that the decline in the technical efficiency change is mainly caused by the decline in the pure technical efficiency change. In terms of improving safety management efficiency, coal enterprises mainly target a single aspect of technical progress efficiency, without realizing that the improvement of safety efficiency is the result of the combined effect of technical efficiency and technical progress efficiency. An accuracy of 90% can be achieved using the BP neural network prediction based on the addition of new indicators, demonstrating that the evaluation and prediction model has good adaptability.

The Significance of Financial Accounting Transformation in the Context of Big Data

In the era of big data, financial accounting transformation is one of the strategic directions that enterprises must pursue. The development of big data technology and application in the field of financial accounting provides enterprises with more, more accurate and more timely information support, enabling enterprises to better grasp market dynamics, quickly respond to customer needs, optimize resource allocation, and reduce costs. The significance of big data for financial accounting is to improve the efficiency and accuracy of data processing, help enterprises better understand market trends and customer needs, optimize financial management and risk control, and promote the digital transformation of enterprises. Therefore, the transformation of financial accounting is one of the important strategic directions for enterprises to pursue competitive advantages and achieve sustainable development.

Research on Decision-Making Strategies of Regional Geography Teaching in High School Based on Data-Driven

With the development of big data technology, it has become a trend to analyze students’ behavioral data to promote teachers’ precision teaching. In this context, this study cooperates with a senior high school geography teacher in Changsha City, Hunan Province, collects the students’ examination data from 2020 to 2021 academic year with the help of Zhixue.com, divides the students into four groups according to the examination results, and takes regional geography knowledge as an example to analyze the weak knowledge points, discipline core literacy and practical application ability of students at all levels. Based on the analysis results, the improvement strategies of teachers’ subsequent precision teaching were proposed, including students’ stratification should be carried out according to daily performance and examination results; The weak knowledge points of students of different grades overlap and differ, so teachers should carry out targeted teaching for students of different grades. At the same time, the multi-level teaching method should be improved to enhance students’ discipline core literacy. Pay attention to contact life reality, enhance students’ practical application ability.

E-commerce consumer behavior analysis based on big data

In the background of the widespread development of big data technology, the actual behavior of e-commerce consumers is affected by different consumer environment and consumer demand. This article conducts in-depth inquiry and analysis of the behavior of different e-commerce users, and more accurately conducts a comprehensive analysis and effective assessment of various factors that affect consumer behavior. The factors and measures of consumer behavior are comprehensively judged to get the evaluation model closer to consumer behavior. By conducting a comprehensive full-process research on different e-commerce platforms, consumer behavior, consumption habits, and consumption processes, a comprehensive full-process research, more effective channel methods and dependence on consumers’ purchase of goods, conduct a comprehensive research other factors of behavior, such as transportation logistics, environmental conditions, advertising, and other factors, conduct comprehensive judgment and analysis, and provide important theoretical basis and certain research in order to promote and improve the orderly development of consumer behavior and e-commerce environment. The research adopts modeling and analysis methods, through the evaluation of the research model, it can be seen that under the background of big data, e-commerce consumer behavior presents the trend of data, diversification and complexity. Factors affecting consumer behavior are more diverse and complex. It effectively expands the research field of e-commerce consumer behavior analysis, and provides scientific reference for the development of e-commerce.

DBSDS: A dual‐blockchain security data sharing model with supervision and privacy‐protection

AbstractWith the rapid development of big data technology and applications, sharing and supervision of massive data have become the demand of industry development. As an emerging technology, blockchain, with excellent characteristics, can promote the prosperity and development of the data‐sharing industry. However, the existing blockchain may lead to some privacy and security issues due to its transparency and lack of supervision. Hence, a secure data sharing model with supervision and privacy protection, named DBSDS, was proposed, which supports illegal content supervision and users revocation, and possesses flexible and fine‐grained access control on data sharing. Firstly, it introduces a dual‐blockchain architecture, one blockchain is used to ensure the integrity and traceability of private data and another is utilized to supervise illegal users so that they can no longer upload and share private data. Secondly, a key tracing tree was built and correspondingly different key generation strategies for different users were designed, which endows regulators with supervision capabilities. Finally, combined ABPER‐KS algorithm, both privacy protection and fine‐grained access control on private data can be accomplished. Furthermore, security analysis and performance comparison manifest that DBSDS is safe and owns better performance, and scheme implementation and experimental analysis indicate the practicality of DBSDS.

A strategy for building a smart sports platform based on machine learning models

With the rapid development of big data technology, it has greatly changed the way people get information, and also improved the speed and quality of information. In this context, smart sports has become a new trend in sports development. This paper creates an intelligent learning environment and builds a smart sports platform through advanced concepts and technical means, which can effectively optimize the integration and sharing of sports resources. Starting from the overall architecture design of smart sports, the key technologies of machine learning model to realize smart sports are sorted out. Through the five basic linking stages with machine learning model as the core, the value innovation path of platform construction structure is analyzed. The current status of sports resources application is studied, and the data mining algorithm is used to calculate the user usage data of the smart sports platform and improve the construction of the smart platform. Through the construction of the smart sports platform, people shift from traditional reading books and watching TV programs to getting information through intelligent mobile terminals, and the proportion of attention to sports information is as high as 58.6%. This shows that by building a smart sports platform, it can provide support and guarantee for the sustainable development of sports.

Context-Aware Point-of-Interest Recommendation Based on Similar User Clustering and Tensor Factorization

The rapid development of big data technology and mobile intelligent devices has led to the development of location-based social networks (LBSNs). To understand users’ behavioral patterns and improve the accuracy of location-based services, point-of-interest (POI) recommendation has become an important task. In contrast to the general task of product recommendation, POI recommendation faces the problems of the sparsity and weak semantics of user check-in data. To address these issues, an increasing number of studies have improved the accuracy of POI recommendations by introducing contextual information such as geographical, temporal, textual, and social relations. However, the rich context also brings great challenges to POI recommendation, such as the low utilization rate of context information, difficulty in balancing the richness of contextual information, and the complexity of the recommendation matrix. Considering that similar users have more interest preferences in common than users generally have, the check-in information of similar users has greater reference meaning. Thus, we propose a personalized POI recommendation method named CULT-TF, which incorporates similar users’ contextual information into the tensor factorization model. First, we present a user activity model and a user similarity model, which integrate contextual information to calculate the user activity and similarity between users. According to user activity, the most representative active users are selected as user clustering centers, and then users are clustered based on user similarity into several similar user clusters (C). Next, we construct a third-order tensor (user-location-time matrix) for each user cluster by using user activity, POI popularity, and time slot popularity as the eigenvalues in the user (U), location (L), and time (T) dimensions, and the eigenvalue of each dimension is modeled by integrating contextual information of users’ check-in behavior at the user, location, and time levels. Similar user clustering reduces the number of users in tensor modeling, reducing the U dimension. To further reduce the complexity of the recommendation matrix, the reduction of the L dimension is achieved through ROI (region of interest) clustering, and the reduction of the T dimension is achieved through time slot encoding. Then, we use tensor factorization (TF) to obtain the recommendation results. Our method decreases the complexity of the tensor matrix and integrates rich contextual information on users’ check-in behavior. Finally, we conducted a comprehensive performance evaluation of CULT-TF using real-world LBSN datasets from Brightkite. The experimental results show that our proposed method performs much better than other recommendation methods in terms of precision and recall.

Research on methods of personalized English teaching promoted by Big Data teaching platform

In the reform and development of modern education, the integration and use of big data education platform to build and promote personalized English teaching system can not only reflect the teaching concept of individualized teaching, but also fully meet the learning needs of different groups of students at different stages. On the basis of understanding the development of big data technology and the research status of personalized English teaching, this paper uses fuzzy control centralized algorithm to improve the recommendation mechanism of English teaching system, so that students can choose suitable professional teachers according to their own needs. At the same time, after drawing students' learning portraits, the portrait technology is used to evaluate students' passing rate. This provides an effective basis for the practice of personalized English teaching guidance. The final experimental results prove that the design of the whole system meets the personalized needs of the student users, and in the future technology research and development, we should pay attention to the design of functional services from the perspective of professional teachers.

Design and Implementation of Project Schedule Management Data Collection and Analysis System

With the development of the information age, the application scope of big data technology is becoming more and more extensive. The rapid development of big data technology provides strong support for the collection and storage of massive data. Mining potential value through data analysis and providing scientific support for decision makers. This paper combines data collection and analysis with project schedule management. First, it introduces the data collection and analysis technology in general, then explains the disadvantages of traditional project schedule management, analyzes the advantages of modern project schedule management, and finally proposes project schedule management data collection and analysis. The design idea of the system is applied to the project management practice of a material acceptance point in a government office, in order to provide a useful reference for the application of data collection and analysis technology in project schedule management.

A causal inference method for improving the design and interpretation of safety research

Although the research into causal effects has always been the core part of safety science, the formal mathematical method to infer causal effects is quite new. With the rapid development of big data technology, safety research based on experience changes to data-driven research. This change improves the reliability and accuracy of safety decision-making to a certain extent. However, researchers pay too much attention to the amount of data and computing power, accelerating the separation of prior knowledge and data. We proposed a systematic method suitable for safety science to carry out quantitative causal inference to study causal problems such as the effectiveness of safety countermeasures. This method aims to integrate safety-related prior knowledge and data organically and provide essential guidance for research design and data collection. Finally, an empirical study was conducted to illustrate how the approach can accurately estimate the rigorous causal relationship between factors and how it can help researchers understand the complex causal mechanism behind the safety or accident phenomenon.