Big Data Problems Research Articles

Efficiency-Aware Dynamic Service Pricing Strategy for Geo-Distributed Fog Computing

Fog computing provides new ideas for solving big data problems in the Internet of Things (IoT) era. With fog, we can save much time on long-distance transmission, thereby increasing the efficiency of network service. However, to transform fog computing from a technology to a service, we first need to properly handle the trading relationship between users and service provider in fog. In this paper, we design an efficiency-aware dynamic service pricing strategy to optimize the payoffs of both users and provider in fog computing. In the modeling, we design a Stackelberg competition-based model while regarding provider as the market leader, and each individual user as a follower. Users in this model care about how to obtain the most cost-effective services. And provider pays attention to taking full advantage of the geo-distributed characteristic in setting prices with multiple customers. In the performance evaluation part, we carry out experiments using real-world datasets to simulate this continuous negotiation process between supply and demand in fog pricing. The results show that our strategy can solve the dual-objective optimization problem while establishing a stable trading relationship between the two sides.

Enhance Document Contextual Using Attention-LSTM to Eliminate Sparse Data Matrix for E-Commerce Recommender System

E-commerce has been the most important service in the last two decades. E-commerce services influence the growth of the economic impact worldwide. A recommender system is an essential mechanism for calculating product information for e-commerce users. The successfulness of recommender system adoption influences the target revenue of an e-commerce company. Collaborative filtering (CF) is the most popular algorithm for creating a recommender system. CF applied a matrix factorization mechanism to calculate the relationship between user and product using rating variable as intersection value between user and product. However, the number of ratings is very sparse, where the number of ratings is less than 4%. Product Document is the product side information representation. The document aims to advance the effectiveness of matrix factorization performance. This research considers to the enhancement of document context using LSTM with an attention mechanism to capture a contextual understanding of product review and incorporate matrix factorization based on probabilistic matrix factorization (PMF) to produce rating prediction. This study employs a real dataset using MovieLens dataset ML.1M and Amazon information video (AIV) to observe our ATT-PMF model. Movielens dataset represents of number sparse rating that only contains below 4% (ML.1M). Our experiment report shows that ATT-PMF outperforms more than 2% on average than previous work. Moreover, our model is also suitable to implement on huge datasets. For further research, enhancement of product document context will be a good factor in eliminating sparse data problems in big data problems.

An Efficient Learning Framework for Federated XGBoost Using Secret Sharing and Distributed Optimization

XGBoost is one of the most widely used machine learning models in the industry due to its superior learning accuracy and efficiency. Targeting at data isolation issues in the big data problems, it is crucial to deploy a secure and efficient federated XGBoost (FedXGB) model. Existing FedXGB models either have data leakage issues or are only applicable to the two-party setting with heavy communication and computation overheads. In this article, a lossless multi-party federated XGB learning framework is proposed with a security guarantee, which reshapes the XGBoost’s split criterion calculation process under a secret sharing setting and solves the leaf weight calculation problem by leveraging distributed optimization. Remarkably, a thorough analysis of model security is provided as well, and multiple numerical results showcase the superiority of the proposed FedXGB compared with the state-of-the-art models on benchmark datasets.

Deep learning methods may not outperform other machine learning methods on analyzing genomic studies.

Deep Learning (DL) has been broadly applied to solve big data problems in biomedical fields, which is most successful in image processing. Recently, many DL methods have been applied to analyze genomic studies. However, genomic data usually has too small a sample size to fit a complex network. They do not have common structural patterns like images to utilize pre-trained networks or take advantage of convolution layers. The concern of overusing DL methods motivates us to evaluate DL methods’ performance versus popular non-deep Machine Learning (ML) methods for analyzing genomic data with a wide range of sample sizes. In this paper, we conduct a benchmark study using the UK Biobank data and its many random subsets with different sample sizes. The original UK Biobank data has about 500k participants. Each patient has comprehensive patient characteristics, disease histories, and genomic information, i.e., the genotypes of millions of Single-Nucleotide Polymorphism (SNPs). We are interested in predicting the risk of three lung diseases: asthma, COPD, and lung cancer. There are 205,238 participants have recorded disease outcomes for these three diseases. Five prediction models are investigated in this benchmark study, including three non-deep machine learning methods (Elastic Net, XGBoost, and SVM) and two deep learning methods (DNN and LSTM). Besides the most popular performance metrics, such as the F1-score, we promote the hit curve, a visual tool to describe the performance of predicting rare events. We discovered that DL methods frequently fail to outperform non-deep ML in analyzing genomic data, even in large datasets with over 200k samples. The experiment results suggest not overusing DL methods in genomic studies, even with biobank-level sample sizes. The performance differences between DL and non-deep ML decrease as the sample size of data increases. This suggests when the sample size of data is significant, further increasing sample sizes leads to more performance gain in DL methods. Hence, DL methods could be better if we analyze genomic data bigger than this study.

A critical evaluation of handling uncertainty in Big Data processing

Big Data is a modern economic and social transformation driver all over the world. The selection has reached a tipping point in terms of big technological advancements that could usher in new ones. It's about making decisions, taking charge of our health, our cities, our wallets, and our education. While the amount, diversity, speed, and reliability of data has increased, so has the complexity of the data (Amzal et al., 2006). Our ability to identify the "value of information through Big Data" will have the most impact. As a result of the resolution of large data concerns, Big Data is revolutionising the IT sector Software vendors in many industries are unable to move forward with dependability and scalability, Information on the project. With the help of relational databases, the explanation for why business requires adequate atomic transactions in the business sector. As a result, to benefit from both technology, advantages, and solutions. Big Data, which refers to databases that are too huge to be processed with current database administration tools, is gaining traction in a variety of significant applications, including internet search, business, social networking, genomics, and meteorology [2]. This concept highlights key research challenges and the promise of data-driven optimization that organically integrates fuzzy, machine learning, and deep learning for decision-making under uncertainty, and identifies potential research opportunities in the business field of Bayesian optimization under uncertainty through a modern data lens. Big Data is a big issue for science and database data mining. Here, we look at the fascinating activities that my community has been doing at this conference to talk about the Big Data problem. As a result, the central focus is linking Big Data to individuals in numerous ways.

Innovation and Reform of Ideological and Political Course Mode in Colleges and Universities Based on Big Data Network Platform.

With the continuous development of the computer field, the Internet has become the main way of education in today's era. The development of computer technology and big data has brought innovation and opportunities to the teaching mode of ideological and political courses in colleges and universities. Due to the dynamic change of big data, there are many difficulties in the establishment of ideological and political course models in colleges and universities. This paper studies the innovation of ideological and political course modes in colleges and universities on the network platform under the background of big data. This paper mainly compares the traditional teaching mode and explores the formal modeling method of network platform education. The discrete dynamic modeling technology of the complex system is used to model the formal process of online education. Then, it carries out nonlinear prediction modeling for the operation efficiency and traffic change of the network platform under the big data environment and uses dynamic modeling to predict the learning effect of college students' ideological and political course mode. The results show that the formal method of education under dynamic modeling can improve the defects of traditional teaching and give full play to the advantages of network platform teaching. Improving the teaching mode of network platform can improve students' learning efficiency. Finally, in the dynamic model prediction, the problem of big data affecting the model results is improved, and the accuracy of the prediction model is improved.

Big Data: An Analytical Approach - Strategy Implementation, Governance and Emergence

Due Big data is a term for an amalgamation of huge amount of varied and complex data which has the difficulty in analysing, storing and utilization. The process to perform research into this large collection of varied data collected from different data sources to get deeper analysis and insight is known as Big data analysis. With an ever growing technological landscape and digital societal changes by means of social media, IOT and other digital platform usage the need to access actionable insight and their corelation is required the most. If the capabilities are there, organisations can use the enormous datasets, but there are also many hurdles involved. Despite the potential benefits, a significant hurdle is to appropriately process and control the data. It requires thorough understanding of big data governance, adoption, implementation methodology and technological choices. The paper analysis the analytical methodologies, available technologies and tools required for the big data analytics. Big Data management's key features, problems, and difficulties have been examined. There have been discussions about multiple frameworks which perform analysis on different kinds of workloads , this include tools developed by open source community as well as OEM’s. The paper also explores its application in various domains in terms of today’s technological enhancements.

Quantifying research interests in 7,521 mammalian species with h-index: a case study.

BackgroundTaxonomic bias is a known issue within the field of biology, causing scientific knowledge to be unevenly distributed across species. However, a systematic quantification of the research interest that the scientific community has allocated to individual species remains a big data problem. Scalable approaches are needed to integrate biodiversity data sets and bibliometric methods across large numbers of species. The outputs of these analyses are important for identifying understudied species and directing future research to fill these gaps.FindingsIn this study, we used the species h-index to quantity the research interest in 7,521 species of mammals. We tested factors potentially driving species h-index, by using a Bayesian phylogenetic generalized linear mixed model (GLMM). We found that a third of the mammals had a species h-index of zero, while a select few had inflated research interest. Further, mammals with higher species h-index had larger body masses; were found in temperate latitudes; had their humans uses documented, including domestication; and were in lower-risk International Union for Conservation of Nature Red List categories. These results surprisingly suggested that critically endangered mammals are understudied. A higher interest in domesticated species suggested that human use is a major driver and focus in mammalian scientific literature.ConclusionsOur study has demonstrated a scalable workflow and systematically identified understudied species of mammals, as well as identified the likely drivers of this taxonomic bias in the literature. This case study can become a benchmark for future research that asks similar biological and meta-research questions for other taxa.

Big Data Analysis and Its Application in Different Industrial Domains

Abstract: Big data can bring “great value” to our lives in almost every aspect. Technologically, Big Data brings changes in our lives as it enables the full integration and analysis of diverse and heterogeneous data to help us make decisions. Today, with Big Data technology, thousands of data from seemingly unrelated domains can help make important decisions. This is the power of Big Data. Application Areas Health, Well-being, Policy Making and Public Opinion, Smart Cities and More Efficient Society and Robotic Interaction, New Online Education Model: Model student-teacher geometry Foundations in Big Data Analytics Research: Develop and research fundamental theories, algorithms, techniques, methodologies, technologies to solve problems effectively and efficiently to allow the application of Big Data problems. Scientists in the field and big data researchers. This Paper Presentation talks about Big Data analysis and its application in different industrial domains under the following sub topic: Introduction: A brief overview of Big Data analysis and its significant application in different industry and in the field of research. Case study: Discussion of a case study of Big Data together with IoT provides the infrastructure for collecting the data for different domain application of smart cities as important application goal. Work flow: Showcases simplified version of overall work flow of the case study. Conclusion: Discussion of concluded outcome and breakthrough of data analytics.

Preface to the Focus Section on Big Data Problems in Seismology

Preface to the Focus Section on Big Data Problems in Seismology

Construction of Data Resource Sharing Platform in College Students’ Ideological and Political Education Based on Deep Learning

At present, the irrational allocation of IPE (ideological and political education) resources in universities reflects the dilemma of university’s control of university students’ IPE system. In order to make educational big data really useful to us and really promote the sharing of educational resources and improve the utilization rate of resources, this paper discusses the construction of data resource sharing platform for college students in IPE field based on DL (Deep learning) based on the characteristics and existing problems of educational big data. Based on DL, the collected data are parameterized, the matching algorithm parameters are determined, the dynamic data sample subset is incrementally reduced, the discernibility matrix and logical analytic formula are established, the conjunctive normal form is obtained through calculation, and the kernel attribute is introduced into each conjunctions to realize incremental mining of dynamic data. The results show that the model in this paper achieves the best classification effect on six samples, with an average classification accuracy of 88.81%. The results show that the comprehensive shared data matching algorithm based on DL can realize the matching in a short time, and the matching process has high stability.

NOFACE: A new framework for irrelevant content filtering in social media according to credibility and expertise

Social networks have taken an irreplaceable role in our lives. They are used daily by millions of people to communicate and inform themselves. This success has also led to a lot of irrelevant content and even misinformation on social media. In this paper, we propose a user-centred framework to reduce the amount of irrelevant content in social networks to support further stages of data mining processes. The system also helps in the reduction of misinformation in social networks, since it selects credible and reputable users. The system is based on the belief that if a user is credible then their content will be credible. Our proposal uses word embeddings in a first stage, to create a set of interesting users according to their expertise. After that, in a later stage, it employs social network metrics to further narrow down the relevant users according to their credibility in the network. To validate the framework, it has been tested with two real Big Data problems on Twitter. One related to COVID-19 tweets and the other to last United States elections on 3rd November. Both are problems in which finding relevant content may be difficult due to the large amount of data published during the last years. The proposed framework, called NOFACE, reduces the number of irrelevant users posting about the topic, taking only those that have a higher credibility, and thus giving interesting information about the selected topic. This entails a reduction of irrelevant information, mitigating therefore the presence of misinformation on a posterior data mining method application, improving the obtained results, as it is illustrated in the mentioned two topics using clustering, association rules and LDA techniques.

Evolving Gaussian on-line clustering in social network analysis

In this paper, we present an evolving data-based approach to automatically cluster Twitter users according to their behavior. The clustering method is based on the Gaussian probability density distribution combined with a Takagi–Sugeno fuzzy consequent part of order zero (eGauss0). This means that this method can be used as a classifier that is actually a mapping from the feature space to the class label space. The eGauss method is very flexible, is computed recursively, and the most important thing is that it starts learning “from scratch”. The structure adapts to the new data using adding and merging mechanisms. The most important feature of the evolving method is that it can process data from thousands of Twitter profiles in real time, which can be characterized as a Big Data problem. The final clusters yield classes of Twitter profiles, which are represented as different activity levels of each profile. In this way, we could classify each member as ordinary, very active, influential and unusual user. The proposed method was also tested on the Iris and Breast Cancer Wisconsin datasets and compared with other methods. In both cases, the proposed method achieves high classification rates and shows competitive results.

Klasifikasi Teks menggunakan Genetic Programming dengan Implementasi Web Scraping dan Map Reduce

Classification of text documents on online media is a big data problem and requires automation. Research has developed a text classification system with pre-processing using map-reduce and web scraping data collection. This study aims to evaluate text classification performance by combining genetic programming algorithms, map-reduce and web scraping for processing large data in the form of text. Data collection was carried out by observing web-based scraping. Data was collected by reducing 8126 duplicates. Map-reduce has tokenized and stopped-word removal with 28507 terms with 4306 unique terms and 24201 duplication terms. Text classification evaluation shows that a single tree produces better accuracy (0.7072) than a decision tree (0.6874), and the lowest is a multi-tree (0.6726). For the acquisition of genetic programming support values with the multi-tree, the highest average support is 0.3854, followed by the decision tree with 0.3584 and the smallest single tree with 0.3494. In general, the amount of support is not in line with the accuracy value achieved.

Twitter Data Visualization using SVM (Machine Learning Approach)

Abstract: Twitter produces a massive amount of data due to its popularity that is one of the reasons underlying big data problems. One of those problems is the classification of tweets due to use of sophisticated and complex language, which makes the current tools in- sufficient. We present our framework HTwitt, built on top of the Hadoop ecosystem, which consists of a MapReduce algorithm and a set of machine learning techniques embedded within a big data analytics platform to efficiently address the following problems. Keywords: Support Vector Machine (SVM), Machine learning, Big Data, MapReduce

Automated 2D, 2.5D, and 3D Segmentation of Coral Reef Pointclouds and Orthoprojections.

Enabled by advancing technology, coral reef researchers increasingly prefer use of image-based surveys over approaches depending solely upon in situ observations, interpretations, and recordings of divers. The images collected, and derivative products such as orthographic projections and 3D models, allow researchers to study a comprehensive digital twin of their field sites. Spatio-temporally located twins can be compared and annotated, enabling researchers to virtually return to sites long after they have left them. While these new data expand the variety and specificity of biological investigation that can be pursued, they have introduced the much-discussed Big Data Problem: research labs lack the human and computational resources required to process and analyze imagery at the rate it can be collected. The rapid development of unmanned underwater vehicles suggests researchers will soon have access to an even greater volume of imagery and other sensor measurements than can be collected by diver-piloted platforms, further exacerbating data handling limitations. Thoroughly segmenting (tracing the extent of and taxonomically identifying) organisms enables researchers to extract the information image products contain, but is very time-consuming. Analytic techniques driven by neural networks offer the possibility that the segmentation process can be greatly accelerated through automation. In this study, we examine the efficacy of automated segmentation on three different image-derived data products: 3D models, and 2D and 2.5D orthographic projections thereof; we also contrast their relative accessibility and utility to different avenues of biological inquiry. The variety of network architectures and parameters tested performed similarly, ∼80% IoU for the genus Porites, suggesting that the primary limitations to an automated workflow are 1) the current capabilities of neural network technology, and 2) consistency and quality control in image product collection and human training/testing dataset generation.

QUANTUM COMPUTING: A REVIEW ON BIG DATA ANALYTICS AND DATA SECURITY

The world's most difficult problem during the preceding decade was the big data problem. The big data challenge refers to the fact that data is growing at a much faster rate than computational rates. People, as well as virtually all organizations and scientific institutions, are keeping a rising amount of data as the cost of storage falls every day. Large amounts of data are generated by social activities, scientific inquiries, biological discoveries, and sensor devices. Big data is beneficial to society and economy, but it also poses challenges to scientific communities. Traditional tools, machine learning algorithms, and methodologies cannot handle, manage, or analyze large amounts of data. Quantum computing, in the realm of Big Data, allows businesses to collect and analyze large volumes of data quickly using quantum algorithms. Separate data sets may be detected, analyzed, integrated, and diagnosed with far greater ease. To find patterns, all of the items of a large database can be studied at the same time. As a result, it may be years before quantum computing makes its way into most businesses or becomes a common data analytics tool. Quantum computing will still be a relatively new technology in 2021. Machine learning algorithms are currently improving thanks to breakthroughs in quantum computing technology. There's still a lot to learn about quantum computing's capabilities and the implications of such a strong technology.

Accuracy and processing speed trade-offs in classical and quantum SVM classifier exploiting PRISMA hyperspectral imagery

ABSTRACT Quantum machine learning (QML) focuses on machine learning models developed explicitly for quantum computers. Availability of the first quantum processor led to further research, particularly exploring possible practical applications of QML algorithms in the remote sensing field. The demand for extensive field data for remote sensing applications has started creating bottlenecks for classical machine learning algorithms. QML is becoming a potential solution to tackle big data problems as it can learn from fewer data. This paper presents a QML model based on a quantum support vector machine (QSVM) to classify Holm Oak trees using PRISMA hyperspectral Imagery. Implementation of quantum models was carried on a quantum simulator and a real-time superconducting quantum processor of IBM. The performance of the QML model is validated in terms of dataset size, overall accuracy, number of qubits, training and predicting speed. Results were indicative that (i) QSVM offered 5% higher accuracy than classical SVM (CSVM) with 50 samples and ≥12 qubits/feature dimensions whereas with 20 samples at 16 Qubits/feature dimension, (ii) training time for QSVM at maximum accuracy was 284 s with 50 samples and with 20 samples was 53.68 s and (iii) predicting time for 400 pixels using the QSVM model trained with 50 samples dataset was 5243 s whereas with 20 samples dataset was 2845 s. Results were indicative that QML offers better accuracy but lack training and predicting speed for hyperspectral data. Another observation is that predicting speed of QSVM depends on the number of samples used to train the model.

An effective approach for Arabic document classification using machine learning

Arabic text classification is one application of Natural Language Processing (NLP). It has been used to analyze and categorize Arabic text. Analyzing text has become an essential part of our lives because of the increasing number of text data which makes text classification a big data problem. Arabic text classification systems become significant to maintain vital information in many domains such as education, and health sector, and public services. In the presented research work, the Arabic text classification model is developed using various algorithms namely Multinomial Naïve Bayesian (MNB), Bernoulli Naïve Bayesian (BNB), Stochastic Gradient Descent (SGD), Logistic Regression (LR), Support vector classifier (SVC), Linear SVC, and convolutional neural networks (CNN). These algorithms have been implemented utilizing the Al-Khaleej dataset. The experiments are carried out with various representation models and it is observed that CNN with character level model outperforms others. The result of CNN exceeds the state-of-the-art machine learning method with an accuracy equal to 98. The presented methods will be useful in different domains, particularly on social media.

Implicit data crimes: Machine learning bias arising from misuse of public data

SignificancePublic databases are an important resource for machine learning research, but their growing availability sometimes leads to "off-label" usage, where data published for one task are used for another. This work reveals that such off-label usage could lead to biased, overly optimistic results of machine-learning algorithms. The underlying cause is that public data are processed with hidden processing pipelines that alter the data features. Here we study three well-known algorithms developed for image reconstruction from magnetic resonance imaging measurements and show they could produce biased results with up to 48% artificial improvement when applied to public databases. We relate to the publication of such results as implicit "data crimes" to raise community awareness of this growing big data problem.