Big Data Volume Research Articles

Astoundingly Smart System Furnishing Ranking of Big Data in Search Engines

The abruptly escalating internet is sensational. It inculcates a humungous volume of big data, which is obsolete and tedious to manage, scrutinize, analyze and perform operations upon them in conventional ways. Big data has thus expedited the search and retrieval of information, necessitating the development of contemporary search algorithms to aid this process. However, the primary hindrance for the first and second generations of conventional search engines was the syntax of keywords devoid of semantic meaning and the lack of a knowledge base that linked disparate web material. This article presents a framework based on trendy technologies, specifically Extracting, Transforming and Integrating (ETI) processes, ontology graphs, and indexing Resource Description Framework (RDF) using the wide-column Not only Structured Query Language (NoSQL) method. The most significant contribution in this regard is developing a mathematical model to compute the similarity score between a query and stored RDF documents using semantic relations. Numerous operations were carried out to evaluate the effectiveness of the proposed methodology in installing data sources, such as DBpedia and YAGO dataset. Insofar as experimental results are concerned, the suggested model achieves greater precision than other comparable systems.

System Identification at the Extreme Edge for Network Load Reduction in Vibration-Based Monitoring

Mechanical complexity, wide dimensions, and big data volume may hamper the implementation of Internet of Things (IoT)-enabled structural health monitoring (SHM) systems. In particular, one of the most important challenges is the reduction of the data payload to be transmitted over the monitoring network. Addressing the problem in the context of vibration-based SHM, this work explores system identification (SysId) as an innovative strategy for data compression at the extreme edge. Indeed, SysId is a signal processing technique aiming at finding a very reduced (i.e., less then one tenth of the total signal length) set of meaningful parameters, which can provide an alternative, but yet completely equivalent, frequency characterization of the structure. In the proposed approach, an embedded system-oriented adaptation of the sequential tall-skinny QR decomposition (eS-TSQR) from the dense linear algebra domain has been exploited to tackle both the memory and computational complexity of the involved algorithms. This yielded to the embodiment of input–output and output-only SysId models into a resource constrained device (i.e., an STM32L5 microcontoller unit), targeted on low-power and low-cost SHM applications, proving high effectiveness for the structural assessment of civil and industrial plants. Besides, a cost-benefit analysis is also presented, in which the energy saving brought by SysId running in a sensor-near manner is comprehensively measured against the power consumption due to data transmission, as implied by state-of-the-art communication protocols for IoT. Results demonstrate that SysId is <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1.19\times $ </tex-math></inline-formula> and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$2.78\times $ </tex-math></inline-formula> less energy demanding (with a payload reduction of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$9\times $ </tex-math></inline-formula> and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$45\times $ </tex-math></inline-formula> ) w.r.t. compressed sensing-driven and compression-free solutions, respectively.

Deep Learning Algorithms and Parallel Distributed Computing Techniques for High-Resolution Load Forecasting Applying Hyperparameter Optimization

Electrical load forecasting is one of the critical tasks that helps power utility companies in planning and operation as well as the energy managementsystem (EMS) in controlling and optimizing the power grid’s performance. It guarantees system adequacy and reliability by reducing the gap between supply and demand, and it bolsters cybersecurity by enabling false data detection and system security by predicting system peak load. This article aims to build an accurate long-term load forecasting model based on deep learning (DL) algorithms. Initially, big-data analytics and feature engineering are applied to the power load and weather historical data. Next, three different load forecasting models are developed based on three corresponding deep neural network models (DNN): the simple RNN, the long short term memory (LSTM), and the gated recurrent unit (GRU), respectively. To find an accurate DNN model, a two-step hyperparameter optimization is performed consisting of cross-validation. In the first step, the random search (RS) algorithm is employed to narrow the parameter search space to a smaller local one. In the second step, the grid search (GS) algorithm is used to find the best hyperparameters in the local area located by RS. However, the computational burden of the DNN training process is too high and may lead to out-of-memory (OOM) issues, especially when we are dealing with a big volume of data. To overcome this problem, we developed our Python program based on Tensorflow framework and implemented the framework on a high-performance computer cluster (HPCC). Then, different jobs are defined to employ parallel distributed computing (PDC) capability on the SIU BigDawg HPCC. To this end, all DNN hyperparameters (e.g., number of layers, neurons, dropout, batch size, epoch, learning rate, etc.) can be optimized and result in a comprehensive and accurate load forecasting model. Our proposed method, along with the programming framework, is evaluated on a real-world dataset. Among the three models developed and tested, the GRU, with load and weather as features, has demonstrated excellent performance and outperformed the other two models for long-term forecasting with the best metrics (i.e., MSE, MAPE, <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$R^2$</tex-math></inline-formula> ) and daily error almost less than 1%. Also, our best model (optimized GRU) is compared with the other reported state-of-the-art models such as DBN, SVR, RM-LSTM, DM-GCNN, TCMS-CNN. It is demonstrated that the proposed framework yields a superior performance than the others.

Simulating a Macrosystem of Cargo Deliveries by Road Transport Based on Big Data Volumes: A Case Study of Poland

Simulation models of transport systems are a key tool for solving many problems in the field of management of these systems. The methodologies for creating such models use datasets on both transport infrastructure and demand for the delivery of goods or passenger transport, however, many factors are considered based on assumptions due to the complexity. This article describes the approach to modeling the cargo transportation system for road transport in Poland based on data obtained by the Central Statistical Office from the TD-E survey. This approach avoids many assumptions about demand as the demand parameters are estimated based on a sample representing the general population—a set of all economic entities generating freight traffic. Basic procedures in the developed approach have been implemented as Python scripts. As a result of the use of the proposed methodology, a country-wide road transport model was obtained based on the TD-E survey from 2018. The adequacy of the developed model was assessed based on the results of the General Traffic Measurement from 2015. The obtained model is of satisfactory quality (the coefficient of determination equals 0.62), which can be improved after calibrating the space resistance functions and improving the traffic distribution procedure.

Application of E-Commerce Recommendation Algorithm in Consumer Preference Prediction

Through user characteristic information, user interaction behavior, commodity characteristic information, recommendation engine, and related technologies in data mining, this paper makes a more in-depth study, and analyzes the problems of "big data volume", "cold start" and "data sparsity" in the recommender system in modern business websites. In response to these problems, this paper transforms the problem of large data volume into the problem of large user groups. Then, after using the k-means clustering algorithm to divide the large user group into homogeneous user groups to alleviate the problem, a combination of collaborative filtering algorithm and content-based recommendation algorithm in the homogeneous user group is proposed to alleviate this problem. The experimental precision and recall are both around 0.4, and when W=0.8, the F value is the largest.

WatMIF: Multimodal Medical Image Fusion-Based Watermarking for Telehealth Applications.

Over recent years, the volume of big data has drastically increased for medical applications. Such data are shared by cloud providers for storage and further processing. Medical images contain sensitive information, and these images are shared with healthcare workers, patients, and, in some scenarios, researchers for diagnostic and study purposes. However, the security of these images in the transfer process is extremely important, especially after the COVID-19 pandemic. This paper proposes a secure watermarking algorithm, termed WatMIF, based on multimodal medical image fusion. The proposed algorithm consists of three major parts: the encryption of the host media, the fusion of multimodal medical images, and the embedding and extraction of the fused mark. We encrypt the host media with a key-based encryption scheme. Then, a nonsubsampled contourlet transform (NSCT)-based fusion scheme is employed to fuse the magnetic resonance imaging (MRI) and computed tomography (CT) scan images to generate the fused mark image. Furthermore, the encrypted host media conceals the fused watermark using redundant discrete wavelet transform (RDWT) and randomised singular value decomposition (RSVD). Finally, denoising convolutional neural network (DnCNN) is used to improve the robustness of the WatMIF algorithm. The simulation experiments on two standard datasets were used to evaluate the algorithm in terms of invisibility, robustness, and security. When compared with the existing algorithms, the robustness is improved by 20.14%. Overall, the implementation of proposed watermarking for hiding fused marks and efficient encryption improved the identity verification, invisibility, robustness and security criteria in our WatMIF algorithm.

Sentiment analysis using Hierarchical Multimodal Fusion (HMF)

The rapid rise of platforms like YouTube and Facebook is due to the spread of tablets, smartphones, and other electronic devices. Massive volumes of data are collected every second on such a platform, demanding large-scale data processing. Because these data come in a variety of modalities, including text, audio, and video, sentiment categorization in various modalities and emotional computing are the most researched fields in today's scenario. Companies are striving to make use of this information by developing automated systems for a variety of purposes, such as automated customer feedback collection from user assessments, where the underlying challenge is to mine user sentiment connected to a specific product or service. The use of efficient and effective sentiment analysis tools is required to solve such a complex problem with such a big volume of data. The sentiment analysis of videos is investigated in this study, with data available in three modalities: audio, video, and text. In today's world, modality fusion is a major problem. This study introduces a novel approach to speaker-independent fusion: utilizing deep learning to fuse in a hierarchical fashion. The work tried to obtain improvement over simple concatenation-based fusion.

Physics Informed Self Supervised Learning For Fault Diagnostics and Prognostics in the Context of Sparse and Noisy Data

Sparse &amp; noisy monitoring data leads to numerous challenges in prognostic and health management (PHM). Big data volume but poor quality with scarce healthy states information limits the performance of training machine learning (ML) and physics-based failure modeling. To address these challenges, this thesis aims to develop a new hybrid PHM framework with the ability to autonomously discover and exploit incomplete implicit physics knowledge in sparse &amp; noisy monitoring data, providing a solution for deep physics knowledge-ML fusion by physics-informed machine learning algorithms. In addition, the developed hybrid framework also applies the self-supervised learning paradigm to significantly improve the learning performance under uncertain, sparse, and noisy data with lower requirements for specialist area knowledge. The performance of the developed algorithms will be investigated on the sparse and noise data generated by simulation data sets, public benchmark data sets, and the PHM platform to demonstrate its applicability.

Layered methods for updating AIoT-compatible TCAMS in B5G-enabled WSNs

Classification is a fundamental processing task in advanced network systems. This technique is exploited in 5G/6G wireless sensors networks where flow-based processing of the internet packets is highly demanded by intelligent applications that analyze big volumes of data in a limited time. In this process, the input packets are classified into specific streams by matching to a set of filters. The ternary content-addressable memory (TCAM) is used in hardware implementation of internet packets. However, due to the parallel search capabilities, this memory leads to an increase in the speed and drop of hardware bundles compared to other types of software bundles, but with the increase in the number of rules stored in its layers, the power required for searching, inserting and eliminating increases. Various architectures have been proposed to solve this problem, but none of them has proposed a plan to reduce power consumption while updating the rules in the TCAM memory. In this paper, two algorithms are presented for reducing power consumption during TCAM memory upgrades. The key idea in the proposed algorithms is the reduction in the search range as well as the number of displacements while inserting and deleting rules in TCAM. Implementation and evaluation of proposed methods represent a reduction of more than 50% of the number of visits to TCAM in both proposed algorithms, as well as reducing the update time in the second proposed algorithm compared to the first proposed algorithm which confirms the efficiency of both methods.

Big data based on IoT in the agriculture industry: developments, opportunities, and challenges ahead

With the creation of mobile phones and new technologies, new opportunities have been created in industries; This is while among these technologies are big data and the Internet of Things, which can help increase the potential of the agricultural industry. Meanwhile, technological advances have also changed the way data is collected, shared, and used. Governments and non-governmental actors are increasingly using digital technologies, and new sources and approaches to data have increased the determination of governments and professionals to further integrate data science with agriculture. Our goal is to evaluate the acceptance conditions of big data technologies in agricultural applications, based on the research conducted in this direction based on the growing technologies in this field. In this regard, the literature of the subject has been reviewed and the most important features, opportunities and challenges facing the agricultural industry in the face of the huge amount of data have been reviewed and evaluated. The results show that among the main characteristics of big data (volume, speed, variety, accuracy) in the field of agriculture, the most emphasis is on speed and variety. The main concern of the stakeholders is the cost, user-friendliness and embedding of the solution in their current working procedure.

Study on the interaction between big data and artificial intelligence

AbstractThe explosive growth of information has rapidly ushered people into the era of big data. Due to the large volume, high variety, and rapid velocity characteristics of big data, most traditional data mining methods developed for a centralized data analysis process cannot be applied directly. AI leverages machine intelligence to provide insights, automation, and new methods to interact with data, thereby promoting data literacy throughout the organization. Based on the theory of Big Data Cycle, this paper discusses the relationship between big data and AI and how they interact and influence each other. It adopts the integrative review research method to screen latest literature and summarizes the role of AI in different phases of big data cycle. We also provide an insight into the applications of big data and AI in three different areas, that is, social network, health care, and finance.

A Ring Learning with Errors-Based Ciphertext-Policy Attribute-Based Proxy Re-Encryption Scheme for Secure Big Data Sharing in Cloud Environment.

Owing to the huge volume of big data, users generally use the cloud to store big data. However, because the data are out of the control of users, sensitive data need to be protected. The ciphertext-policy attribute-based encryption scheme can not only effectively control the access of big data, but also decrypt the ciphertext as long as the user's attributes satisfy the access structure of ciphertext, so as to realize one to many big data sharing. When the user's attributes do not satisfy the access structure of ciphertext, the attribute-based proxy re-encryption scheme can be used for big data sharing. The ciphertext-policy attribute-based proxy re-encryption (CP-ABPRE) scheme combines the characteristics of the ciphertext-policy attribute-based encryption scheme and proxy re-encryption scheme. In a CP-ABPRE scheme, on the one hand, the data owner can use the ciphertext-policy attribute-based encryption scheme to encrypt the big data for cloud storage, to realize the access control of the big data. On the other hand, the proxy (cloud service provider) can convert ciphertext under one access structure into ciphertext under another access structure, thus realizing big data sharing between users of different attribute sets. In this article, we modify the existing attribute-based encryption scheme based on Ring Learning With Errors (RLWE), add re-encryption key generation algorithm, re-encryption ciphertext generation algorithm, and re-encryption ciphertext decryption algorithm, and construct CP-ABPRE scheme. In the construction of the re-encryption key, we introduce a random vector and hide the vector in the key by threshold technology. Finally, a CP-ABPRE scheme supporting threshold access structure is constructed based on RLWE. Compared with the existing attribute-based proxy re-encryption schemes, our scheme has smaller public parameters, can encrypt multiple plaintext bits at a time, and can resist selective access structure and chosen plaintext attack, so it is more suitable for big data sharing in cloud environment.

Deep recurrent Gaussian Nesterovs recommendation using multi-agent in social networks.

Due to increasing volume of big data the high volume of information in Social Network put a stop to users from acquiring serviceable information intelligently so many recommendation systems have emerged. Multi-agent Deep Learning gains rapid attraction, and the latest accomplishments address problems with real-world complexity. With big data precise recommendation has yet to be answered. In proposed work Deep Recurrent Gaussian Nesterov’s Optimal Gradient (DR-GNOG) that combines deep learning with a multi-agent scenario for optimal and precise recommendation. The DR-GNOG is split into three layers, an input layer, two hidden layers and an output layer. The tweets obtained from the users are provided to the input layer by the Tweet Accumulator Agent. Then, in the first hidden layer, Tweet Classifier Agent performs optimized and relevant tweet classification by means of Gaussian Nesterov’s Optimal Gradient model. In the second layer, a Deep Recurrent Predictive Recommendation model is designed to concentrate on the vanishing gradient issue arising due to updated tweets obtained from same user at different time instance. Finally, with the aid of hyperbolic activation function in the output layer, building block of the predictive recommendation is obtained. In the experimental study the proposed method is found better than existing GANCF and Bootstrapping method 13–21% in case of recommendation accuracy, 22–32% better in recommendation time and 15–22% better in recall rate.

Big data architecture and data mining analysis for market segment applications of differential global navigation satellite system (GNSS) services: case study of the analysis of the demand for navigation and agriculture

AbstractLocation and navigation services based on global navigation satellite systems (GNSS) are needed for real-time high-precision positioning applications in relevant economic sectors, such as precision agriculture, transport, civil engineering or mapping. Real-time navigation users of GNSS networks have significantly increased all around the world, since the 1990s, and usage has exceeded initial expectations. Therefore, if the evolution of GNSS network users is monitored, the dynamics of market segments can be studied. The implementation of this hypothesis requires the treatment of big volumes of navigation data over several years and the continuous monitoring of customers. This paper is focused on the management of massive connection of GNSS users in an efficient way, in order to obtain analysis and statistics. Big data architecture and data analyses based on data mining algorithms have been implemented as the best way to approach the hypothesis. Results demonstrate the dynamic of users of different market segments, the increasing demand over the years and, specifically, conclusions are obtained about the trends, year-on-year correlation and business volume recovering after economic crisis periods.

The role of data mining techniques and tools in big data management in healthcare field

Data mining is one of the most important modern techniques used to achieve high output standards at all levels. The twenty-first century saw the advent of a new trend to improve medical services in the healthcare sector. To bridge the gap between previous studies and the practical applications of data mining, this study aimed to review the theoretical literature and previous studies related to the demonstration of data mining techniques and tools and their role in big data management. To achieve the objectives of the study, the researchers used a descriptive, analytical, documentary method. The study concluded many results including that in the era of the knowledge and technology revolution, data mining is one of the important issues, that requires everyone to take into account its achievements in our current era, as well as the existence of a correlation between big data and the provision of a separate health service in the field of healthcare, and work to address epidemics and discover vaccines for them. In the healthcare industry, data mining plays a vital role, especially in predicting various types of diseases. In detecting diseases, diagnosis is the main tool. The study recommended the need to conduct more experimental and exploratory studies dealing with healthcare data mining techniques and tools and their effect on the management of big data volumes, especially in our Arab countries and the need for the development of models and action plans and the development of processes and methods from which data in the healthcare sector can be explored.

Optical Interconnects for Future Advanced Antenna Systems: Architectures, Requirements and Technologies

This paper aims to give an outlook of future advanced antenna systems for 5G and 6G wireless networks. The trend to increase the peak data rate and to reduce latency and power consumption will continue in the future. This will be made possible by the exploitation of millimeter wave frequency bands and by the introduction of massive multiple-input multiple-output antenna technology which requires a transformation of the antenna hardware architectures and technologies. Advanced antenna systems based on arrays of several hundreds of antenna elements are under development and to keep the fronthaul throughput manageable, some digital signal processing functions have been moved inside the antenna, creating the need to distribute very big volumes of data at high bit rate across the antenna. The characteristics and hardware architecture of future advanced antenna systems are discussed and the relevant interconnect requirements are presented. A new type of monolithically integrated optical transceiver is presented, integrating electronic and photonic circuits on the same chip with high bandwidth density, high energy efficiency and low latency. The transceiver technology is discussed and results of experimental demonstrations are given.

HiPerMovelets: high-performance movelet extraction for trajectory classification

ABSTRACT In the last decade, trajectory classification has received significant attention. The vast amount of data generated on social media, the use of sensor networks, IOT devices and other Internet-enabled sources allowed the semantic enrichment of mobility data, making the classification task more challenging. Existing trajectory classification methods have mainly considered space, time and numerical data, ignoring the semantic dimensions. Only recently proposed methods as Movelets and MASTERMovelets can handle all types of dimensions. MASTERMovelets is the only method that automatically discovers the best dimension combination and subtrajectory size for trajectory classification. However, although it outperformed the state-of-the-art in terms of accuracy, MASTERMovelets is computationally expensive and results in a high dimensionality problem, which makes it unfeasible for most real trajectory datasets that contain a big volume of data. To overcome this problem and enable the application of the movelets approach on large datasets, in this paper we propose a new high-performance method for extracting movelets and classifying trajectories, called HiPerMovelets (High-performance Movelets). Experimental results show that HiPerMovelets is 10 times faster than MASTERMovelets, reduces the high-dimensionality problem, is more scalable, and presents a high classification accuracy in all evaluated datasets with both raw and semantic trajectories.

Improvement of K-means Cluster Quality by Post Processing Resulted Clusters

A big volume of new data is generated in every moment of the day by different devices and domains as social network, mobile and desktop devices, financial transaction, online websites, different search engines and a lot of smart home devices. The generated data is diversified and can be structured or unstructured. Clustering is the process of categorizing a dataset in groups of records that are similar and are called clusters, the grouping process being performed using a specific criterion. The K-means clustering algorithm is still popular after many years. Different versions of the K-means algorithm emerged along the time and were focused on improving the K-means algorithm by performing some preprocessing steps or by reducing the number of iterations having as a final objective the improvement of the processing time. This paper presents a way of improving the resulted clusters generated by the K-means algorithm by post processing the resulted clusters with a supervised learning algorithm. The proposed approach is focused on improving the quality of the resulting clusters and not on reducing the processing time.

Predictive Models for Emergency Department Triage using Machine Learning: A Review

Background: Recently, many research groups have tried to develop emergency department triage decision support systems based on big volumes of historical clinical data to differentiate and prioritize patients. Machine learning models might improve the predictive capacity of emergency department triage systems. The aim of this review was to assess the performance of recently described machine learning models for patient triage in emergency departments, and to identify future challenges.

Efficient and scalable patients clustering based on medical big data in cloud platform

With the outbreak and popularity of COVID-19 pandemic worldwide, the volume of patients is increasing rapidly all over the world, which brings a big risk and challenge for the maintenance of public healthcare. In this situation, quick integration and analysis of the medical records of patients in a cloud platform are of positive and valuable significance for accurate recognition and scientific diagnosis of the healthy conditions of potential patients. However, due to the big volume of medical data of patients distributed in different platforms (e.g., multiple hospitals), how to integrate these data for patient clustering and analysis in a time-efficient and scalable manner in cloud platform is still a challenging task, while guaranteeing the capability of privacy-preservation. Motivated by this fact, a time-efficient, scalable and privacy-guaranteed patient clustering method in cloud platform is proposed in this work. At last, we demonstrate the competitive advantages of our method via a set of simulated experiments. Experiment results with competitive methods in current research literatures have proved the feasibility of our proposal.