Efficient Big Data Processing Research Articles

LEVERAGING DATA-DRIVEN TECHNIQUES FOR EFFICIENT DATA MINING IN CLOUD COMPUTING ENVIRONMENTS

The capacity to efficiently use big data and analytics is becoming a critical differentiator for company growth in today's data-driven environment. Using important trends, obstacles, and best practices as a framework, this article investigates how to promote company growth via the use of big data and analytics. An important issue in cloud computing is deciding on an acceptable amount and location of data. Decisions about resource management are based on data aspects and operations in data-driven infrastructure management (DDIM), a novel solution to this problem. It is critical to have a unified system that can manage various forms of big data and the analysis of that data, as well as common knowledge management functions. The approach stated in this research is DD-DM-CCE, or Data-Driven Methods for Efficient Data Mining in Cloud Computing Environments. Improving data using derived information from maximum frequent correlated pattern mining is the main focus of the work. By considering the centrality factor, the DD-DM-CCE method may help choose the best locations to store data in order to reduce access latency. In order to gain a competitive edge, this study offers a cloud-based conceptual framework that can analyze large data in real time and improve decision making. Efficient big data processing is possible with cloud computing infrastructures that can store and analyze massive amounts of data, as this reduces the upfront cost of the massively parallel computer infrastructure needed for big data analytics. According to simulations run on cloud computing, the DD-DM-CCE approach does better than the status quo regarding hit ratio, effective network utilization, and average response time. According to this study, data mining methods are valuable and successful in predicting how consumers will utilize cloud services.

Quality of service management in telecommunication network using machine learning technique

<span>Designing and implementing a fail-safe, real-time telecommunications network is complex. In modern networks, traditional quality of service (QoS) methods for monitoring and analyzing data have some problems, such as accuracy and efficient processing of big data in real time. To solve this problem, should use an appropriate intelligent crash classification system to detect and diagnose run-time errors. The article proposes to use a comprehensive fault detection system that includes QoS and machine learning technologies using information about the state of a point-to-point protocol over ethernet (PPPoE) session on PPPoE active discovery termination (PADT) virtual local area network (VLAN) routes. </span><span>This intelligent system is built using the machine learning method and is independent of the main real-time system. Demonstrated the operation of seven machine learning algorithms and presented the results of training and fault detection. Based on the received information about the state of the PPPoE session, the PADT type allows you to control the behavior of the real-time system.</span>

Survey of Distributed Computing Frameworks for Supporting Big Data Analysis

Distributed computing frameworks are the fundamental component of distributed computing systems. They provide an essential way to support the efficient processing of big data on clusters or cloud. The size of big data increases at a pace that is faster than the increase in the big data processing capacity of clusters. Thus, distributed computing frameworks based on the MapReduce computing model are not adequate to support big data analysis tasks which often require running complex analytical algorithms on extremely big data sets in terabytes. In performing such tasks, these frameworks face three challenges: computational inefficiency due to high I/O and communication costs, non-scalability to big data due to memory limit, and limited analytical algorithms because many serial algorithms cannot be implemented in the MapReduce programming model. New distributed computing frameworks need to be developed to conquer these challenges. In this paper, we review MapReduce-type distributed computing frameworks that are currently used in handling big data and discuss their problems when conducting big data analysis. In addition, we present a non-MapReduce distributed computing framework that has the potential to overcome big data analysis challenges.

Estimating hidden fishing activity hotspots from vessel transmitted data

Monitoring fishery activity is essential for resource planning and guaranteeing fisheries sustainability. Large fishing vessels constantly and continuously communicate their positions via Automatic Identification System (AIS) or Vessel Monitoring Systems (VMSs). These systems can use radio or Global Positioning System (GPS) devices to transmit data. Processing and integrating these big data with other fisheries data allows for exploring the relations between socio-economic and ecosystem assets in marine areas, which is fundamental in fishery monitoring. In this context, estimating actual fishing activity from time series of AIS and VMS data would enhance the correct identification of fishing activity patterns and help assess regulations' effectiveness. However, these data might contain gaps because of technical issues such as limited coverage of the terrestrial receivers or saturated transmission bands. Other sources of data gaps are adverse meteorological conditions and voluntary switch-offs. Gaps may also include hidden (unreported) fishing activity whose quantification would improve actual fishing activity estimation. This paper presents a workflow for AIS/VMS big-data analysis that estimates potential unreported fishing activity hotspots in a marine area. The workflow uses a statistical spatial analysis over vessel speeds and coordinates and a multi-source data integration approach that can work on multiple areas and multiple analysis scales. Specifically, it (i) estimates fishing activity locations and rebuilds data gaps, (ii) estimates the potential unreported fishing hour distribution and the unreported-over-total ratio of fishing hours at a 0.01° spatial resolution, (iii) identifies potential unreported fishing activity hotspots, (iv) extracts the stocks involved in these hotspots (using global-scale repositories of stock and species observation data) and raises an alert about their possible endangered, threatened, and protected (ETP) status. The workflow is also a free-to-use Web Service running on an open science-compliant cloud computing platform with a Web Processing Service (WPS) standard interface, allowing efficient big data processing. As a study case, we focussed on the Adriatic Sea. We reconstructed the monthly reported and potential unreported trawling activity in 2019, using terrestrial AIS data with a 5-min sampling period, containing ~50 million records transmitted by ~1,600 vessels. The results highlight that the unreported fishing activity hotspots especially impacted Italian coasts and some forbidden and protected areas. The potential unreported activity involved 33 stocks, four of which were ETP species in the basin. The extracted information agreed with expert studies, and the estimated trawling patterns agreed with those produced by the Global Fishing Watch.

Large-scale quantification of groundwater recharge threshold conditions using machine learning classifications: An attempt over the Australian continent

The large-scale quantification of groundwater recharge threshold conditions is a bottleneck in the field of research, considering that groundwater recharge processes can be quite complex and affected by different factors in different regions. Traditional statistical approaches have either focused on local to regional scales or are limited to a single factor such as rainfall, resulting in a lack of a comprehensive understanding of groundwater recharge mechanisms on large scales. Considering that finding a threshold to instantly determine whether the groundwater is in the state of recharge or discharge is a classification task involving multiple factors, machine learning techniques with self-learning capabilities, efficient processing of big data, and mining relationships between multiple factors is, thus, applied to this study. In this study, three traditional machine learning classification techniques are explored; Classification and Regression Tree (CART), Random Forest (RF), and Logistical Regression (LR), to quantify specific threshold values for each condition (including rainfall, evaporation, soil moisture, runoff and vegetation) since they are more interpretable compared to advanced machine learning techniques such as neural networks. Using the Australian continent as an example, CART is the fastest method and provides an average classification accuracy (76%) at almost the same levels compared to RF (77–78%) and LR (79–80%). Meanwhile, CART is the only method to rank the importance of conditions while providing a specific threshold for each condition. When inferring the groundwater recharge mechanism through threshold conditions, the CART method suggests that a primary (the most important) threshold condition is sufficient for most parts of the Australian continent, e.g., rainfall as the primary threshold condition in northern Australia indicates a direct recharge mechanism. Only the coastal areas of southeastern and southwestern Australia are dominated by multiple threshold conditions, showing the complex groundwater recharge mechanism. Overall, the significance of this work is to provide the spatial maps of threshold conditions that mainly control groundwater recharge and their specific threshold values, which are crucial for managers to carry out a good plan over a large area. More importantly, this study demonstrates the potential of traditional machine learning methods to reveal groundwater recharge mechanisms, which may help further refine the development of groundwater models in the future.

USING A TENSOR MODEL TO HANDLE UNCERTAINTY IN COMPLEX DYNAMICAL SYSTEMS

The article is devoted to research related to the processing of uncertainty by means of tensor algebra in complex dynamical systems. The “smart electronic hitch” system is considered as an example of a complex dynamic system. The use of such a system is especially important when organizing traffic in difficult conditions. The solution of this problem occurs under conditions of uncertainty that may appear at different levels of the traffic management process. To identify and study in detail the properties of uncertainty, the author suggests using a tensor model. The tensor model makes it possible to identify additional properties of uncertainty, the study of which is not available when using traditional formalisms to represent uncertainty. Using the tensor model allows us to study the spatial model of uncertainty, real and imaginary values of uncertainty, as well as uncertainty invariants with respect to various transformations of the coordinate system. The article proposes a classification of uncertainty in a complex system. Using the example of the organization of interaction of “smart” controllers in an electronic coupling, the author shows the results of applying tensor analysis methods of networks to obtain a computational base of an electronic coupling. Tensor equations provide efficient processing of big data, obtaining information in real time, the stability of the dynamic system to changes in the topology of the connection of controllers and changes in the soft and hard components of the connection. The results obtained in the article show that the tensor uncertainty model can be successfully implemented in a dynamic system of any complexity level.

Третья платформа информации и Big Data

Despite the fact that the introduction of Big Data in many areas of life is an inevitable reality of the near future, there are difficulties with the classification of information, the allocation of knowledge, storage, security and scalability issues. The paper describes the concept of the third platform of informatization and Big Data, which play an important role in this process, is described. The problems associated with the need for efficient processing and analysis of Big Data are posed. The spheres of life in which new technologies are being actively introduced and developed are shown and considered. The purpose of the study is to discuss the trends in the implementation of Big Data within the framework of the third informatization platform. The problems of Big Data and some ways to solve them are investigated. The interrelations and patterns in the field of Big Data and their impact on the development of information technologies are considered. Research methodology: analytical research methods based on the analysis of data and concepts of the third informatization platform are used. The results of the study: a deep analysis was carried out for full immersion in the issue of the third platform of informatization and the concept of Big Data. The advantages of introducing technology components into the spheres of life are shown, as well as the problems of Big Data and some ways of their solution are studied. The main theoretical results of the study are presented, confirming the interrelationships and patterns between the elements of developing information technologies. The practical significance of the third informatization platform and its components is discussed. Recommendations related to the application of these technologies in real conditions are proposed, as well as their mutual influence of technologies to achieve more effective results.

Bitcoin and Cryptocurrency Exchange Market Prediction and Analysis Using Big Data and Machine Learning Algorithms

Due to economic uncertainty and the financial crisis of 2008, a desire for an unregulated currency arose, leading to the invention of Bitcoin. Using a pseudonym called Satoshi Nakamoto, Bitcoin was created in 2009, anonymously or by a group of unknown individuals. Since Bitcoin has been the most valuable cryptocurrency in recent years, its prices have fluctuated dramatically, making it difficult to predict their prices. Investors, businesses, risk managers, and market analysts can all benefit from being able to predict Bitcoin prices. By using the Bitcoin transaction data obtained from the Bitstamp website in this study, several different Machine Learning models are employed to determine the most accurate model for predicting Bitcoin prices. These models are based on 1-minute interval exchange rates in USD from January 1, 2012, to January 8, 2022. Analysis was performed primarily with Python, but it was also used and Hadoop, a distributed data storage and processing framework that uses the map-reduce programming model to allow efficient parallel processing of Big Data. Based on the results of our research, comprising three experiments, autoregres-sive-integrated moving average (ARIMA) makes the most accurate prediction of Bitcoin prices, with a 95.98% success rate.

A Learning-Rate Modulable and Reliable TiOx Memristor Array for Robust, Fast, and Accurate Neuromorphic Computing.

Realization of memristor‐based neuromorphic hardware system is important to achieve energy efficient bigdata processing and artificial intelligence in integrated device system‐level. In this sense, uniform and reliable titanium oxide (TiO x ) memristor array devices are fabricated to be utilized as constituent device element in hardware neural network, representing passive matrix array structure enabling vector‐matrix multiplication process between multisignal and trained synaptic weight. In particular, in situ convolutional neural network hardware system is designed and implemented using a multiple 25 × 25 TiO x memristor arrays and the memristor device parameters are developed to bring global constant voltage programming scheme for entire cells in crossbar array without any voltage tuning peripheral circuit such as transistor. Moreover, the learning rate modulation during in situ hardware training process is successfully achieved due to superior TiO x memristor performance such as threshold uniformity (≈2.7%), device yield (> 99%), repetitive stability (≈3000 spikes), low asymmetry value of ≈1.43, ambient stability (6 months), and nonlinear pulse response. The learning rate modulable fast‐converging in situ training based on direct memristor operation shows five times less training iterations and reduces training energy compared to the conventional hardware in situ training at ≈95.2% of classification accuracy.

Research on Improved Shallow Neural Network Big Data Processing Model based on Gaussian Function

The application of the current new generation communication technology is gradually diversified, and the global Internet users are increasing, leading some large enterprises to increasingly rely on faster and more efficient big data processing technology. In order to solve the shortcomings of the current big data processing algorithms, such as slow computing speed, computing accuracy to be improved, and poor online real-time learning ability, this research combines incremental learning and sliding window ideas to design two improved radial basis function (RBF) neural network algorithms with Gaussian function as the kernel function. The Duffing equation example and the data of "Top 100 single products for Taobao search glasses sales" were used to verify the performance of the design algorithm. The experimental results of Duffing equation example show that when the total sample is 100000, the mean square errors of IOL, SWOL, SVM and ResNet50 algorithms are 1.86e-07, 1.59e-07, 3.37e-07 and 2.67e-07 respectively. The experimental results of the data set of "Top 100 SKUs for Taobao Search Glasses Sales" show that when the number of samples in the test set is 800, the root mean square errors of IOL, SWOL, SVM and ResNet50 algorithms are 0.0060, 0.0056, 0.0069 and 0.0073 respectively. This shows that the RBF online learning algorithm designed in this study, which integrates sliding windows, has a stronger comprehensive ability to process big data, and has certain application value for improving the accuracy of online data based commodity recommendation in e-commerce and other industries.

A Review on Big Data Stream Processing Applications: Contributions, Benefits, and Limitations

The amount of data in our world has been rapidly keep growing from time to time. Â In the era of big data, the efficient processing and analysis of big data using machine learning algorithm is highly required, especially when the data comes in form of streams. There is no doubt that big data has become an important source of information and knowledge in making decision process. Nevertheless, dealing with this kind of data comes with great difficulties; thus, several techniques have been used in analyzing the data in the form of streams. Many techniques have been proposed and studied to handle big data and give decisions based on off-line batch analysis. Today, we need to make a constructive decision based on online streaming data analysis. Many researchers in recent years proposed some different kind of frameworks for processing the big data streaming. In this work, we explore and present in detail some of the recent achievements in big data streaming in term of contributions, benefits, and limitations. As well as some of recent platforms suitable to be used for big data streaming analytics. Moreover, we also highlight several issues that will be faced in big data stream processing. In conclusion, it is hoped that this study will assist the researchers in choosing the best and suitable framework for big data streaming projects.

I4sea: a big data platform for sea area monitoring and analysis of fishing vessels activity

ABSTRACT The i4sea research project provides effective and efficient big data integration, processing, and analysis technologies to deliver both real-time and historical operational snapshots of fishing vessels activity in national sea areas. This paper presents the architecture of the i4sea big data platform for sea area monitoring and analysis of fishing vessels activity and demonstrates the operation of some use-case pilot scenarios.

Employing Vertical Elasticity for Efficient Big Data Processing in Container-Based Cloud Environments

Recently, “Big Data” platform technologies have become crucial for distributed processing of diverse unstructured or semi-structured data as the amount of data generated increases rapidly. In order to effectively manage these Big Data, Cloud Computing has been playing an important role by providing scalable data storage and computing resources for competitive and economical Big Data processing. Accordingly, server virtualization technologies that are the cornerstone of Cloud Computing have attracted a lot of research interests. However, conventional hypervisor-based virtualization can cause performance degradation problems due to its heavily loaded guest operating systems and rigid resource allocations. On the other hand, container-based virtualization technology can provide the same level of service faster with a lightweight capacity by effectively eliminating the guest OS layers. In addition, container-based virtualization enables efficient cloud resource management by dynamically adjusting the allocated computing resources (e.g., CPU and memory) during the runtime through “Vertical Elasticity”. In this paper, we present our practice and experience of employing an adaptive resource utilization scheme for Big Data workloads in container-based cloud environments by leveraging the vertical elasticity of Docker, a representative container-based virtualization technique. We perform extensive experiments running several Big Data workloads on representative Big Data platforms: Apache Hadoop and Spark. During the workload executions, our adaptive resource utilization scheme periodically monitors the resource usage patterns of running containers and dynamically adjusts allocated computing resources that could result in substantial improvements in the overall system throughput.

An Incremental Tensor-Train Decomposition for Cyber-Physical-Social Big Data

Cyber-physical-social big data generated from ubiquitous devices and diverse spaces generally are multi-source, heterogeneous, and deeply intertwined. To efficiently analyze and handle the ubiquitous cyber-physical-social big data, tensor is considered as an effective tool, but the curse of dimensionality is still the main bottleneck of tensor-based big data analysis. Tensor networks can considerably alleviate or overcome it through the tensor approximate theory. Therefore, this paper focuses on developing an efficient big data processing framework based on tensor networks and providing an incremental tensor train decomposition approach for the streaming big data. Concretely, this paper first presents a hierarchical cyber-physical-social big data processing framework composed of three planes, namely, data representation and decomposition, data storage and processing, and data analysis and service, in which tensor train (TT) and quantized TT decompositions are particularly introduced to remarkably overcome the curse of dimensionality. Besides, to efficiently handle the continuous streaming big data and avoid the repeated decomposition for the history data, an incremental tensor train decomposition (ITTD) approach is proposed and the complexities are further analyzed in detail. Experimental results demonstrate that ITTD demonstrably outperforms the nonincremental TT decomposition in execution time on the precise of guaranteeing the nearly equal approximation error.

Beyond 5G: Big Data Processing for Better Spectrum Utilization

This article emphasizes the great potential of big data processing for advanced user- and situation-oriented, so context-aware resource utilization in future wireless networks. In particular, we consider the application of dedicated, detailed, and rich-in-content maps and records called Radio Service Maps, (RSM) for unlocking the spectrum opportunities in 6G networks. Due to the characteristics of 5G, in the future, there will be a need for high convergence of various types of wireless networks, such as cellular and the Internet-of-Things (IoT) networks, which are steadily growing and consequently considered as the studied use case in this work. We show that the 6G network significantly benefits from effective Dynamic Spectrum management (DSM) based on RSM which provides rich and accurate knowledge of the radio context; a knowledge that is stored and processed within database-oriented subsystems designed to support wireless networks for improving spectral efficiency. In this article, we discuss context-aware RSM subsystem architecture and operation for DSM in convergent 6G radio and IoT networks. By providing various use-cases, we demonstrate that the accurate definition and access to the rich context information lead to a significant improvement of the system performance. In consequence, we also claim that efficient big-data processing algorithms will be necessary for future applications.

Study of Systematic Bias in Measuring Surface Deformation With SAR Interferometry

This article investigates the presence of a new interferometric signal in multilooked synthetic aperture radar (SAR) interferograms that cannot be attributed to the atmospheric or Earth-surface topography changes. The observed signal is short-lived and decays with the temporal baseline; however, it is distinct from the stochastic noise attributed to temporal decorrelation. The presence of such a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">fading signal</i> introduces a systematic phase component, particularly in short temporal baseline interferograms. If unattended, it biases the estimation of Earth surface deformation from SAR time series. Here, the contribution of the mentioned phase component is quantitatively assessed. The biasing impact on the deformation-signal retrieval is further evaluated. A quality measure is introduced to allow the prediction of the associated error with the fading signals. Moreover, a practical solution for the mitigation of this physical signal is discussed; special attention is paid to the efficient processing of Big Data from modern SAR missions such as Sentinel-1 and NISAR. Adopting the proposed solution, the deformation bias is shown to decrease significantly. Based on these analyses, we put forward our recommendations for efficient and accurate deformation-signal retrieval from large stacks of multilooked interferograms.

A decentralised framework for efficient storage and processing of big data using HDFS and IPFS

Big data revolution emerged with greater opportunities as well as challenges. Some of the major challenges include capturing, storing, transferring, analysing, processing and updating these large and complex datasets. Traditional data handling techniques cannot manage this fast growing data. Apache Hadoop is one of the best technologies which can address the challenges involved in big data handling. Hadoop is a centralised, distributed data storage model. InterPlanetary file system (IPFS) is an emerging technology which can provide a decentralised distributed storage. By integrating both these technologies, we can create a better framework for the distributed storage and processing of big data. In the proposed work, we formulated a model for big data placement, replication and processing by combining the features of Hadoop and IPFS. Hadoop distributed file system and IPFS jointly handle the data placement and replication tasks and the programming framework MapReduce in Hadoop handle the data processing task. The experimental result shows that the proposed framework can achieve cost-effective storage as well as faster processing of big data.

A decentralised framework for efficient storage and processing of big data using HDFS and IPFS

Big data revolution emerged with greater opportunities as well as challenges. Some of the major challenges include capturing, storing, transferring, analysing, processing and updating these large and complex datasets. Traditional data handling techniques cannot manage this fast growing data. Apache Hadoop is one of the best technologies which can address the challenges involved in big data handling. Hadoop is a centralised, distributed data storage model. InterPlanetary file system (IPFS) is an emerging technology which can provide a decentralised distributed storage. By integrating both these technologies, we can create a better framework for the distributed storage and processing of big data. In the proposed work, we formulated a model for big data placement, replication and processing by combining the features of Hadoop and IPFS. Hadoop distributed file system and IPFS jointly handle the data placement and replication tasks and the programming framework MapReduce in Hadoop handle the data processing task. The experimental result shows that the proposed framework can achieve cost-effective storage as well as faster processing of big data.

Cloud‐based difference algorithm using big GPR data for roadbed damage detection

SummaryGround‐penetrating radar (GPR) technology is being widely used for urban roadbed damage detection. In this paper, we present the development of a cloud‐based difference detection method using big GPR data for roadbed damage detection. Unlike many processing methods relying on standalone servers, the proposed method can achieve high accuracy and processing efficiency by distributing the computation over a server cluster. The method includes a difference detection algorithm for GPR image enhancement and damage extraction, the Hadoop Distributed File System, and MapReduce distributed parallel computing framework for big GPR data parallel processing and interpretation. Moreover, to automatize roadbed damage classification, a criterion with seven grades is established based on the wave group shape, amplitude, phase, and attenuation characteristics. We validate the proposed method through a detection experiment that was conducted on the Fourth Ring Road in Beijing, China. The detection revealed 187 damage from the efficient processing of big GPR data, thus suggesting the ability of the proposed method to provide timely support for road safety in cities.

TCQG-Software-Defined Transmission Control Scheme in 5G Networks from Queuing Game Perspective.

The efficient processing and forwarding of big data is one of the key problems and challenges facing the next generation wireless communication network. Using a software definition method to virtualize the network can improve the efficiency of network operation and reduce the cost of network operation and maintenance. A software-defined transmission control scheme was presented to solve the excessive controller flow problem for 5G networks. Based on the queuing game theory, a system model was built due to the competition among the requests of the switch. The transmission control platform was in charge of resource allocation. It got maximum social welfare under a profit-maximizing fee. In this model, the optimal queue length was calculated and discussed in a first-come-first-served and last-come-first-served with preemption discipline. The optimal queue length was obtained and the optimal admission fee was calculated. Then, the single switch single controller transmission control model was extended to the multi-switches single controller model. As a result, the social welfare of the system containing the controller’s profit and switch surplus reaches the maximum.