Data Processing Framework Research Articles

RelJoin: Relative-cost-based selection of distributed join methods for query plan optimization

Selecting appropriate distributed join methods for logical join operations in a query plan is crucial for the performance of data-intensive scalable computing (DISC). Different network communication patterns in the data exchange phase generate varying network communication workloads and significantly affect the distributed join performance. However, most cost-based query optimizers focus on the local computing cost and do not precisely model the network communication cost. We propose a cost model for various distributed join methods to optimize join queries in DISC platforms. Our method precisely measures the network and local computing workloads in different execution phases, using information on the size and cardinality statistics of datasets and cluster join parallelism. Our cost model reveals the importance of the relative size of the joining datasets. We implement an efficient distributed join selection strategy, known as RelJoin in SparkSQL, which is an industry-prevalent distributed data processing framework. RelJoin uses runtime adaptive statistics for accurate cost estimation and selects optimal distributed join methods for logical joins to optimize the physical query plan. The evaluation results on the TPC-DS benchmark show that RelJoin performs best in 62 of the 97 queries and can reduce the average query time by 21% compared with other strategies.1

Prediction of Clean Coal Ash Content in Coal Flotation through a Convergent Model Unifying Deep Learning and Likelihood Function, Incorporating Froth Velocity and Reagent Dosage Parameters

This study successfully achieved high-precision detection of the clean coal ash content in the coal froth flotation domain by integrating deep learning with the likelihood function. Methodologically, a novel data processing and prediction framework was established by combining a deep learning Keras neural network with the likelihood function from probability statistics. The SIFT algorithm was utilized to extract key feature points and descriptors from the images, and keypoint matching and mean-shift clustering algorithms were employed to accurately obtain information on foam motion trajectories and velocities. For parameter optimization, the maximum likelihood estimation was applied to find the optimal parameter estimates of the likelihood function, ensuring enhanced model accuracy. By incorporating the optimized likelihood function parameters into the Keras deep neural network, an efficient prediction model was constructed for the dosage of flotation reagents, froth velocity, and clean coal ash content. The model’s evaluation involved six performance metrics. The experimental results were highly significant, with R2 at 0.99997%, RMSE at 0.04458%, MAE at 0.00170%, MAPE at 0.02329%, RRSE at 0.00994%, and MAAPE at 0.00067%.

An optimization in big data time series prediction method by Parzen estimation with Spark

With the development and change of big data related technologies, more and more large amounts of data need to be analyzed. Now there are companies like Google, Yahoo, etc. Frameworks such as MapReduce, Hadoop, Spark, etc. are developed for processing large amounts of data. In this paper, relevant discussions and researches are carried out on time series forecasting under the new era of big data. Now there are time series forecasting methods based on map reduce, Hadoop, spark data processing framework, including nearest neighbor distribution method, neural network method, etc., which have made quite good achievements in big data time series forecasting. By reading the relevant research literature, it is universally acknowledged that the Sparks framework has good application prospects and potential in predicting big data time series. As a result, this paper is mainly aimed at the optimization and improvement of the big data time series forecasting method on the basis of the spark framework. The author noticed that most of the default configurations of spark clusters are generated by default or automatically, rather than the optimal solution obtained after algorithm optimization, so there is still room for improvement in this regard. In this regard, this paper proposes a kernel method for visual data processing of related configurations and parameters, and then optimizes the default data configuration as much as possible to improve the accuracy and feasibility of the big data time series prediction method on the basis of the spark framework. In this paper, the optimized scheme is used to forecast the domestic electricity consumption in the past five years, and the results show that the optimized scheme has a good improvement performance on the basis of the original method.

Parametrized Optimization Based on an Investigation of Musical Similarities Using SPARK and Hadoop

The big data processing framework Spark is used to power a parameterizable recommender system that can make recommendations for music based on a user’s individual tastes and take into account a variety of musical tonal qualities. The system as it is presently built is completely scalable, which means that additional songs can be contributed to the data, the cluster size could be increased, and new types of audio information, in addition to more cutting-edge similarity evaluations, may be included. Another issue discussed in this research paper is the parallel collection of required audio characteristics on a computer cluster. Song recommendations for a dataset including more than 114,000 songs may be created on a Spark cluster with 16 nodes in under 12 s by integrating eight distinct audio feature types and similarity assessments. After the features have been retrieved, they are sent to the Spark-based recommender system to be processed. The calculated distance was displayed, examined, and graphically depicted. By computing the distance depending on the melody, rhythmic, and timbral components of the music, the final software controls song suggestion.

A study of performance comparison of databases for HALF data archiving system

The Hefei Advanced Light Facility (HALF) is a 2.2 GeV fourth-generation synchrotron radiation light source. The HALF control system is a distributed control system based on Experimental Physics and Industrial Control System (EPICS). As the essential part of the HALF control system, the Historical Data Archiving System (HDAS) is responsible to store operational data for the entire facility including the accelerator and beamlines, and provides the functions for data query and analysis. According to the estimation based on the HALF scale, approximately 25,000 EPICS PVs will be stored in HDAS, and these accumulated massive data require a dedicated database for persistent storage and management. Under the EPICS PV data scenario of HALF, a fair database test platform is designed and built to test the read-write performance of databases commonly used in the particle accelerator field. The tested objects include EPICS Archiver Appliance and the five databases MongoDB, HBase, InfluxDB, TimescaleDB, and Cassandra. Additionally, research on the five databases software ecosystem is conducted for the development and extension of HDAS. The test results indicate that TimescaleDB has the fastest read performance, and 1.4×10^6 items of data can be read per second. TimescaleDB also has superior write performance. 1.2 × 105 items of data can be written per second, which can fully meet the writing requirements of HDAS. Regarding the software ecosystem, TimescaleDB is compatible with a larger number of data visualization software and big data processing frameworks. Therefore, TimescaleDB is more suitable as the HDAS database. In the future, a TimescaleDB distributed cluster will be designed and deployed, and an HDAS prototype system will be developed based on this cluster.

Collaborative Hotspot Data Collection with Drones and 5G Edge Computing in Smart City

The construction and governance of smart cities require the collaboration of different systems and different regions. How to realize the monitoring of abnormal hot spots through the collaboration of subsystems with limited resources is related to the stability and efficiency of the city. This work constructs a hot data processing framework for drones and 5G edge computing infrastructure, as well as an Ensemble Multi-Objective Cooperative Learning method to process three different types of hot data. The data collection phase combines set operations with the 0-1 multi-knapsack model, and the cooperative learning phase realizes the degree of cooperation control while retaining the ability of independent optimization of the subsystem. Finally, the advantages of the framework are verified by hot data coverage and collaborative processing efficiency, resource use cost, and balance.

Consistent Matrix: A Feature Selection Framework for Large-Scale Datasets

Large-scale data processing based on limited computing resources has always been a difficult problem in data mining, where feature selection is often used as an effective data compressing mechanism. For granular computing of big data, discernibility matrix and dependency degree are the most representative methods for matrix-based and feature importance degree-based feature selection, respectively. However, their temporal and space complexities are high and often lead to poor performance. In this paper, a novel feature selection framework for large scale data processing with linear complexities was proposed for the first time. Firstly, a much more concise fuzzy granule set, called fuzzy arithmetic covering, was introduced to reduce computational costs. Then a new matrix-based feature selection framework, namely consistent matrix, was proposed for general rough set models. As a result, a heuristic attribute reduction algorithm, i.e., HARCM, was designed accordingly. Compared with six state-of-the-art algorithms for feature selection, the average running time of the newly proposed algorithm was reduced up to 2913 times, with a comparable or even better classification performance.

Integration of Residual Terrain Modelling and the Equivalent Source Layer Method in Gravity Field Synthesis for Airborne Gravity Gradiometer Test Site Determination

To calibrate airborne gravity gradiometers currently in development in China, it is urgent to build an airborne gravity gradiometer test site. The site’s selection depends on the preknowledge of high-resolution gravity and gradient structures. The residual terrain modelling (RTM) technique is generally applied to recover the short-scale gravity field signals. However, due to limitations in the quality and resolution of density models, RTM terrain generally assumes a constant density. This assumption can introduce significant errors in areas with substantial density anomalies and of reggued terrain, such as volcano areas. In this study, we promote a method to determine a high-resolution gravity field by integrating long-wavelength signals generated by EGM2008 with short-wavelength signals from terrain relief and shallow density anomalies. These short wavelength signals are recovered using the RTM technique with both constant density and density anomalies obtained through the equivalent source layer (ESL) method, utilizing sparse terrestrial gravity measurements. Compared to the recovery rate of 54.62% using the classical RTM method, the recovery rate increases to 86.22% after involving density anomalies. With this method, we investigate the gravity field signals over the Wudalianchi Volcano Field (WVF) both on the Earth’s surface and at a flight height of 100 m above the terrain. The contribution of each part and their attenuation characters are studied. In particular, the 5 km × 5 km area surrounding Bijiashan (BJS) and Wohushan (WHS) volcanos shows a strong gravity signature, making it a good candidate for the test site location. This study gives the location of the airborne gravity gradiometer test site which is an essential step in the instruments’ development. Furthermore, the method presented in this study offers a foundational framework for future data processing within the test site.

Iot Data Processing and Scheduling Based on Deep Reinforcement Learning

With the continuous integration of IoT technology and information technology, edge computing, as an emerging computing paradigm, makes full use of terminals to process and analyse real-time data. The explosion of Internet of Things (IoT) devices has created challenges for traditional cloud-based data processing models due to high latency and availability requirements. This paper proposes a new edge computation-based framework for iot data processing and scheduling using deep reinforcement learning. The system architecture incorporates distributed iot data access, realtime processing, and an intelligent scheduler based on Deep q networks (DQN). A large number of experiments show that compared with traditional scheduling methods, the average task completion time is reduced by 20% and resource utilization is increased by 15%. The unique integration of edge computing and deep reinforcement learning provides a flexible and efficient platform for lowlatency iot applications. Key results obtained from testing the proposed system, such as reduced task completion time and increased resource utilization.

Outliers in diffusion-weighted MRI: Exploring detection models and mitigation strategies

Diffusion-weighted MRI (dMRI) is a medical imaging method that can be used to investigate the brain microstructure and structural connections between different brain regions. The method, however, requires relatively complex data processing frameworks and analysis pipelines. Many of these approaches are vulnerable to signal dropout artefacts that can originate from subjects moving their head during the scan.To combat these artefacts and eliminate such outliers, researchers have proposed two approaches: to replace outliers or to downweight outliers during modelling and analysis. With the rising interest in dMRI for clinical research, these types of corrections are increasingly important. Therefore, we set out to investigate the differences between outlier replacement and weighting approaches to help the dMRI community to select the best tool for their data processing pipelines. We evaluated dMRI motion correction registration and single tensor model fit pipelines using Gaussian Process and Spherical Harmonic based replacement approaches and outlier downweighting using highly realistic whole-brain simulations. As a proof of concept, we applied these approaches to dMRI infant data sets that contained varying numbers of dropout artefacts.Based on our results, we concluded that the Gaussian Process based outlier replacement provided similar tensor fit results to Gaussian Process based outlier detection and downweighting. Therefore, if only the least-squares estimate of the single tensor model is of interest, our recommendation is to use outlier replacement. However, outlier downweighting can potentially provide a more accurate estimate of the model precision which could be relevant for applications such as probabilistic tractoraphy.

Genomic Variations Explorer (GenVarX): a toolset for annotating promoter and CNV regions using genotypic and phenotypic differences

The rapid growth of sequencing technology and its increasing popularity in biology-related research over the years has made whole genome re-sequencing (WGRS) data become widely available. A large amount of WGRS data can unlock the knowledge gap between genomics and phenomics through gaining an understanding of the genomic variations that can lead to phenotype changes. These genomic variations are usually comprised of allele and structural changes in DNA, and these changes can affect the regulatory mechanisms causing changes in gene expression and altering the phenotypes of organisms. In this research work, we created the GenVarX toolset, that is backed by transcription factor binding sequence data in promoter regions, the copy number variations data, SNPs and Indels data, and phenotypes data which can potentially provide insights about phenotypic differences and solve compelling questions in plant research. Analytics-wise, we have developed strategies to better utilize the WGRS data and mine the data using efficient data processing scripts, libraries, tools, and frameworks to create the interactive and visualization-enhanced GenVarX toolset that encompasses both promoter regions and copy number variation analysis components. The main capabilities of the GenVarX toolset are to provide easy-to-use interfaces for users to perform queries, visualize data, and interact with the data. Based on different input windows on the user interface, users can provide inputs corresponding to each field and submit the information as a query. The data returned on the results page is usually displayed in a tabular fashion. In addition, interactive figures are also included in the toolset to facilitate the visualization of statistical results or tool outputs. Currently, the GenVarX toolset supports soybean, rice, and Arabidopsis. The researchers can access the soybean GenVarX toolset from SoyKB via https://soykb.org/SoybeanGenVarX/, rice GenVarX toolset, and Arabidopsis GenVarX toolset from KBCommons web portal with links https://kbcommons.org/system/tools/GenVarX/Osativa and https://kbcommons.org/system/tools/GenVarX/Athaliana, respectively.

Construction and Application of Trajectory Data Analysis Model Based on Big Data and Stochastic Gradient Descent Algorithm

This paper studies the model construction of computing and storage resource management system framework based on Hadoop and the implementation of trajectory data analysis function under big data. Relying on the cloud platform infrastructure, in order to support the rapid data growth and massive data processing needs, it provides a mixed storage and analysis platform for structured and unstructured data, and uses big data technology to build a highly scalable and distributed data processing framework. The distributed computation, overall frame model of the memory system, and function module have been built with the aim of constructing the system in consideration. Second, by using Hadoop to preprocess the original data and concentrating on the data hierarchical design model and key technology analysis of big data systems, the design model, functional modules, technological solutions, and SGD algorithm are suggested, along with the detailed implementation procedure. Lastly, by merging the data of running vehicles, the system accomplishes the data analysis of vehicle trajectory, empty and load cars, and load and unload people.

BIG DATA PROCESSING WITH PYTHON IN SOCIAL NETWORKS

Social networks generate a huge amount of data about the behavior, interests, preferences and connections of users, which can be used for various purposes such as marketing, advertising, analytics, research, etc. However, these data have a large volume, variety and update rate, and require special methods and tools for their analysis and processing. Python is a high-level programming language that has many advantages for working with big data, such as simplicity and flexibility of syntax, a rich set of libraries and frameworks for data processing and analysis, support for multithreading and distributed computing, open source code, and a large community of developers and users. The article shows how to use Python to parse and collect data from various sources, to clean and pre-process data, such as removing duplicates, gaps, outliers and noise, normalizing and standardizing data, encoding categorical features, etc., for analysis and data visualization, such as building statistical characteristics, graphs, charts and maps to study the distribution, correlation and dependence of data, for machine learning and data mining. Keywords: Python, big data, social networks, data parsing, data analysis, machine learning.

Repeated Transcranial Photobiomodulation with Light-Emitting Diodes Improves Psychomotor Vigilance and EEG Networks of the Human Brain.

Transcranial photobiomodulation (tPBM) has been suggested as a non-invasive neuromodulation tool. The repetitive administration of light-emitting diode (LED)-based tPBM for several weeks significantly improves human cognition. To understand the electrophysiological effects of LED-tPBM on the human brain, we investigated alterations by repeated tPBM in vigilance performance and brain networks using electroencephalography (EEG) in healthy participants. Active and sham LED-based tPBM were administered to the right forehead of young participants twice a week for four weeks. The participants performed a psychomotor vigilance task (PVT) during each tPBM/sham experiment. A 64-electrode EEG system recorded electrophysiological signals from each participant during the first and last visits in a 4-week study. Topographical maps of the EEG power enhanced by tPBM were statistically compared for the repeated tPBM effect. A new data processing framework combining the group's singular value decomposition (gSVD) with eLORETA was implemented to identify EEG brain networks. The reaction time of the PVT in the tPBM-treated group was significantly improved over four weeks compared to that in the sham group. We observed acute increases in EEG delta and alpha powers during a 10 min LED-tPBM while the participants performed the PVT task. We also found that the theta, beta, and gamma EEG powers significantly increased overall after four weeks of LED-tPBM. Combining gSVD with eLORETA enabled us to identify EEG brain networks and the corresponding network power changes by repeated 4-week tPBM. This study clearly demonstrated that a 4-week prefrontal LED-tPBM can neuromodulate several key EEG networks, implying a possible causal effect between modulated brain networks and improved psychomotor vigilance outcomes.

A coordinates-based hierarchical computing framework towards spatial data processing

The vehicle-based mobile base station (VMBS) has been developed by NTT a few years ago, which plays an important role in reconstructing an emergency communication network by integrating with mobile edge computing to support the cloud service for data pre-processing tasks. Spatial data processing refers to characteristics, such as fast-changing, massive, and potentially infinite. Considering the capacities of network bandwidth and storage of VMBS, new challenging issues have emerged such as how to quantify information loss and how to optimize data processing in the paradigm of mobile edge computing. To address these two challenges, we design a coordinates-based hierarchical computing framework for spatial data processing, named HierCom. Using HierCom, the optimal position of VMBS can be selected to decrease the transmission delay, thereby reducing information loss in the data processing in an edge computing environment. Furthermore, we also propose a near least neighbor-based algorithm to optimize the information loss. Finally, simulations on the real-world dataset show that the information loss of our method is reduced by 27% compared with the baseline method.

A Modular Framework for Data Processing at the Edge: Design and Implementation.

There is a rapid increase in the number of edge devices in IoT solutions, generating vast amounts of data that need to be processed and analyzed efficiently. Traditional cloud-based architectures can face latency, bandwidth, and privacy challenges when dealing with this data flood. There is currently no unified approach to the creation of edge computing solutions. This work addresses this problem by exploring containerization for data processing solutions at the network's edge. The current approach involves creating a specialized application compatible with the device used. Another approach involves using containerization for deployment and monitoring. The heterogeneity of edge environments would greatly benefit from a universal modular platform. Our proposed edge computing-based framework implements a streaming extract, transform, and load pipeline for data processing and analysis using ZeroMQ as the communication backbone and containerization for scalable deployment. Results demonstrate the effectiveness of the proposed framework, making it suitable for time-sensitive IoT applications.

A systematic literature review of Internet of Video Things: Trends, techniques, datasets, and framework

In the last decade, Internet of Things (IoT) and machine learning (ML) have been used for developing semi or fully automated systems of any application. Due to the advancement in networking technologies, the contribution of video-based input is also rising for IoT systems. Hence, Internet of Video Things (IoVT) has been an emerging field to gain more human trust in the systems. IoVT is an integrated attempt of computer vision sensors, networking, video or image processing and information storage systems. In this paper, we have highlighted the literature evidence in the IoVT field. There is a need to analyze it in terms of different aspects, that enables the researchers to understand research trends in the field and lead them to future research directions. Therefore, this article presents a systematic literature review (SLR) in IoVT using various research publications, since its evolution. The study applies article selection criteria on different search methods and compares the relevant published articles. We have analyzed these filtered articles for identifying the study trends, their used techniques & frameworks for data processing and inputs or datasets used for their studies. At the end, various research opportunities and challenges have been framed in the field for future research references.

High-quality images and data augmentation based on inverse projection transformation significantly improve the estimation accuracy of biomass and leaf area index

Estimating above-ground biomass (AGB) and leaf area index (LAI) is crucial for determining crop developmental status and estimating grain yield, thus facilitating high-throughput phenotyping of maize (Zea Mays L.). Recently, unmanned aerial vehicle (UAV) have been widely used to monitor crop growth and estimate LAI and AGB. However, few studies have reported on the quantitative analysis of improved accuracy in biomass and leaf area index estimation using undistorted images obtained through inverse projection transformation algorithms. Additionally, there is limited research on predicting biomass and LAI during various growth stages by replacing traditional data augmentation methods with images captured from different azimuth and zenith angles. The results showed that when using undistorted image at single solar zenith angle of 0°, the accuracy of AGB and LAI estimation was improved by 20.3% and 7.3%, respectively, compared to the models estimated from orthoimages. The undistorted images with various azimuth and zenith angles were 9–15 times larger than the orthomosaic image dataset, and could be used as a new training data that is similar to the original data but with variations. The AGB and LAI estimation model constructed using the data augmentation method achieved better performance with higher average R2 (0.98). Model performance was increased by 40% and 16.7% for estimating AGB and by 19.3% and 11.3% for estimating LAI based on undistorted images, respectively, in comparison to models based on orthophotos and undistorted images due to the increased diversity and quantity of training data. Canopy structural including canopy coverage and plant height, characterized by 96.9%, was found to be the most reliable indication for estimating AGB and LAI, while other RGB sensor-derived feature combinations contributed only 3.1% to the estimation. Spectral features took the second place, and the textural features was the weakest. In summary, the data processing framework using data augmentation based on various azimuth and zenith angles' images can improve the performance of crop growth estimation and provide a mechanism for precision agriculture under field conditions.

Field-driven design and performance evaluation of dual functionally graded triply periodic minimal surface structures for additive manufacturing

Triply periodic minimal surface (TPMS) structures prepared by additive manufacturing (AM) have been widely used in aerospace and thermal management systems due to their functional advantages such as high specific strength and excellent heat transfer coefficient. However, lattices with uniform or simple gradients cannot match the regional diversity needs of functional advantages in service. This study proposed a field-driven AM data processing framework for dual functionally graded TPMS lattices. For unified data processing, mathematical functions, simulated outcomes, and geometric models were transformed into fields. I-WP and Diamond units were selected as the primitive and filling structures of Ti-6Al-4V lattices, respectively. According to the functional requirements, the gradient distribution design guided by the simulated stress field is completed through the proposed framework. The results illustrate that the field-driven dual graded lattices present superior compressive mechanical response. At almost equal volume fractions, their compressive strength is approximately 100% more than that of primitive lattices and 69% greater than that of dual uniform lattices. Furthermore, the dual lattices have larger specific surface areas than the primitive lattices, making them better at heat insulation and heat dissipation. This field-driven digital framework will show enormous potential in the matching design of functionally graded lattices.

Analysis of Cyber-Intelligence Frameworks for AI Data Processing

This paper deals with the concept of cyber intelligence and its components as a fundamental tool for the protection of information today. After that, the main cyber-intelligence frameworks that are currently applied worldwide (Diamond Model, Cyberkill Chain, and Mitre Att&ck) are described to subsequently analyse them through their practical application in a real critical cyber incident, as well as analyse the strengths and weaknesses of each one of them according to the comparison of seventeen variables of interest. From this analysis and considering the two actions mentioned, it is concluded that Mitre Att&ck is the most suitable framework due to its flexibility, permanent updating, and the existence of a powerful database. Finally, an explanation is given for how Mitre Att&ck can be integrated with the research and application of artificial intelligence in the achievement of the objectives set and the development of tools that can serve as support for the detection of the patterns and authorship of cyberattacks.