Data Processing Framework Research Articles

Collaborative Hotspot Data Collection with Drones and 5G Edge Computing in Smart City

The construction and governance of smart cities require the collaboration of different systems and different regions. How to realize the monitoring of abnormal hot spots through the collaboration of subsystems with limited resources is related to the stability and efficiency of the city. This work constructs a hot data processing framework for drones and 5G edge computing infrastructure, as well as an Ensemble Multi-Objective Cooperative Learning method to process three different types of hot data. The data collection phase combines set operations with the 0-1 multi-knapsack model, and the cooperative learning phase realizes the degree of cooperation control while retaining the ability of independent optimization of the subsystem. Finally, the advantages of the framework are verified by hot data coverage and collaborative processing efficiency, resource use cost, and balance.

Consistent Matrix: A Feature Selection Framework for Large-Scale Datasets

Large-scale data processing based on limited computing resources has always been a difficult problem in data mining, where feature selection is often used as an effective data compressing mechanism. For granular computing of big data, discernibility matrix and dependency degree are the most representative methods for matrix-based and feature importance degree-based feature selection, respectively. However, their temporal and space complexities are high and often lead to poor performance. In this paper, a novel feature selection framework for large scale data processing with linear complexities was proposed for the first time. Firstly, a much more concise fuzzy granule set, called fuzzy arithmetic covering, was introduced to reduce computational costs. Then a new matrix-based feature selection framework, namely consistent matrix, was proposed for general rough set models. As a result, a heuristic attribute reduction algorithm, i.e., HARCM, was designed accordingly. Compared with six state-of-the-art algorithms for feature selection, the average running time of the newly proposed algorithm was reduced up to 2913 times, with a comparable or even better classification performance.

Integration of Residual Terrain Modelling and the Equivalent Source Layer Method in Gravity Field Synthesis for Airborne Gravity Gradiometer Test Site Determination

To calibrate airborne gravity gradiometers currently in development in China, it is urgent to build an airborne gravity gradiometer test site. The site’s selection depends on the preknowledge of high-resolution gravity and gradient structures. The residual terrain modelling (RTM) technique is generally applied to recover the short-scale gravity field signals. However, due to limitations in the quality and resolution of density models, RTM terrain generally assumes a constant density. This assumption can introduce significant errors in areas with substantial density anomalies and of reggued terrain, such as volcano areas. In this study, we promote a method to determine a high-resolution gravity field by integrating long-wavelength signals generated by EGM2008 with short-wavelength signals from terrain relief and shallow density anomalies. These short wavelength signals are recovered using the RTM technique with both constant density and density anomalies obtained through the equivalent source layer (ESL) method, utilizing sparse terrestrial gravity measurements. Compared to the recovery rate of 54.62% using the classical RTM method, the recovery rate increases to 86.22% after involving density anomalies. With this method, we investigate the gravity field signals over the Wudalianchi Volcano Field (WVF) both on the Earth’s surface and at a flight height of 100 m above the terrain. The contribution of each part and their attenuation characters are studied. In particular, the 5 km × 5 km area surrounding Bijiashan (BJS) and Wohushan (WHS) volcanos shows a strong gravity signature, making it a good candidate for the test site location. This study gives the location of the airborne gravity gradiometer test site which is an essential step in the instruments’ development. Furthermore, the method presented in this study offers a foundational framework for future data processing within the test site.

Iot Data Processing and Scheduling Based on Deep Reinforcement Learning

With the continuous integration of IoT technology and information technology, edge computing, as an emerging computing paradigm, makes full use of terminals to process and analyse real-time data. The explosion of Internet of Things (IoT) devices has created challenges for traditional cloud-based data processing models due to high latency and availability requirements. This paper proposes a new edge computation-based framework for iot data processing and scheduling using deep reinforcement learning. The system architecture incorporates distributed iot data access, realtime processing, and an intelligent scheduler based on Deep q networks (DQN). A large number of experiments show that compared with traditional scheduling methods, the average task completion time is reduced by 20% and resource utilization is increased by 15%. The unique integration of edge computing and deep reinforcement learning provides a flexible and efficient platform for lowlatency iot applications. Key results obtained from testing the proposed system, such as reduced task completion time and increased resource utilization.

Outliers in diffusion-weighted MRI: Exploring detection models and mitigation strategies

Diffusion-weighted MRI (dMRI) is a medical imaging method that can be used to investigate the brain microstructure and structural connections between different brain regions. The method, however, requires relatively complex data processing frameworks and analysis pipelines. Many of these approaches are vulnerable to signal dropout artefacts that can originate from subjects moving their head during the scan.To combat these artefacts and eliminate such outliers, researchers have proposed two approaches: to replace outliers or to downweight outliers during modelling and analysis. With the rising interest in dMRI for clinical research, these types of corrections are increasingly important. Therefore, we set out to investigate the differences between outlier replacement and weighting approaches to help the dMRI community to select the best tool for their data processing pipelines. We evaluated dMRI motion correction registration and single tensor model fit pipelines using Gaussian Process and Spherical Harmonic based replacement approaches and outlier downweighting using highly realistic whole-brain simulations. As a proof of concept, we applied these approaches to dMRI infant data sets that contained varying numbers of dropout artefacts.Based on our results, we concluded that the Gaussian Process based outlier replacement provided similar tensor fit results to Gaussian Process based outlier detection and downweighting. Therefore, if only the least-squares estimate of the single tensor model is of interest, our recommendation is to use outlier replacement. However, outlier downweighting can potentially provide a more accurate estimate of the model precision which could be relevant for applications such as probabilistic tractoraphy.

Genomic Variations Explorer (GenVarX): a toolset for annotating promoter and CNV regions using genotypic and phenotypic differences

The rapid growth of sequencing technology and its increasing popularity in biology-related research over the years has made whole genome re-sequencing (WGRS) data become widely available. A large amount of WGRS data can unlock the knowledge gap between genomics and phenomics through gaining an understanding of the genomic variations that can lead to phenotype changes. These genomic variations are usually comprised of allele and structural changes in DNA, and these changes can affect the regulatory mechanisms causing changes in gene expression and altering the phenotypes of organisms. In this research work, we created the GenVarX toolset, that is backed by transcription factor binding sequence data in promoter regions, the copy number variations data, SNPs and Indels data, and phenotypes data which can potentially provide insights about phenotypic differences and solve compelling questions in plant research. Analytics-wise, we have developed strategies to better utilize the WGRS data and mine the data using efficient data processing scripts, libraries, tools, and frameworks to create the interactive and visualization-enhanced GenVarX toolset that encompasses both promoter regions and copy number variation analysis components. The main capabilities of the GenVarX toolset are to provide easy-to-use interfaces for users to perform queries, visualize data, and interact with the data. Based on different input windows on the user interface, users can provide inputs corresponding to each field and submit the information as a query. The data returned on the results page is usually displayed in a tabular fashion. In addition, interactive figures are also included in the toolset to facilitate the visualization of statistical results or tool outputs. Currently, the GenVarX toolset supports soybean, rice, and Arabidopsis. The researchers can access the soybean GenVarX toolset from SoyKB via https://soykb.org/SoybeanGenVarX/, rice GenVarX toolset, and Arabidopsis GenVarX toolset from KBCommons web portal with links https://kbcommons.org/system/tools/GenVarX/Osativa and https://kbcommons.org/system/tools/GenVarX/Athaliana, respectively.

Construction and Application of Trajectory Data Analysis Model Based on Big Data and Stochastic Gradient Descent Algorithm

This paper studies the model construction of computing and storage resource management system framework based on Hadoop and the implementation of trajectory data analysis function under big data. Relying on the cloud platform infrastructure, in order to support the rapid data growth and massive data processing needs, it provides a mixed storage and analysis platform for structured and unstructured data, and uses big data technology to build a highly scalable and distributed data processing framework. The distributed computation, overall frame model of the memory system, and function module have been built with the aim of constructing the system in consideration. Second, by using Hadoop to preprocess the original data and concentrating on the data hierarchical design model and key technology analysis of big data systems, the design model, functional modules, technological solutions, and SGD algorithm are suggested, along with the detailed implementation procedure. Lastly, by merging the data of running vehicles, the system accomplishes the data analysis of vehicle trajectory, empty and load cars, and load and unload people.

BIG DATA PROCESSING WITH PYTHON IN SOCIAL NETWORKS

Social networks generate a huge amount of data about the behavior, interests, preferences and connections of users, which can be used for various purposes such as marketing, advertising, analytics, research, etc. However, these data have a large volume, variety and update rate, and require special methods and tools for their analysis and processing. Python is a high-level programming language that has many advantages for working with big data, such as simplicity and flexibility of syntax, a rich set of libraries and frameworks for data processing and analysis, support for multithreading and distributed computing, open source code, and a large community of developers and users. The article shows how to use Python to parse and collect data from various sources, to clean and pre-process data, such as removing duplicates, gaps, outliers and noise, normalizing and standardizing data, encoding categorical features, etc., for analysis and data visualization, such as building statistical characteristics, graphs, charts and maps to study the distribution, correlation and dependence of data, for machine learning and data mining. Keywords: Python, big data, social networks, data parsing, data analysis, machine learning.

Repeated Transcranial Photobiomodulation with Light-Emitting Diodes Improves Psychomotor Vigilance and EEG Networks of the Human Brain.

Transcranial photobiomodulation (tPBM) has been suggested as a non-invasive neuromodulation tool. The repetitive administration of light-emitting diode (LED)-based tPBM for several weeks significantly improves human cognition. To understand the electrophysiological effects of LED-tPBM on the human brain, we investigated alterations by repeated tPBM in vigilance performance and brain networks using electroencephalography (EEG) in healthy participants. Active and sham LED-based tPBM were administered to the right forehead of young participants twice a week for four weeks. The participants performed a psychomotor vigilance task (PVT) during each tPBM/sham experiment. A 64-electrode EEG system recorded electrophysiological signals from each participant during the first and last visits in a 4-week study. Topographical maps of the EEG power enhanced by tPBM were statistically compared for the repeated tPBM effect. A new data processing framework combining the group's singular value decomposition (gSVD) with eLORETA was implemented to identify EEG brain networks. The reaction time of the PVT in the tPBM-treated group was significantly improved over four weeks compared to that in the sham group. We observed acute increases in EEG delta and alpha powers during a 10 min LED-tPBM while the participants performed the PVT task. We also found that the theta, beta, and gamma EEG powers significantly increased overall after four weeks of LED-tPBM. Combining gSVD with eLORETA enabled us to identify EEG brain networks and the corresponding network power changes by repeated 4-week tPBM. This study clearly demonstrated that a 4-week prefrontal LED-tPBM can neuromodulate several key EEG networks, implying a possible causal effect between modulated brain networks and improved psychomotor vigilance outcomes.

A coordinates-based hierarchical computing framework towards spatial data processing

The vehicle-based mobile base station (VMBS) has been developed by NTT a few years ago, which plays an important role in reconstructing an emergency communication network by integrating with mobile edge computing to support the cloud service for data pre-processing tasks. Spatial data processing refers to characteristics, such as fast-changing, massive, and potentially infinite. Considering the capacities of network bandwidth and storage of VMBS, new challenging issues have emerged such as how to quantify information loss and how to optimize data processing in the paradigm of mobile edge computing. To address these two challenges, we design a coordinates-based hierarchical computing framework for spatial data processing, named HierCom. Using HierCom, the optimal position of VMBS can be selected to decrease the transmission delay, thereby reducing information loss in the data processing in an edge computing environment. Furthermore, we also propose a near least neighbor-based algorithm to optimize the information loss. Finally, simulations on the real-world dataset show that the information loss of our method is reduced by 27% compared with the baseline method.

A Modular Framework for Data Processing at the Edge: Design and Implementation.

There is a rapid increase in the number of edge devices in IoT solutions, generating vast amounts of data that need to be processed and analyzed efficiently. Traditional cloud-based architectures can face latency, bandwidth, and privacy challenges when dealing with this data flood. There is currently no unified approach to the creation of edge computing solutions. This work addresses this problem by exploring containerization for data processing solutions at the network's edge. The current approach involves creating a specialized application compatible with the device used. Another approach involves using containerization for deployment and monitoring. The heterogeneity of edge environments would greatly benefit from a universal modular platform. Our proposed edge computing-based framework implements a streaming extract, transform, and load pipeline for data processing and analysis using ZeroMQ as the communication backbone and containerization for scalable deployment. Results demonstrate the effectiveness of the proposed framework, making it suitable for time-sensitive IoT applications.

A systematic literature review of Internet of Video Things: Trends, techniques, datasets, and framework

In the last decade, Internet of Things (IoT) and machine learning (ML) have been used for developing semi or fully automated systems of any application. Due to the advancement in networking technologies, the contribution of video-based input is also rising for IoT systems. Hence, Internet of Video Things (IoVT) has been an emerging field to gain more human trust in the systems. IoVT is an integrated attempt of computer vision sensors, networking, video or image processing and information storage systems. In this paper, we have highlighted the literature evidence in the IoVT field. There is a need to analyze it in terms of different aspects, that enables the researchers to understand research trends in the field and lead them to future research directions. Therefore, this article presents a systematic literature review (SLR) in IoVT using various research publications, since its evolution. The study applies article selection criteria on different search methods and compares the relevant published articles. We have analyzed these filtered articles for identifying the study trends, their used techniques & frameworks for data processing and inputs or datasets used for their studies. At the end, various research opportunities and challenges have been framed in the field for future research references.

High-quality images and data augmentation based on inverse projection transformation significantly improve the estimation accuracy of biomass and leaf area index

Estimating above-ground biomass (AGB) and leaf area index (LAI) is crucial for determining crop developmental status and estimating grain yield, thus facilitating high-throughput phenotyping of maize (Zea Mays L.). Recently, unmanned aerial vehicle (UAV) have been widely used to monitor crop growth and estimate LAI and AGB. However, few studies have reported on the quantitative analysis of improved accuracy in biomass and leaf area index estimation using undistorted images obtained through inverse projection transformation algorithms. Additionally, there is limited research on predicting biomass and LAI during various growth stages by replacing traditional data augmentation methods with images captured from different azimuth and zenith angles. The results showed that when using undistorted image at single solar zenith angle of 0°, the accuracy of AGB and LAI estimation was improved by 20.3% and 7.3%, respectively, compared to the models estimated from orthoimages. The undistorted images with various azimuth and zenith angles were 9–15 times larger than the orthomosaic image dataset, and could be used as a new training data that is similar to the original data but with variations. The AGB and LAI estimation model constructed using the data augmentation method achieved better performance with higher average R2 (0.98). Model performance was increased by 40% and 16.7% for estimating AGB and by 19.3% and 11.3% for estimating LAI based on undistorted images, respectively, in comparison to models based on orthophotos and undistorted images due to the increased diversity and quantity of training data. Canopy structural including canopy coverage and plant height, characterized by 96.9%, was found to be the most reliable indication for estimating AGB and LAI, while other RGB sensor-derived feature combinations contributed only 3.1% to the estimation. Spectral features took the second place, and the textural features was the weakest. In summary, the data processing framework using data augmentation based on various azimuth and zenith angles' images can improve the performance of crop growth estimation and provide a mechanism for precision agriculture under field conditions.

Field-driven design and performance evaluation of dual functionally graded triply periodic minimal surface structures for additive manufacturing

Triply periodic minimal surface (TPMS) structures prepared by additive manufacturing (AM) have been widely used in aerospace and thermal management systems due to their functional advantages such as high specific strength and excellent heat transfer coefficient. However, lattices with uniform or simple gradients cannot match the regional diversity needs of functional advantages in service. This study proposed a field-driven AM data processing framework for dual functionally graded TPMS lattices. For unified data processing, mathematical functions, simulated outcomes, and geometric models were transformed into fields. I-WP and Diamond units were selected as the primitive and filling structures of Ti-6Al-4V lattices, respectively. According to the functional requirements, the gradient distribution design guided by the simulated stress field is completed through the proposed framework. The results illustrate that the field-driven dual graded lattices present superior compressive mechanical response. At almost equal volume fractions, their compressive strength is approximately 100% more than that of primitive lattices and 69% greater than that of dual uniform lattices. Furthermore, the dual lattices have larger specific surface areas than the primitive lattices, making them better at heat insulation and heat dissipation. This field-driven digital framework will show enormous potential in the matching design of functionally graded lattices.

Analysis of Cyber-Intelligence Frameworks for AI Data Processing

This paper deals with the concept of cyber intelligence and its components as a fundamental tool for the protection of information today. After that, the main cyber-intelligence frameworks that are currently applied worldwide (Diamond Model, Cyberkill Chain, and Mitre Att&ck) are described to subsequently analyse them through their practical application in a real critical cyber incident, as well as analyse the strengths and weaknesses of each one of them according to the comparison of seventeen variables of interest. From this analysis and considering the two actions mentioned, it is concluded that Mitre Att&ck is the most suitable framework due to its flexibility, permanent updating, and the existence of a powerful database. Finally, an explanation is given for how Mitre Att&ck can be integrated with the research and application of artificial intelligence in the achievement of the objectives set and the development of tools that can serve as support for the detection of the patterns and authorship of cyberattacks.

Industrial Data-Driven Processing Framework Combining Process Knowledge for Improved Decision Making—Part 1: Framework Development

Data management systems are increasingly used in industrial processes. However, data collected as part of industrial process operations, such as sensor or measurement instruments data, contain various sources of errors that can hamper process analysis and decision making. The authors propose an operating-regime-based data processing framework for industrial process decision making. The framework was designed to increase the quality and take advantage of available process data use to make informed offline strategic business operation decisions, i.e., environmental, cost and energy analysis, optimization, fault detection, debottlenecking, etc. The approach was synthesized from best practices derived from the available framework and improved upon its predecessor by putting forward the combination of process expertise and data-driven approaches. This systematic and structured approach includes the following stages: (1) scope of the analysis, (2) signal processing, (3) steady-state operating periods detection, (4) data reconciliation and (5) operating regime detection and identification. The proposed framework is applied to the brownstock washing department of a dissolving pulp mill. Over a 5-month period, the process was found to be in steady-state 32% of the time. Twenty (20) distinct operating regimes were identified. Further processing with the help of data reconciliation techniques, principal component analysis and k-means clustering showed that the main drivers explaining the operating regimes are the pulp level in tanks, its density, and the shower wash water flow rate. Additionally, it was concluded that the top four persistently problematic sensors across the steady-state spans that would need to be verified are three flow meters (06FIC137, 06FIC152, and 06FIC433), and one consistency sensor (06NIC423). This information was relayed to process experts contacts at the plant for further investigation.

Cortical microvascular blood flow velocity mapping by combining dynamic light scattering optical coherence tomography and two-photon microscopy.

The accurate large-scale mapping of cerebral microvascular blood flow velocity is crucial for a better understanding of cerebral blood flow (CBF) regulation. Although optical imaging techniques enable both high-resolution microvascular angiography and fast absolute CBF velocity measurements in the mouse cortex, they usually require different imaging techniques with independent system configurations to maximize their performances. Consequently, it is still a challenge to accurately combine functional and morphological measurements to co-register CBF speed distribution from hundreds of microvessels with high-resolution microvascular angiograms. We propose a data acquisition and processing framework to co-register a large set of microvascular blood flow velocity measurements from dynamic light scattering optical coherence tomography (DLS-OCT) with the corresponding microvascular angiogram obtained using two-photon microscopy (2PM). We used DLS-OCT to first rapidly acquire a large set of microvascular velocities through a sealed cranial window in mice and then to acquire high-resolution microvascular angiograms using 2PM. The acquired data were processed in three steps: (i)2PM angiogram coregistration with the DLS-OCT angiogram, (ii)2PM angiogram segmentation and graphing, and (iii)mapping of the CBF velocities to the graph representation of the 2PM angiogram. We implemented the developed framework on the three datasets acquired from the mice cortices to facilitate the coregistration of the large sets of DLS-OCT flow velocity measurements with 2PM angiograms. We retrieved the distributions of red blood cell velocities in arterioles, venules, and capillaries as a function of the branching order from precapillary arterioles and postcapillary venules from more than 1000 microvascular segments. The proposed framework may serve as a useful tool for quantitative analysis of large microvascular datasets obtained by OCT and 2PM in studies involving normal brain functioning, progression of various diseases, and numerical modeling of the oxygen advection and diffusion in the realistic microvascular networks.

YARN Schedulers for Hadoop MapReduce Jobs: Design Goals, Issues and Taxonomy

Objective: Big Data processing is a demanding task, and several big data processing frameworks have emerged during recent decades. The performance of these frameworks greatly dependent on resource management models. Methods: YARN is one of such models which acts as a resource management layer and provides computational resources for execution engines (Spark, MapReduce, storm, etc.) through its schedulers. The most important aspect of resource management is job scheduling. Results: In this paper, we first present the design goal of YARN real-life schedulers (FIFO, Capacity, and Fair) for the MapReduce engine. Later, we discuss the scheduling issues of the Hadoop MapReduce cluster. Conclusion: Many efforts have been carried out in the literature to address issues of data locality, heterogeneity, straggling, skew mitigation, stragglers and fairness in Hadoop MapReduce scheduling. Lastly, we present the taxonomy of different scheduling algorithms available in the literature based on some factors like environment, scope, approach, objective and addressed issues.

A New Big Data Processing Framework for the Online Roadshow

The Online Roadshow, a new type of web application, is a digital marketing approach that aims to maximize contactless business engagement. It leverages web computing to conduct interactive game sessions via the internet. As a result, massive amounts of personal data are generated during the engagement process between the audience and the Online Roadshow (e.g., gameplay data and clickstream information). The high volume of data collected is valuable for more effective market segmentation in strategic business planning through data-driven processes such as web personalization and trend evaluation. However, the data storage and processing techniques used in conventional data analytic approaches are typically overloaded in such a computing environment. Hence, this paper proposed a new big data processing framework to improve the processing, handling, and storing of these large amounts of data. The proposed framework aims to provide a better dual-mode solution for processing the generated data for the Online Roadshow engagement process in both historical and real-time scenarios. Multiple functional modules, such as the Application Controller, the Message Broker, the Data Processing Module, and the Data Storage Module, were reformulated to provide a more efficient solution that matches the new needs of the Online Roadshow data analytics procedures. Some tests were conducted to compare the performance of the proposed frameworks against existing similar frameworks and verify the performance of the proposed framework in fulfilling the data processing requirements of the Online Roadshow. The experimental results evidenced multiple advantages of the proposed framework for Online Roadshow compared to similar existing big data processing frameworks.

Data Compression-Aware Performance Analysis of Dask and Spark for Earth Observation Data Processing

High-performance computing is a good choice for handling Big Earth Observation data, allowing the processing of the data in a distributed and performance-efficient way using in-memory computing frameworks. The data compression technique reduces the amount of storage and network transfer time and improves processing performance. The article aims to investigate the effectiveness of widely used distributed data processing frameworks in conjunction with lossless data compression techniques, to find the optimal compression method and processing framework for specific earth observation workflows. Normalized Difference Vegetation Index has been evaluated for the territory of Armenia, obtaining data from the Sentinel satellite and considering the supported compression methods to compare the performance of in-memory Dask and Spark frameworks. Experiments show that the Zstandard compression method and the Dask framework are the best choices for such workflows.