Data Processing Tasks Research Articles

Self-Supervised Learning for Efficient Antialiasing Seismic Data Interpolation

Reconstruction of seismic data is an important but challenging task in seismic data processing. Different machine-learning-based algorithms have been developed to solve this ill-posed problem and achieved great progress. However, most machine-learning-based methods rely on supervised learning where a good training dataset with many complete shot-gathers are required to train the model. Although the generative model has been used for unsupervised learning and reconstructing signals in a shot-gather, it fails to accurately resolve the fine features, especially when aliasing is the main concern. In addition, multiple shots’ interpolation problems have not been fully investigated by the unsupervised machine-learning-based approaches. In this work, we propose a self-supervised learning method using a blind-trace network and two antialiasing techniques (automatic spectrum suppression and mix-training) for seismic data reconstruction. The method is validated using challenging and realistic scenarios. Test results show that the method can be applied to single-shot or multiple shots’ cases and adapt well to different decimation patterns.

Community Detection in Fully-Connected Multi-layer Networks Through Joint Nonnegative Matrix Factorization

Modern data analysis and processing tasks typically involve large sets of structured data. Graphs provide a powerful tool to describe the structure of such data, where the entities and the relationships between them are modeled as the nodes and edges of the graph. Traditional single layer network models are insufficient for describing the multiple entity types and modes of interaction encountered in real-world applications. Recently, multi-layer network models, which consider the different types of interactions both within and across layers, have emerged to model these systems. One of the important tools in understanding the topology of these high-dimensional networks is community detection. In this paper, a joint nonnegative matrix factorization approach is proposed to detect the community structure in multi-layer networks. The proposed approach models the multi-layer network as the union of a multiplex and bipartite network and formulates community detection as a regularized optimization problem. This optimization problem simultaneously finds the nonnegative low-rank embedding of the intra- and inter-layer adjacency matrices while minimizing the distance between the two to guarantee pair-wise similarity across embeddings. The proposed approach can detect the community structure for both homogeneous and heterogeneous multi-layer networks and is robust to noise and sparsity. The performance of the proposed approach is evaluated for both simulated and real networks and compared to state-of-the-art methods.

Multilevel Pulse Position Modulation With Level Trimming for Electromagnetic Nanocommunications in the Terahertz Band

<italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Nanomachines</i> are submicrometer-scale devices led by nanotechnology that can perform simple sensing, local actuation, limited data processing, storage, and communication tasks in the terahertz (THz) band, that is, from 0.1-10 THz. Electromagnetic <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">nanocommunication</i> among nanomachines results in a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">nanonetwork</i> which could breakthrough promising applications in multiple domains such as software-defined metamaterials, in-body communication, and on-chip communication. This study adopts a modulation scheme for nanomachine communication based on <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">multilevel</i> pulse position modulation (ML-PPM). The multilevel scheme uses several orthogonal codes and is combined with PPM to generate the final transmit signal consisting of several multilevels. In this paper, we propose a more advanced scheme called <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">level trimming</i> to further boost the data rates of the ML-PPM scheme. Employing level-trimming, we transmit a fewer number of levels than required in ML-PPM, which will results in an spectral efficiency gain at the nanoreceiver. The simulation results reveal that the link capacity of the proposed scheme can be increased more than twofold using the level-trimming approach while the error rate performance remains better than the conventional ML-PPM. Moreover, the computational complexity of transceivers is reduced with the transmission of fewer levels, which is a significant necessity in nanocommunication. Also, although level-trimming causes artificial errors, it also improves the decoding performance by reducing the number of levels. We believe that the potential impact of this study will open doors for further investigations on various possible modulation formats for THz nanocommunication.

Trust-Based Scheduling Framework for Big Data Processing with MapReduce

Security and privacy have become a great concern in cloud computing platforms in which users risk the leakage of their private data. The leakage can happen while the data is at rest (in storage), in processing, or on moving within a cloud or between different cloud infrastructures, e.g., from private to public clouds. This paper focuses on protecting data “in processing”. For big data applications, the MapReduce framework has been proven as an efficient solution and has been widely deployed, e.g., in healthcare and business data analysis. In this article, we propose a trust-based framework for MapReduce in big data processing tasks. Specifically, we first quantify and propose to assign the sensitive values for data and trust values for map and reduce slots. We then compute the trust value of each resource employed in the big data processing tasks. Depending on the data's sensitivity level of a task, the task requires a given level of trust (i.e., higher sensitive data requires servers/slots with higher trust level). The MapReduce scheduling problem is then formulated as the maximum weighted matching problem of a bipartite graph that aims to maximize the total trust value over all possible assignments subject to various trust requirement of different tasks. The problem is known to be NP-hard. To tackle it, we observe that within a computing node (VM), slots share the same trust value granted from the secured transformation phase. This helps reduce the number of slot nodes of a weight bipartite graph. Leveraging this fact, we propose an efficient heuristic algorithm that achieves 94.7 percent of the optimal solution obtained via exhaustive search. Extensive simulations show that the trust-based scheduling scheme provides much higher protection for data sensitivity while ensuring good performance for big data applications.

Ranked enumeration of join queries with projections

Join query evaluation with ordering is a fundamental data processing task in relational database management systems. SQL and custom graph query languages such as Cypher offer this functionality by allowing users to specify the order via the ORDER BY clause. In many scenarios, the users also want to see the first k results quickly (expressed by the LIMIT clause), but the value of k is not predetermined as user queries are arriving in an online fashion. Recent work has made considerable progress in identifying optimal algorithms for ranked enumeration of join queries that do not contain any projections. In this paper, we initiate the study of the problem of enumerating results in ranked order for queries with projections. Our main result shows that for any acyclic query, it is possible to obtain a near-linear (in the size of the database) delay algorithm after only a linear time preprocessing step for two important ranking functions: sum and lexicographic ordering. For a practical subset of acyclic queries known as star queries, we show an even stronger result that allows a user to obtain a smooth tradeoff between faster answering time guarantees using more preprocessing time. Our results are also extensible to queries containing cycles and unions. We also perform a comprehensive experimental evaluation to demonstrate that our algorithms, which are simple to implement, improve up to three orders of magnitude in the running time over state-of-the-art algorithms implemented within open-source RDBMS and specialized graph databases.

Taming System Dynamics on Resource Optimization for Data Processing Workflows: A Probabilistic Approach

In many data-intensive applications, workflow is often used as an important model for organizing data processing tasks and resource provisioning is an important and challenging problem for improving the performance of workflows. Recently, system variations in the cloud and large-scale clusters, such as those in I/O and network performances and failure events, have been observed to greatly affect the performance of workflows. Traditional resource provisioning methods, which overlook these variations, can lead to suboptimal resource provisioning results. In this article, we provide a general solution for workflow performance optimizations considering system variations. Specifically, we model system dynamics as time-dependent random variables and take their probability distributions as optimization input. Despite its effectiveness, this solution involves heavy computation overhead. Thus, we propose three pruning techniques to simplify workflow structure and reduce the probability evaluation overhead. We implement our techniques in a runtime library, which allows users to incorporate efficient probabilistic optimization into existing resource provisioning methods. Experiments show that probabilistic solutions can improve the performance by up to 65 percent compared to state-of-the-art static solutions, and our pruning techniques can greatly reduce the overhead of our probabilistic approach.

ЗАДАЧА РАЗРАБОТКИ ИНТЕЛЛЕКТУАЛЬНОЙ АВТОМАТИЗИРОВАННОЙ СИСТЕМЫ ПОДБОРА ПЕРСОНАЛА «УМНОГО» ПРЕДПРИЯТИЯ СТРОИТЕЛЬНОЙ ОТРАСЛИ

The work of the personnel department of the company largely determines the effectiveness of its activities. Planning, organizing, accounting, and controlling the staff work are management functions that can and should be automated in the era of digital transformation. The authors have assessed the functionality of the HR-service of the construction firm; they have carried out the decomposition of first and second level of recruitment and data processing tasks to assess the possible priority of automation of personnel processes in the intelligent system of personnel planning of the construction enterprise.

Serverless data pipeline approaches for IoT data in fog and cloud computing

With the increasing number of Internet of Things (IoT) devices, massive amounts of raw data is being generated. The latency, cost, and other challenges in cloud-based IoT data processing have driven the adoption of Edge and Fog computing models, where some data processing tasks are moved closer to data sources. Properly dealing with the flow of such data requires building data pipelines, to control the complete life cycle of data streams from data acquisition at the data source, edge and fog processing, to Cloud side storage and analytics. Data analytics tasks need to be executed dynamically at different distances from the data sources and often on very heterogeneous hardware devices. This can be streamlined by the use of a Serverless (or FaaS) cloud computing model, where tasks are defined as virtual functions, which can be migrated from edge to cloud (and vice versa) and executed in an event-driven manner on data streams. In this work, we investigate the benefits of building Serverless data pipelines (SDP) for IoT data analytics and evaluate three different approaches for designing SDPs: (1) Off-the-shelf data flow tool (DFT) based, (2) Object storage service (OSS) based and (3) MQTT based. Further, we applied these strategies on three fog applications (Aeneas, PocketSphinx, and custom Video processing application) and evaluated the performance by comparing their processing time (computation time, network communication and disk access time), and resource utilization. Results show that DFT is unsuitable for compute-intensive applications such as video or image processing, whereas OSS is best suitable for this task. However, DFT is nicely fit for bandwidth-intensive applications due to the minimum use of network resources. On the other hand, MQTT-based SDP is observed with increase in CPU and Memory usage as the number of users rose, and experienced a drop in data units in the pipeline for PocketSphinx and custom video processing applications, however it performed well for Aeneas which had low size data units.

Pyramid transform of manifold data via subdivision operators

Abstract Multiscale transforms have become a key ingredient in many data processing tasks. With technological development we observe a growing demand for methods to cope with nonlinear data structures such as manifold values. In this paper we propose a multiscale approach for analyzing manifold-valued data using a pyramid transform. The transform uses a unique class of downsampling operators that enable a noninterpolating subdivision schemes as upsampling operators. We describe this construction in detail and present its analytical properties, including stability and coefficient decay. Next, we numerically demonstrate the results and show the application of our method to denoising and anomaly detection.

Predictive Wireless Channel Modeling of MmWave Bands Using Machine Learning

The exploitation of higher millimeter wave (MmWave) is promising for wireless communication systems. The goals of machine learning (ML) and its subcategories of deep learning beyond 5G (B5G) is to learn from the data and make a prediction or a decision other than relying on the classical procedures to enhance the wireless design. The new wireless generation should be proactive and predictive to avoid the previous drawbacks in the existing wireless generations to meet the 5G target services pillars. One of the aspects of Ultra-Reliable Low Latency Communications (URLLC) is moving the data processing tasks to the cellular base stations. With the rapid usage of wireless communications devices, base stations are required to execute and make decisions to ensure communication reliability. In this paper, an efficient new methodology using ML is applied to assist base stations in predicting the frequency bands and the path loss based on a data-driven approach. The ML algorithms that are used and compared are Multilelayers Perceptrons (MLP) as a neural networks branch and Random Forests. Systems that consume different bands such as base stations in telecommunications with uplink and downlink transmissions and other internet of things (IoT) devices need an urgent response between devices to alter bands to maintain the requirements of the new radios (NR). Thus, ML techniques are needed to learn and assist a base station to fluctuate between different bands based on a data-driven system. Then, to testify the proposed idea, we compare the analysis with other deep learning methods. Furthermore, to validate the proposed models, we applied these techniques to different case studies to ensure the success of the proposed works. To enhance the accuracy of supervised data learning, we modified the random forests by combining an unsupervised algorithm to the learning process. Eventually, the superiority of ML towards wireless communication demonstrated great accuracy at 90.24%.

Comparative analysis of software optimization methods in context of branch predication on GPUs

General Purpose computing for Graphical Processing Units (GPGPU) technology is a powerful tool for offloading parallel data processing tasks to Graphical Processing Units (GPUs). This technology finds its use in variety of domains – from science and commerce to hobbyists. GPU-run general-purpose programs will inevitably run into performance issues stemming from code branch predication. Code predication is a GPU feature that makes both conditional branches execute, masking the results of incorrect branch. This leads to considerable performance losses for GPU programs that have large amounts of code hidden away behind conditional operators. This paper focuses on the analysis of existing approaches to improving software performance in the context of relieving the aforementioned performance loss. Description of said approaches is provided, along with their upsides, downsides and extents of their applicability and whether they address the outlined problem. Covered approaches include: optimizing compilers, JIT-compilation, branch predictor, speculative execution, adaptive optimization, run-time algorithm specialization, profile-guided optimization. It is shown that the aforementioned methods are mostly catered to CPU-specific issues and are generally not applicable, as far as branch-predication performance loss is concerned. Lastly, we outline the need for a separate performance improving approach, addressing specifics of branch predication and GPGPU workflow.

An Energy-Efficient In-Network Computing Paradigm for 6G

In the 6G network era, energy-efficient computing means higher resource utilization and lower energy consumption. The traditional computing model regards the network as a transmission pipeline and has not fully explored the potential of network devices. In-network computing is a new type of computing model that delegates application-layer processing functions to the network data plane. The traffic can be processed during transmission, and the amount of traffic is reduced. As a result, the computing pressure and energy consumption of the cloud-side system is reduced. Most of the existing in-network computing models are based on programable network devices. Due to the limitations of the TCP/IP protocol, these methods can only handle application messages at the packet level. We propose an energy-efficient in-network computing paradigm for 6G that integrates network functions into a general computing platform instead of delegating computing tasks to network devices. The computing platform that integrates network functions replaces traditional network devices and acts as a network node. Unlike traditional network devices, the network node provides a unified operating environment for application tasks through hypervisors and containers. Therefore, data processing tasks can be deployed on cloud servers or on network nodes. The network controller schedules the processing tasks to execute on the network node passing the application traffic. The experimental results show that our paradigm can make full use of the computing resources of network nodes and significantly reduce the computing pressure, network transmission overhead and energy consumption of the data center.

Sparse inversion-based seismic random noise attenuation via self-paced learning

Seismic random noise reduction is an important task in seismic data processing at the Chinese loess plateau area, which benefits the geologic structure interpretation and further reservoir prediction. The sparse inversion is one of the widely used tools for seismic random noise reduction, which is often solved via the sparse approximation with a regularization term. The ℓ1 norm and total variation (TV) regularization term are two commonly used regularization terms. However, the ℓ1 norm is only a relaxation of the ℓ0 norm, which cannot always provide a sparse result. The TV regularization term may provide unexpected staircase artifacts. To avoid these disadvantages, we proposed a workflow for seismic random noise reduction by using the self-paced learning (SPL) scheme and a sparse representation (i.e. the continuous wavelet transform, CWT) with a mixed norm regularization, which includes the ℓp norm and the TV regularization. In the implementation, the SPL, which is inspired by human cognitive learning, is introduced to avoid the bad minima of the non-convex cost function. The SPL can first select the high signal-to-noise ratio (SNR) seismic data and then gradually select the low SNR seismic data into the proposed workflow. Moreover, the generalized Beta wavelet (GBW) is adopted as the basic wavelet of the CWT to better match for seismic wavelets and then obtain a more localized time-frequency (TF) representation. It should be noted that the GBW can easily constitute a tight frame, which saves the calculation time. Synthetic and field data examples are adopted to demonstrate the effectiveness of the proposed workflow for effectively suppressing seismic random noises and accurately preserving valid seismic reflections.

Integrated computer digital decision for Offshore production automated management

Rising requirements for the hydrocarbons production management system state are based on the industrial situation control necessity according to the environmental conditions. Development of the system construction and complexity indicates the integrative approach implementation reasonability for the producing capacity main indexes estimation and regularity of pace parameters evaluation as well as the layer productiveness level identification. Data processing and management tasks are the waymarks for the industrial multilevel structures models creation. Computer integrated model includes the estimated analytical tool for the extended operational functional support for the production parameters evaluation and technological process state diagnostics in changeable conditions and productive region origin and features traceability. Based on the well observations it is possible to apply the systematic approach for the trends occurrence and dynamics evaluation of the reservoir development time series data. And integrated decision includes the information value degree identification tool for the hydrocarbons production digital model updating.

Network-Aware Optimization of Distributed Learning for Fog Computing

Fog computing promises to enable machine learning tasks to scale to large amounts of data by distributing processing across connected devices. Two key challenges to achieving this goal are (i) heterogeneity in devices’ compute resources and (ii) topology constraints on which devices communicate with each other. We address these challenges by developing a novel network-aware distributed learning methodology where devices optimally share local data processing and send their learnt parameters to a server for periodic aggregation. Unlike traditional federated learning, our method enables devices to offload their data processing tasks to each other, with these decisions optimized to trade off costs associated with data processing, offloading, and discarding. We analytically characterize the optimal data transfer solution under different assumptions on the fog network scenario, showing for example that the value of offloading is approximately linear in the range of computing costs in the network when the cost of discarding is modeled as decreasing linearly in the amount of data processed at each node. Our experiments on real-world data traces from our testbed confirm that our algorithms improve network resource utilization substantially without sacrificing the accuracy of the learned model, for varying distributions of data across devices. We also investigate the effect of network dynamics on model learning and resource costs.

FPGA-Based Real-Time Image Manipulation and Advanced Data Acquisition for 2-D-XRAY Detectors

Scientific experiments rely on some type of measurements that provide the required data to extract aimed information or conclusions. Data production and analysis are, therefore, essential components at the heart of any scientific experimental application. Traditionally, efforts on detector development for photon sources have focused on the properties and performance of the detection front-ends. In many cases, the data acquisition chain as well as data processing are treated as a complementary component of the detector system and added at a later stage of the project. In most cases, data processing tasks are entrusted to CPUs; achieving, thus, the minimum bandwidth requirements and keeping hardware relatively simple in terms of functionalities. This also minimizes design effort, complexity, and implementation cost. This approach is changing in the last few years as it does not accommodate new high-performance detectors; field-programmable gate array (FPGA) and GPUs are now used to perform complex image manipulation tasks such as image reconstruction, image rotation, accumulation, filtering, and data analysis. This frees up CPUs for simpler tasks. The objective of this article is to present both the implementation of real-time FPGA-based image manipulation techniques, as well as the performance of the European Synchrotron radiation Facility (ESRF) data acquisition platform called remote direct memory access (RDMA)-based acquisition system for high-performance applications (RASHPA), into the back-end board of the SMARTPIX photon-counting detector developed at the ESRF.

The application of fully unmanned robotic systems for inspection of subsea pipelines

This paper focuses on recent innovations in the methods used for external remote subsea pipeline inspection. An unmanned method is revealed, in which an Autonomous Underwater Vehicle (AUV), paired with an Unmanned Surface Vessel (USV) was utilised to complete inspections. Results are presented from a recent project, in which existing hardware and software were integrated and combined with automated workflows to form a solution, involving an AUV operated from a USV, with operations remotely controlled from shore. As inspection data was acquired, self-actuating workflows ran onboard the AUV, enabling data processing tasks to commence and QC messages/alerts to be transmitted to the control centre, this allowed execution of in-water mission adjustments. The primary focus of this paper is on the development and implementation of an automated, fully unmanned system for subsea inspection operations. Initially, a brief review is presented of recent developments in this field. Links are drawn between these wider developments and progress made within the project, and areas are highlighted where further work is required for realisation of a comprehensive unmanned pipeline inspection solution.

AI-enabled IoT-Edge Data Analytics for Connected Living

As deep learning, virtual reality, and other technologies become mature, real-time data processing applications running on intelligent terminals are emerging endlessly; meanwhile, edge computing has developed rapidly and has become a popular research direction in the field of distributed computing. Edge computing network is a network computing environment composed of multi-edge computing nodes and data centers. First, the edge computing framework and key technologies are analyzed to improve the performance of real-time data processing applications. In the system scenario where the collaborative deployment tasks of multi-edge nodes and data centers are considered, the stream processing task deployment process is formally described, and an efficient multi-edge node-computing center collaborative task deployment algorithm is proposed, which solves the problem of copy-free task deployment in the task deployment problem. Furthermore, a heterogeneous edge collaborative storage mechanism with tight coupling of computing and data is proposed, which solves the contradiction between the limited computing and storage capabilities of data and intelligent terminals, thereby improving the performance of data processing applications. Here, a Feasible Solution (FS) algorithm is designed to solve the problem of placing copy-free data processing tasks in the system. The FS algorithm has excellent results once considering the overall coordination. Under light load, the V value is reduced by 73% compared to the Only Data Center-available (ODC) algorithm and 41% compared to the Hash algorithm. Under heavy load, the V value is reduced by 66% compared to the ODC algorithm and 35% compared to the Hash algorithm. The algorithm has achieved good results after considering the overall coordination and cooperation and can more effectively use the bandwidth of edge nodes to transmit and process data stream, so that more tasks can be deployed in edge computing nodes, thereby saving time for data transmission to the data centers. The end-to-end collaborative real-time data processing task scheduling mechanism proposed here can effectively avoid the disadvantages of long waiting times and unable to obtain the required data, which significantly improves the success rate of the task and thus ensures the performance of real-time data processing.

Current status and future prospects of drug-target interaction prediction.

Drug-target interaction prediction is important for drug development and drug repurposing. Many computational methods have been proposed for drug-target interaction prediction due to their potential to the time and cost reduction. In this review, we introduce the molecular docking and machine learning-based methods, which have been widely applied to drug-target interaction prediction. Particularly, machine learning-based methods are divided into different types according to the data processing form and task type. For each type of method, we provide a specific description and propose some solutions to improve its capability. The knowledge of heterogeneous network and learning to rank are also summarized in this review. As far as we know, this is the first comprehensive review that summarizes the knowledge of heterogeneous network and learning to rank in the drug-target interaction prediction. Moreover, we propose three aspects that can be explored in depth for future research.

How to Use Crowdsourcing Effectively

Crowdsourcing has become the mot du jour when it comes to resolving any types of problems, online or offline, that require sustained human involvement. We see it applied in order to motivate employees to engage with less rewarding daily routines, to attract the best possible ideas and approaches to boost innovation, or to complete data processing tasks that computing technology has yet to master quickly and accurately.In this paper we look at its various forms and flavors, from gamification to human computation and grand challenges, and discuss how it could be used to turn conventional content management applications into social machines in which tasks are performed as optimal combinations of human and computational intelligence. We introduce a framework for the analysis of the most important building blocks of such systems, as well as design and participation best practices that should guide their development.