Multi-node Cluster Research Articles

PASS4SWAT: Orchestration of containerized SWAT for facilitating computational reproducibility of model calibration and uncertainty analysis

When performing modeling routines with iterative simulations, a research gap exists in developing the parallelizing frameworks that are adaptable to the increasing complexities of the hydrologic models. This study addresses this gap by proposing a parallel simulation stack for SWAT (PASS4SWAT) that leverages the power of cloud-native infrastructure. Cloud-native applications are designed to be scalable, agile, and resilient, making them ideal for computationally intensive tasks such as hydrologic modeling. To achieve this goal, PASS4SWAT was created by integrating a message broker, a container runtime and orchestration tool, and certain components from SWAT-CUP. We evaluated PASS4SWAT using the Jinjiang watershed model and a synthetic model with high input/output demands to demonstrate its effectiveness. The results show that PASS4SWAT can consistently replicate the results obtained with SWAT-CUP and significantly reduce the runtime, achieving 91.5% and 93.2% reductions in single-node and multinode Kubernetes clusters, respectively. Therefore, in this paper, we conclude that PASS4SWAT can effectively address the high computational demands of the SWAT model by scaling out parallel tasks on a cluster, and can adapt flexibly to diverse environments, including single-node clusters, multinode clusters, and potential cloud platforms.

Task-Level Checkpointing and Localized Recovery to Tolerate Permanent Node Failures for Nested Fork–Join Programs in Clusters

Exascale supercomputers consist of millions of processing units, and this number is still growing. Therefore, hardware failures, such as permanent node failures, become increasingly frequent. They can be tolerated with system-level Checkpoint/Restart, which saves the whole application state transparently and, if needed, restarts the application from the saved state; or with application-level checkpointing, which saves only relevant data via explicit calls in the program. The former approach requires no additional programming expense, whereas the latter is more efficient and allows to continue program execution after failures on the intact resources (localized shrinking recovery). An increasingly popular programming paradigm is asynchronous many-task (AMT) programming. Here, programmers identify parallel tasks, and a runtime system assigns the tasks to worker threads. Since tasks have clearly defined interfaces, the runtime system can automatically extract and save their interface data. This approach, called task-level checkpointing (TC), combines the respective strengths of system-level and application-level checkpointing. AMTs come in many variants, and so far, TC has only been applied to a few, rather simple variants. This paper considers TC for a different AMT variant: nested fork–join (NFJ) programs that run on clusters of multicore nodes under work stealing. We present the first TC scheme for this setting. It performs a localized shrinking recovery and can handle multiple node failures. In experiments with four benchmarks, we observed execution time overheads of around 44 % at 1536 workers, and negligible recovery costs. Additionally, we developed and experimentally validated a prediction model for the running times of the scheme.

BigDEC: A multi-algorithm Big Data tool based on the [formula omitted]-mer spectrum method for scalable short-read error correction

Despite the significant improvements in both throughput and cost provided by modern Next-Generation Sequencing (NGS) platforms, sequencing errors in NGS datasets can still degrade the quality of downstream analysis. Although state-of-the-art correction tools can provide high accuracy to improve such analysis, they are limited to apply a single correction algorithm while also requiring long runtimes when processing large NGS datasets. Furthermore, current parallel correctors generally only provide efficient support for shared-memory systems lacking the ability to scale out across a cluster of multicore nodes, or they require the availability of specific hardware devices or features. In this paper we present a Big Data Error Correction (BigDEC) tool that overcomes all those limitations by: (1) implementing three different error correction algorithms based on the widely extended k-mer spectrum method; (2) providing scalable performance for large datasets by efficiently exploiting the capabilities of Big Data technologies on multicore clusters based on commodity hardware; (3) supporting two different Big Data processing frameworks (Spark and Flink) to provide greater flexibility to end users; (4) including an efficient, stream-based merge operation to ease downstream processing of the corrected datasets; and (5) significantly outperforming existing parallel tools, being up to 79% faster on a 16-node multicore cluster when using the same underlying correction algorithm. BigDEC is publicly available to download at https://github.com/UDC-GAC/BigDEC.

Comparative analysis of MapReduce and Apache Tez Performance in Multinode clusters with data compression

This article conducts a thorough comparative analysis of Apache Tez and MapReduce in the context of big data processing. It focuses on key performance metrics, scalability, and ease of use. The analysis begins with an overview of the architectural distinctions between the two frameworks, emphasizing their fundamental design principles. A detailed performance evaluation follows, considering factors such as execution time, resource utilization, and throughput across diverse workloads. The study explores scalability by examining how Apache Tez and MapReduce respond to increasing data volumes and computational demands. Cluster size effects, resource allocation strategies, and adaptability to dynamic workloads are scrutinized. Additionally, the article evaluates the frameworks' ease of use for developers and administrators, incorporating aspects like programming model simplicity, debugging capabilities, and system configurability. User experiences are gathered through surveys and practical use cases. The conclusions drawn from this analysis offer valuable insights for organizations and practitioners seeking suitable distributed computing frameworks. By addressing both performance and user experience, the article aims to provide a comprehensive perspective on the strengths and weaknesses of Apache Tez and MapReduce, assisting decision-makers in making informed choices for their big data processing requirements.

Improving Data Processing Speed on Large Datasets in a Hadoop Multi-node Cluster using Enhanced Apriori Algorithm

For large data, data mining methods were used on a Hadoop-based distributed infrastructure, using map reduction paradigm approaches for rapid data processing. Though data mining approaches are established methodologies, the Apriori algorithm provides a specific strategy for increasing data processing performance in big data analytics by applying map reduction. Apriori property is used to increase the efficiency of level-wise creation of frequent itemsets by minimizing the search area. A frequent itemset’s subsets must also be frequent (Apriori property). If an itemset is rarely, then all of its supersets are infrequent as well. We refined the apriori approach by varying the degree of order in locating frequent item sets in large clusters using map reduction programming. Fixed Pass Combined Counting (FPC) and Dynamic Pass Combined Counting (DPC) is a classical algorithm which are used for data processing from the huge datasets but their accuracy is not up to the mark. In this article, updated Apriori algorithms such as multiplied-fixed-pass combined counting (MFPC) and average time-based dynamic combined counting (ATDFC) are used to successfully achieve data processing speed. The proposed approaches are based on traditional Apriori core notions in data mining and will be used in the map-reduce multi-pass phase by ignoring pruning in some passes. The optimized-MFPC and optimized-ATDFC map-reduce framework model algorithms were also presented. The results of the experiments reveal that MFPC and ATDFC are more efficient in terms of execution time than previously outmoded approaches such as Fixed Pass Combined Counting (FPC) and Dynamic Pass Combined Counting (DPC). In a Hadoop multi-node cluster, this paradigm accelerates data processing on big data sets. Previous techniques were stated in terms of reducing execution time by 60–80% through the use of several passes. Because of the omitted trimming operation in data pre-processing, our proposed new approaches will save up to 84–90% of that time.

A massively parallel implicit 3D unstructured grid solver for computing turbulent flows on latest distributed memory computational architectures

An implicit unstructured grid density-based solver–PRAVAHA based on a parallel variant of the lower upper symmetric Gauss-Seidel (LUSGS) method is developed to compute large-scale engineering problems. A four-layered parallel algorithm is designed to efficiently compute three-dimensional turbulent flows on massively parallel modern multiple instruction multiple data-stream (MIMD) computational hardware. This data parallel approach achieves multiple layers of parallelism including continuity of flow solution, transfer of solution gradients, and calculation of drag/lift/solution residuals, right up to the innermost implicit LUSGS solver sub-routine, which is relatively less explored in the literature. Domain decomposition is performed using the METIS software based on multi-level graph partitioning algorithms. Non-blocking message-passing interface library functions are used to manage inter-processor communication through explicit message passing, efficiently. Super-linear scalability of the parallel solver is established on the current state-of-the-art supercomputing facility, the 838 teraflops PARAM seva on up to 6144 cores. Linear or even super-linear speedup on problems of significant size is observed even on ad-hoc parallel computing platforms like workstations and multi-node clusters, for turbulent flow simulations.

Real-Time FaaS: Towards a Latency Bounded Serverless Cloud

Today, Function-as-a-Service is the most promising concept of serverless cloud computing. It makes possible for developers to focus on application development without any system management effort: FaaS ensures resource allocation, fast response time, schedulability, scalability, resiliency, and upgradability. Applications of 5G, IoT, and Industry 4.0 raise the idea to open cloud-edge computing infrastructures for time-critical applications too, i.e., there is a strong desire to pose real-time requirements for computing systems like FaaS. However, multi-node systems make real-time scheduling significantly complex since guaranteeing real-time task execution and communication is challenging even on one computing node with multi-core processors. In this paper, we present an analytical model and a heuristic partitioning scheduling algorithm suitable for real-time FaaS platforms of multi-node clusters. We show that our task scheduling heuristics could outperform existing algorithms by 55%. Furthermore, we propose three conceptual designs to enable the necessary real-time communications. We present the architecture of the envisioned real-time FaaS platform, emphasize its benefits and the requirements for the underlying network and nodes, and survey the related work that could meet these demands.

NoSQL Databases: Modern Data Systems for Big Data Analytics

Because of the massive utilization of the world wide web and the drastic use of electronic gadgets to access the online world, there is an exponential growth in the information produced by these hardware gadgets. The data produced by different sources, such as smart transportation, healthcare, and e-commerce, are large, complex, and heterogeneous. Therefore, storing and querying this data, coined "Big Data," is challenging. This paper compares relational databases with a few of the popular NoSQL databases. The performance of various databases in executing join queries, filter queries, and aggregate queries on large datasets are compared on a single node and multinode clusters. The experimental results demonstrate the suitability of NoSQL databases for Big Data Analytics and for supporting large userbase interactive web applications.

Research on the Routing Method of Swarm Network Based on Moth Search Link Distance and Residual Energy Weights

Routing method is a key technology for swarm communication. In order to reduce the node energy consumption in large-scale swarm communication, a routing method based on moth search link distance and residual energy is proposed. First, the swarm network is divided into sub-cluster networks by filtering of cluster head nodes. The node with more residual energy is used as cluster head to forward data signals from other nodes. The link between cluster head and edge nodes is established by relaying the signals from the nodes. The routing function of the link is established by calculating the distance and residual energy weight of the nodes. During cluster head broadcast communication, a modified moth-based search algorithm is applied to solve the combined weights of the routes. The optimal solution of the function constitutes the best route within the subcluster. Simulation results show that the proposed method meets the communication requirements, while the performance of the proposed method is higher than other mainstream algorithms by comparing the magnitude of three metrics: network coverage, residual energy and BER. The proposed routing method reduces the energy consumption of multi-node cluster network communication. The method achieves routing for large-scale cluster communication.

Image mining technique using Hadoop map reduce over distributed multi-node computers connections

The amount of today's image collections has exploded to petabytes of data. A fair amount of time cannot be allotted for the analysis of such massive datasets using a personal computer. As a result, distributed computing is required for current image collection mining. This work used multi-nodes-computers for image mining in order to improve reliability, fall tolerance, and time efficiency. Because of this, the data was divided up across the nodes in the Hadoop multi-node cluster, and the results were then compiled to create an image clustering algorithm. One master node and two slave nodes were used to test our technique on a huge dataset. Using multi-node Hadoop, we found that we could get a speed-up in implementing as high as a single-node Hadoop implementation.

Image mining technique using Hadoop map reduce over distributed multi-node computers connections

The amount of today's image collections has exploded to petabytes of data. A fair amount of time cannot be allotted for the analysis of such massive datasets using a personal computer. As a result, distributed computing is required for current image collection mining. This work used multi-nodes-computers for image mining in order to improve reliability, fall tolerance, and time efficiency. Because of this, the data was divided up across the nodes in the Hadoop multi-node cluster, and the results were then compiled to create an image clustering algorithm. One master node and two slave nodes were used to test our technique on a huge dataset. Using multi-node Hadoop, we found that we could get a speed-up in implementing as high as a single-node Hadoop implementation.

TOP-DOWN AND BOTTOM-UP APPROACH FOR MINING MULTILEVEL ASSOCIATION RULES FROM CONCEPT HIERARCHICAL DATA IN DISTRIBUTED ENVIRONMENT

Hierarchical Data mining using distributed environment is an imperative in big data analysis. Multilevel association rules can provide more substantial information than single level rules, and it also determines hierarchical knowledge from the dataset. Nowadays, numerous e-commerce and social networking sites generates vast amount of structural/semi-structural data in the form of sales data, tweets, text mails, web usages and so on. The data generated from such sources is so large that it becomes very difficult to process and analyze it using conventional approaches. This paper overcomes the computing limitation of single node by distributing the task on multi-node cluster. The performance of this system is compared based on minimum support threshold at diverse levels of concept hierarchy and by varying the dataset size. In this paper, the transactional dataset is created from huge sales dataset using Hadoop MapReduce framework. Then, two distributed multilevel frequent pattern mining algorithms MR-MLAB (MapReduce based Multilevel Apriori using Bottom-up approach) and MR-MLAT (MapReduce based Multilevel Apriori using Top-down approach) are implemented to find interesting level-crossing frequent itemset for each level of concept hierarchy. The hierarchical redundancy in multilevel association rules affects the quality of the market basket analysis. Hence, to improve the performance of the system, the hierarchical redundancy has to be removed from it. Finally, the time efficiency of proposed algorithms is compared with existing Traditional Multilevel Apriori (TMLA) Algorithm. The proposed algorithms with MapReduce framework are found efficient compared to the traditional algorithms.

A System-on-Chip Based Hybrid Neuromorphic Compute Node Architecture for Reproducible Hyper-Real-Time Simulations of Spiking Neural Networks.

Despite the great strides neuroscience has made in recent decades, the underlying principles of brain function remain largely unknown. Advancing the field strongly depends on the ability to study large-scale neural networks and perform complex simulations. In this context, simulations in hyper-real-time are of high interest, as they would enable both comprehensive parameter scans and the study of slow processes, such as learning and long-term memory. Not even the fastest supercomputer available today is able to meet the challenge of accurate and reproducible simulation with hyper-real acceleration. The development of novel neuromorphic computer architectures holds out promise, but the high costs and long development cycles for application-specific hardware solutions makes it difficult to keep pace with the rapid developments in neuroscience. However, advances in System-on-Chip (SoC) device technology and tools are now providing interesting new design possibilities for application-specific implementations. Here, we present a novel hybrid software-hardware architecture approach for a neuromorphic compute node intended to work in a multi-node cluster configuration. The node design builds on the Xilinx Zynq-7000 SoC device architecture that combines a powerful programmable logic gate array (FPGA) and a dual-core ARM Cortex-A9 processor extension on a single chip. Our proposed architecture makes use of both and takes advantage of their tight coupling. We show that available SoC device technology can be used to build smaller neuromorphic computing clusters that enable hyper-real-time simulation of networks consisting of tens of thousands of neurons, and are thus capable of meeting the high demands for modeling and simulation in neuroscience.

A Comprehensive Spark-Based Layer for Converting Relational Databases to NoSQL

Currently, the continuous massive growth in the size, variety, and velocity of data is defined as big data. Relational databases have a limited ability to work with big data. Consequently, not only structured query language (NoSQL) databases were utilized to handle big data because NoSQL represents data in diverse models and uses a variety of query languages, unlike traditional relational databases. Therefore, using NoSQL has become essential, and many studies have attempted to propose different layers to convert relational databases to NoSQL; however, most of them targeted only one or two models of NoSQL, and evaluated their layers on a single node, not in a distributed environment. This study proposes a Spark-based layer for mapping relational databases to NoSQL models, focusing on the document, column, and key–value databases of NoSQL models. The proposed Spark-based layer comprises of two parts. The first part is concerned with converting relational databases to document, column, and key–value databases, and encompasses two phases: a metadata analyzer of relational databases and Spark-based transformation and migration. The second part focuses on executing a structured query language (SQL) on the NoSQL. The suggested layer was applied and compared with Unity, as it has similar components and features and supports sub-queries and join operations in a single-node environment. The experimental results show that the proposed layer outperformed Unity in terms of the query execution time by a factor of three. In addition, the proposed layer was applied to multi-node clusters using different scenarios, and the results show that the integration between the Spark cluster and NoSQL databases on multi-node clusters provided better performance in reading and writing while increasing the dataset size than using a single node.

Ubiquitous Power Internet of Things-Oriented Low-Latency Edge Task Scheduling Optimization Strategy

Internet of things of cloud computing offers high-performance computing, storage and networking services, but there are still suffers from a high transmission and processing latency, poor scalability and other problems. Internet of things of edge computing can better meet the increasing requirements of electricity consumers for service quality, especially the increasingly stringent need for low delay. On the other hand, edge intelligent network technology can offers edge smart sensing while significantly improve the efficiency of task execution, but it will lead to a massive collaborative task scheduling optimization problem. In order to solve this problem, This paper studies an ubiquitous power internet of things (UPIoT) smart sensing network edge computing model and an improved multi node cluster cooperative scheduling optimization strategy. The cluster server is added to the edge aware computing network, and an improved low delay edge task collaborative scheduling algorithm (LLETCS) is designed by using the vertical cooperation and multi node cluster collaborative computing scheme between edge aware networks. Then the problem is transformed based on linear reconstruction technology, and a parallel optimization framework for solving the problem is proposed. The simulation results suggest that the proposed scheme can more effectively reduce the UPIoT edge computing latency, and improve the quality of service in UPIoT smart sensing networks.

A Hybrid MPI-OpenMP Parallel Algorithm for the Assessment of the Multifractal Spectrum of River Networks

The possibility to create a flood wave in a river network depends on the geometric properties of the river basin. Among the models that try to forecast the Instantaneous Unit Hydrograph (IUH) of rainfall precipitation, the so-called Multifractal Instantaneous Unit Hydrograph (MIUH) rather successfully connects the multifractal properties of the river basin to the observed IUH. Such properties can be assessed through different types of analysis (fixed-size algorithm, correlation integral, fixed-mass algorithm, sandbox algorithm, and so on). The fixed-mass algorithm is the one that produces the most precise estimate of the properties of the multifractal spectrum that are relevant for the MIUH model. However, a disadvantage of this method is that it requires very long computational times to produce the best possible results. In a previous work, we proposed a parallel version of the fixed-mass algorithm, which drastically reduced the computational times almost proportionally to the number of Central Processing Unit (CPU) cores available on the computational machine by using the Message Passing Interface (MPI), which is a standard for distributed memory clusters. In the present work, we further improved the code in order to include the use of the Open Multi-Processing (OpenMP) paradigm to facilitate the execution and improve the computational speed-up on single processor, multi-core workstations, which are much more common than multi-node clusters. Moreover, the assessment of the multifractal spectrum has also been improved through a direct computation method. Currently, to the best of our knowledge, this code represents the state-of-the-art for a fast evaluation of the multifractal properties of a river basin, and it opens up a new scenario for an effective flood forecast in reasonable computational times.

Deploying and scaling distributed parallel deep neural networks on the Tianhe-3 prototype system

Due to the increase in computing power, it is possible to improve the feature extraction and data fitting capabilities of DNN networks by increasing their depth and model complexity. However, the big data and complex models greatly increase the training overhead of DNN, so accelerating their training process becomes a key task. The Tianhe-3 peak speed is designed to target E-class, and the huge computing power provides a potential opportunity for DNN training. We implement and extend LeNet, AlexNet, VGG, and ResNet model training for a single MT-2000+ and FT-2000+ compute nodes, as well as extended multi-node clusters, and propose an improved gradient synchronization process for Dynamic Allreduce communication optimization strategy for the gradient synchronization process base on the ARM architecture features of the Tianhe-3 prototype, providing experimental data and theoretical basis for further enhancing and improving the performance of the Tianhe-3 prototype in large-scale distributed training of neural networks.

MPI-dot2dot: A parallel tool to find DNA tandem repeats on multicore clusters

Tandem Repeats (TRs) are segments that occur several times in a DNA sequence, and each copy is adjacent to other. In the last few years, TRs have gained significant attention as they are thought to be related with certain human diseases. Therefore, identifying and classifying TRs have become a highly important task in bioinformatics in order to analyze their disorders and relationships with illnesses. Dot2dot, a tool recently developed to find TRs, provides more accurate results than the previous state-of-the-art, but it requires a long execution time even when using multiple threads. This work presents MPI-dot2dot, a novel version of this tool that combines MPI and OpenMP so that it can be executed in a cluster of multicore nodes and thus reduces its execution time. The performance of this new parallel implementation has been tested using different real datasets. Depending on the characteristics of the input genomes, it is able to obtain the same biological results as Dot2dot but more than 100 times faster on a 16-node multicore cluster (384 cores). MPI-dot2dot is publicly available to download from https://sourceforge.net/projects/mpi-dot2dot.

Hybrid parallel iterative sparse linear solver framework for reservoir geomechanical and flow simulation

We discuss new developments of a hybrid parallel iterative sparse linear solver framework focused on petroleum reservoir flow and geomechanical simulation. It runs efficiently on several platforms, from desktop workstations to clusters of multicore nodes, with or without multiple GPUs, using a two-tier hierarchical architecture for distributed matrices and vectors. Results show good parallel scalability. Comparisons with a well-established library and a proprietary commercial solver indicate that our solver is competitive with the best available tools. We present results of the solver's application to simulations of real and synthetic reservoir models of up to billions of unknowns, running on CPUs and GPUs on up to 2000 processes.

HADOOP: A Comparative Study between Single-Node and Multi-Node Cluster

Data analysis has become a challenge in recent years as the volume of data generated has become difficult to manage, therefore more hardware and software resources are needed to store and process this huge amount of data. Apache Hadoop is a free framework, widely used thanks to the Hadoop Distributed Files System (HDFS) and its ability to relate to other data processing and analysis components such as MapReduce for processing data, Spark - in-memory Data Processing, Apache Drill - SQL on Hadoop, and many other. In this paper, we analyze the Hadoop framework implementation making a comparative study between Single-node and Multi-node cluster on Hadoop. We will explain in detail the two layers at the base of the Hadoop architecture: HDFS Layer with its deamons NameNode, Secondary NameNode, DataNodes and MapReuce Layer with JobTrackers, TaskTrackers daemons. This work is part of a complex one aiming to perform data processing in Data Lake structures.