Cluster Computer System Research Articles

АНАЛИЗ ВОЗМОЖНОСТЕЙ СОВРЕМЕННЫХ ВЫЧИСЛИТЕЛЬНЫХ ТЕХНОЛОГИЙ ДЛЯ РАСЧЕТА ТОЧНЫХ ПРИБЛИЖЕНИЙ РАСПРЕДЕЛЕНИЙ ВЕРОЯТНОСТЕЙ ЗНАЧЕНИЙ СТАТИСТИК

In the paper we consider the solution of a computationally expensive problem such as calculationof statistics probability distribution with the help of modern computer technologies. To reducecomputational complexity and to provide a sufficient level of criteria efficiency not less thanthe specified threshold, we suggest to use Δ-exact approximations. To calculate exact approximations,we use the method of second order, based on solution of a system of linear equations. Owingto this method, it is possible to calculate exact approximations for the maximum values of sampleparameters for available computational resource. The most laborious part of the method of secondorder is the procedure of sequential detection of the vectors of possible solutions and test if thevectors belong to the set of solutions. The system solution set membership test for the vectors ofpossible solutions is data independent, so the algorithm can be data-parallelized. We give the algorithmcomplexity equation for calculation of exact approximations of statistics probability distributions.Using this equation, we calculated the complexity of modern practical problems for thesamples with the parameters (N, n) of the alphabet power and the sample size: (256,1280),(128,640), (128, 320), and (192,3200) for the accuracy of calculations =10-5. The computationalcomplexity is 9.68·1022-1.60·1052 operations, and its average value is about 4.55·1025 operations,the number of tested vectors is 6.50·1023-1.39·1050, and the number of solutions is 4.67·1012-5.60·1025, respectively. The total solution time for clock-round duration of calculations cannotexceed 30 days or 2.592·106 sec. For the obtained complexity evaluation, we analysed abilities ofmodern cluster computer systems based on general-purpose processors, graphic accelerators, andFPGA-based reconfigurable computer systems. For each technology, we determined the number ofcomputational nodes needed for calculation of exact approximations with the specified parametersduring the specified time. We proved that it is impossible to obtain a solution for the required parametersof exact approximations of statistics probability with the help of the reviewed moderncomputer technologies. In conclusion, we claim that it is necessary to analyse the abilities of advancedcomputer technologies based of quantum and photonic computers, and also hybrid computersystems for calculation of exact approximations of statistics probability distributions withthe specified parameters during reasonable time.

HIGH PERFORMANCE PROCESSING CLUSTER FOR REMOTE SENSING SPACECRAFTS

Recently, there has been a tendency to complicate real time control algorithms, to process a large amount of information from the satellite payload (optical, radar, communication systems) directly on board and to create highly informative communication lines with both ground stations and other satellites. The conditions of outer space (vacuum, radiation) impose significant restrictions on the computing capabilities of onboard equipment, which is used as an onboard digital computer complex, and lead to a significant increasing in the cost of space components. A high-performance and relatively cheap computing cluster structure of an on-board computer based on widely available single-board minicomputers is proposed, which allows distributing the computing load among several nodes and simultaneously backing up the system. As a component of a computing cluster, it is proposed to use a computing cluster system based on COTS (Commercial off-the-shelf) components, which increases performance by several orders while reducing cost. Calculations have shown that the introduction of redundancy and distribution of computational tasks makes it possible to achieve an MTBF of about 3 years, which is quite enough for the active existence of university satellites. The proposed structure of the onboard computer complex is installed on the university satellite for remote sensing of the Earth in the visible range with controlled optical magnification, after the launch of which it is planned to confirm the reliability of the results obtained in this work. An assessment of the performance and reliability of such a cluster system is given, which has shown the possibility of implementing such a system on a university satellite for Earth remote sensing.

Performance Evaluation of Parallel Structure from Motion (SfM) Processing with Public Cloud Computing and an On-Premise Cluster System for UAS Images in Agriculture

Thanks to sensor developments, unmanned aircraft system (UAS) are the most promising modern technologies used to collect imagery datasets that can be utilized to develop agricultural applications in these days. UAS imagery datasets can grow exponentially due to the ultrafine spatial and high temporal resolution capabilities of UAS and sensors. One of the main obstacles to processing UAS data is the intensive computational resource requirements. The structure from motion (SfM) is the most popular algorithm to generate 3D point clouds, orthomosaic images, and digital elevation models (DEMs) in agricultural applications. Recently, the SfM algorithm has been implemented in parallel computing to process big UAS data faster for certain applications. This study evaluated the performance of parallel SfM processing on public cloud computing and on-premise cluster systems. The UAS datasets collected over cropping fields were used for performance evaluation. We used multiple computing nodes and centralized network storage with different network environments for the SfM workflow. In single-node processing, an instance with the most computing power in the cloud computing system performed approximately 20 and 35 percent faster than in the most powerful machine in the on-premises cluster. The parallel processing results showed that the cloud-based system performed better in speed-up and efficiency metrics for scalability, although the absolute processing time was faster in the on-premise cluster. The experimental results also showed that the public cloud computing system could be a good alternative computing environment in UAS data processing for agricultural applications.

HPC cluster-based user-defined data integration platform for deep learning in geoscience applications

Objective:Efficient and flexible data integration platforms are important to apply various geoscience data for deep learning applications. Recently, deep learning techniques have been applied to analyze and predict natural phenomena in Earth sciences using geoscience data. Because geoscience data are considered as “big data”, data driven approaches such as deep learning are promising tools for understanding natural phenomena. In this paper, we propose a Geoscience Data Integration Platform (GeoDIP) for managing big geoscience data based on High-performance Computing (HPC) cluster systems. Methodology:GeoDIP provides data pre-processing and analyzes modules according to user defined configurations when creating Artificial Intelligence (AI) ready datasets. To determine the application of GeoDIP, we demonstrated precipitation prediction performance using multiple datasets. We collected three datasets from different sources: two from satellite observations and one from reanalysis data for weather forecasting. We then compared the results obtained for each dataset and the results obtained for integrated datasets. Results:The results confirmed that the integrated dataset generated from GeoDIP provided 9% improved prediction performance for F1-score over 2 h. This suggests that various types of information on atmospheric conditions explaining precipitation genesis from multiple data sources are crucial for precipitation prediction. For understanding model performance, we conducted a permutation-based feature importance test, which confirmed that the upper level information is important over time. In addition, we evaluated the performance of GeoDIP by comparing sequential and parallel tasks and obtained a performance improvement of approximately 97%. Conclusions:The proposed geoDIP facilitates the utilization of multiple datasets to analyze geophysical phenomena using parallel processes of HPC clusters with reduced computational time for data pre-processing and analysis.

MATHEMATICAL MODELS PRODUCTIVITY OF CLUSTER SYSTEM BASED ON RASPBERRY PI 3B+

Context. High-performance computing systems are needed to solve many scientific problems and to work with complex applied problems. Previously, real parallel data processing was supported only by supercomputers, which are very limited and difficult to access. Currently, one way to solve this problem is to build small, cheap clusters based on single-board computers Raspberry Pi.
 Objective. The goal of the work is the creation of a complex criterion for the efficiency of the cluster system, which could properly characterize the operation of such a system and find the dependences of the performance of the cluster system based on Raspberry Pi 3B+ on the number of boards in it with different cooling systems.
 Method. It is offered to apply in the analysis of small cluster computer systems the complex criterion of efficiency of work of cluster system which will consider the general productivity of cluster computer system, productivity of one computing element in cluster computer system, electricity consumption by cluster system, electricity consumption per one computing element, the cost of calculating 1 Gflops cluster computer system, the total cost of the cluster computer system.
 Results. The developed complex criterion of cluster system efficiency was used to create an experimental cluster system based on single-board computers Raspberry Pi 3B+. Mathematical models of the dependence of the performance of a small cluster system based on single-board computers Raspberry Pi 3B+ depending on the number of boards in it with different cooling systems have also been developed.
 Conclusions. The conducted experiments confirmed the expediency of using the developed complex criterion of efficiency of the cluster system and allow to recommend it for use in practice when creating small cluster systems. Prospects for further research are to determine the weights of the constituent elements of the complex criterion of efficiency of the cluster system, as well as in the experimental study of the proposed weights.

Disaggregated Accelerator Management System for Cloud Data Centers

A conventional data center that consists of monolithic-servers is confronted with limitations including lack of operational flexibility, low resource utilization, low maintainability, etc. Resource disaggregation is a promising solution to address the above issues. We propose a concept of disaggregated cloud data center architecture called Flow-in-Cloud (FiC) that enables an existing cluster computer system to expand an accelerator pool through a high-speed network. FlowOS-RM manages the entire pool resources, and deploys a user job on a dynamically constructed slice according to a user request. This slice consists of compute nodes and accelerators where each accelerator is attached to the corresponding compute node. This paper demonstrates the feasibility of FiC in a proof of concept experiment running a distributed deep learning application on the prototype system. The result successfully warrants the applicability of the proposed system.

An automated fault detection system for communication networks and distributed systems

Automating fault detection in communication networks and distributed systems is a challenging process that usually requires the involvement of supporting tools and the expertise of system operators. Automated event monitoring and correlating systems produce event data that is forwarded to system operators for analyzing error events and creating fault reports. Machine learning methods help not only analyzing event data more precisely but also forecasting possible error events by learning from existing faults. This study introduces an automated fault detection system that assists system operators in detecting and forecasting faults. This system is characterized by the capability of exploiting bug knowledge resources at various online repositories, log events and status parameters from the monitored system; and applying bug analysis and event filtering methods for evaluating events and forecasting faults. The system contains a fault data model to collect bug reports, a feature and semantic filtering method to correlate log events, and machine learning methods to evaluate the severity, priority and relation of log events and forecast the forthcoming critical faults of the monitored system. We have evaluated the prototyping implementation of the proposed system on a high performance computing cluster system and provided analysis with lessons learned.

SW‐LZMA: Parallel Implementation of LZMA Based on SW26010 Many‐Core Processor

With the development of high‐performance computing and big data applications, the scale of data transmitted, stored, and processed by high‐performance computing cluster systems is increasing explosively. Efficient compression of large‐scale data and reducing the space required for data storage and transmission is one of the keys to improving the performance of high‐performance computing cluster systems. In this paper, we present SW‐LZMA, a parallel design and optimization of LZMA based on the Sunway 26010 heterogeneous many‐core processor. Combined with the characteristics of SW26010 processors, we analyse the storage space requirements, memory access characteristics, and hotspot functions of the LZMA algorithm and implement the thread‐level parallelism of the LZMA algorithm based on Athread interface. Furthermore, we make a fine‐grained layout of LDM address space to achieve DMA double buffer cyclic sliding window algorithm, which optimizes the performance of SW‐LZMA. The experimental results show that compared with the serial baseline implementation of LZMA, the parallel LZMA algorithm obtains a maximum speedup ratio of 4.1 times using the Silesia corpus benchmark, while on the large‐scale data set, speedup is 5.3 times.

Performance Comparison of Python and Scala APIs in Spark Distributed Cluster Computing System

Performance Comparison of Python and Scala APIs in Spark Distributed Cluster Computing System

Organization of cluster systems based on the PCI Express switching environment

This article is devoted to the organization of computing cluster systems that have a limit on the maximum distance between computing nodes and data storage nodes. Known options of building clusters based on the PCI Express switching environment are considered as the most.

Dynamic Traffic Control of Staging Traffic on the Interconnect of the HPC Cluster System

High-performance computing (HPC) cluster systems sometimes adopt a two-layered file system composed of local and global file systems to achieve both capacity and performance in storage. In such a cluster system, the input data of an application needs to be staged from the global storage into the local storage, and the output data needs to be staged from the local storage out to the global storage. This staging operation must be efficiently and quickly performed to gain higher job throughput because an inefficient staging operation prevents waiting job requests from being executed. In particular, in the case of the cluster system with the oversubscribed interconnect shared by the storage and the computing nodes, the inter-node communication and this staging operation traffic collides, which may degrade the job throughput. In this research, we focus on the traffic collision of the inter-node communication and the staging traffic to improve job throughput, targeting the cluster system with the oversubscribed interconnect where these two types of traffic flow. In other words, whether the dynamic control of the traffic flow derived from the staging operation leads to the improvement in the job throughput or not is investigated. For the investigation, we present a traffic collision avoidance method to dynamically configure a set of data paths for each type of the traffic only while the staging operation is conducted. The evaluation in this article shows that the proposed method avoids a traffic collision and accelerates the staging operation by 22.0% on our cluster system. Also, this evaluation indicates the overhead of the application incurred by the proposed method is negligible. Furthermore, 8.7% of the job execution time is reduced by the proposed method.

An enhanced cuckoo optimization algorithm for task graph scheduling in cluster-computing systems

Optimized task scheduling is key to achieve high performance in the cluster-computing systems whose application is broad ranging from scientific to the military purposes. This combinatorial problem is NP-hard from the time complexity perspective, where applying newly proposed metaheuristics to it deserves further investigation based on the well-known no-free-lunch theorem. Accordingly, in this paper, an enhanced version of cuckoo optimization algorithm (COA) named E-COA is proposed to cope with the static task scheduling problem in the mesh topology cluster-computing environments. The proposed approach is equipped with an efficient adaptive semi-stochastic egg-laying strategy that significantly improves the local and global search potentiality of the basic COA. The experiments on a comprehensive set of randomly generated task graphs with different structural parameters reveal the efficiency of the proposed approach from the performance point of view, especially for the small-scale samples, and where the number of clusters in the machine is very restricted, i.e., we are in the lack of computational resource.

Enhanced Parallel Application Scheduling Algorithm with Energy Consumption Constraint in Heterogeneous Distributed Systems

Energy consumption has always been one of the main design problems in heterogeneous distributed systems, whether for large cluster computer systems or small handheld terminal devices. And as energy consumption explodes for complex performance, many efforts and work are focused on minimizing the schedule length of parallel applications that meet the energy consumption constraints currently. In prior studies, a pre-allocation method based on dynamic voltage and frequency scaling (DVFS) technology allocates unassigned tasks with minimal energy consumption. However, this approach does not necessarily result in minimal scheduling length. In this paper, we propose an enhanced scheduling algorithm, which allocates the same energy consumption for each task by selecting a relatively intermediate value among the unequal allocations. Based on the two real-world applications (Fast Fourier transform and Gaussian elimination) and the randomly generated parallel application, experiments show that the proposed algorithm not only achieves better scheduling length while meeting the energy consumption constraints, but also has better performance than the existing parallel algorithms.

Random forest implementation and optimization for Big Data analytics on LexisNexis\u2019s high performance computing cluster platform

In this paper, we comprehensively explain how we built a novel implementation of the Random Forest algorithm on the High Performance Computing Cluster (HPCC) Systems Platform from LexisNexis. The algorithm was previously unavailable on that platform. Random Forest’s learning process is based on the principle of recursive partitioning and although recursion per se is not allowed in ECL (HPCC’s programming language), we were able to implement the recursive partition algorithm as an iterative split/partition process. In addition, we analyze the flaws found in our initial implementation and we thoroughly describe all the modifications required to overcome the bottleneck within the iterative split/partition process, i.e., the optimization of the data gathering of selected independent variables which are used for the node’s best-split analysis. Essentially, we describe how our initial Random Forest implementation has been optimized and has become an efficient distributed machine learning implementation for Big Data. By taking full advantage of the HPCC Systems Platform’s Big Data processing and analytics capabilities, we succeed in enhancing the data gathering method from an inefficient Pass them All and Filter approach into an effective and completely parallelized Fetching on Demand approach. Finally, based upon the results of our learning process runtime comparison between these two approaches, we confirm the speed up of our optimized Random Forest implementation.

Block Partitioning and Merging for Processing Large-Scale Structure From Motion Problems in Distributed Manner

The processing time of incremental structure from motion increases exponentially with the number of images. As a result, a huge amount of time is needed for large datasets. In this paper, to improve time efficiency, a block partitioning and a merging strategy are proposed. We automatically split the image set into several overlapping subsets, and then each subset can be processed in parallel. Finally, the reconstruction results of each subset can be merged together according to the shared images and tie points. The image adjacency matrix obtained from the feature matching result is the input of our block partitioning algorithm. And by repeatedly using the matrix bandwidth reduction algorithm to reorder the images, the block can be partitioned into subsets. The partitioning result is satisfactory, namely, images assigned into a subset have a very strong connection, and the shape of each subset is compact. Most importantly, the algorithm is very simple and fast. We have successfully processed many large-scale aerial image datasets in a computer cluster system with 10 processing nodes. And, the time efficiency and the precision of the reconstruction are satisfactory.

Automating Account Management on an Academic HPC System with Authentication Federation

There is increasing demand for High-Performance Computing (HPC) cluster systems because more and more computational resources are necessary to carry out large-scale simulations and machine learning which are a crucial part of research and business today. There are two choices of HPC cluster systems: academic HPC cluster systems and public cloud services. Academic HPC cluster systems can be used with reasonable usage fees, although they take a long time until the cluster system is ready to use, because of its user registration procedure. On the other hand, public cloud services, such as Amazon Web Services, can be used immediately, but the utilization fees are expensive compared with those for academic systems. To fill the gaps between them, we propose an academic HPC cluster system with automated user registration that allows users to utilize the HPC cluster on demand.

Minimal re-computation for exploratory data analysis in astronomy

Abstract We present a technique to automatically minimise the re-computation when a data analysis program is iteratively changed, or added to, as is often the case in exploratory data analysis in astronomy. A typical example is flagging and calibration of demanding or unusual observations where visual inspection suggests improvement to the processing strategy. The technique is based on memoization and referentially transparent tasks. We describe the implementation of this technique for the CASA radio astronomy data reduction package. We also propose a technique for optimising efficiency of storage of memoized intermediate data products using copy-on-write and block level de-duplication and measure their practical efficiency. We find that the minimal recomputation technique improves the efficiency of data analysis while reducing the possibility for user error and improving the reproducibility of the final result. It also aids exploratory data analysis on batch-schedule cluster computer systems.

Mathematical Simulation of Vortex Structures in Rapidly Rotating Astrophysical Objects

A conservative finite-volume method is used to model rapidly rotating self-gravitating astrophysical objects. Large-scale vortex structures are obtained in evolutionary computations. The numerical simulation is based on parallel algorithms implemented on a cluster computer system.

Autonomous multimedia cluster computing based on Cooperative Cognition data behavior measurement under multi cloud computing

In order to meet the needs of multimedia communication in multi cloud environment and improve the experience quality of mobile multimedia users, based on multi cloud computing, based on Cooperative cognitive data behavior measurement, this paper proposes an autonomous multimedia cluster computer system and its architecture. First of all, dispersed edge clouds are distributed and integrated, and cooperate to provide multimedia storage and computing functions. In cloudy environment, a hierarchical access service point is designed between edge cloud and core cloud. On this basis, a multimedia cluster computing system suitable for cloudy environment is built. Secondly, a mulch-dimensional mapping mechanism is built between the link management interface array and the task scheduling array in the edge cloud array. Mulch-dimensional multimedia data and real-time task scheduling cooperation cognition, and core cloud are used to interact with DP vectors with dedicated channels. On this basis, we propose an autonomous computer system based on Cooperative Cognition and multimedia data behavior measure. Finally, it is analyzed by three groups of experiments. The resource utilization of the multimedia cluster computing system, the behavior measurement accuracy based on the cooperative cognitive multimedia data behavior measure and the performance of the proposed autonomous multimedia cluster computing (AMC-CCC) in large-scale real-time multimedia communication applications. The results show that the proposed AMC-CCC mechanism has excellent performance in multimedia QoS, resource management and data behavior measurement.

Forward Observer system for radar data workflows: Big data management in the field

There are unique challenges in managing data collection and management from instruments in the field in general. These issues become extreme when “in the field” means “in a plane over the Antarctic”. In this paper we present the design and function of the Forward Observer a computer cluster and data analysis system that flies in a plane in the Arctic and Antarctic to collect, analyze in real time, and store Synthetic Aperture Radar (SAR) data. SAR is used to analyze the thickness and structure of polar ice sheets. We also discuss the processing of data once it is returned to the continental US and made available via data grids. The needs for in-flight data analysis and storage in the Antarctic and Arctic are highly unusual, and we have developed a novel system to meet those needs.We describe the constraints and requirements that led to the creation of this system and the general functionality which it applies to any instrument. We discuss the main means for handling replication and creating checksum information to ensure that data collected in polar regions are returned safely to mainland US for analysis. So far, not a single byte of data collected in the field has failed to make it home to the US for analysis (although many particular data storage devices have failed or been damaged due to the challenges of the extreme environments in which this system is used).While the Forward Observer system is developed for the extreme situation of data management in the field in the Antarctic, the technology and solutions we have developed are applicable and potentially usable in many situations where researchers wish to do real time data management in the field in areas that are constrained in terms of electrical supply.