Job Submission Research Articles

High Throughput Calculation and Characteristic Structure Selection of Cu-Cr-Sn Copper Alloy Based on Matclould Platform

In this paper, high-throughput crystal structure modeling and high-throughput screening optimization algorithm are used to optimize the performance parameters such as characteristic microstructure type, generation conditions, strength and conductivity. The theoretical criterion of multi parameter collaborative optimization and control of key properties of matrix and second phase is established, based on which the optimal composition range of matrix and second phase is determined, which guides high-throughput experimental verification and material screening, and provides basic data for material composition design and structure control. Realize 102 secondary high-throughput parallel computing and screening based on matcloud platform, and calculate and screen the number of candidate material samples ≧106 and other related technologies. Through the functions of online generation, submission and job monitoring of first principle high-throughput jobs, all computing processes are realized automatically, and the computing results are extracted, standardized processed and stored in the database according to certain rules. In this project, the prediction model and process design platform of material composition structure process performance relationship are established according to the dynamic selection parameters of copper alloy structure, pseudopotential recommendation, k-point file generated according to calculation tasks and structure, and the correlation of some other main parameters.

Improvements to Supercomputing Service Availability Based on Data Analysis

As the demand for high-performance computing (HPC) resources has increased in the field of computational science, an inevitable consideration is service availability in large cluster systems such as supercomputers. In particular, the factor that most affects availability in supercomputing services is the job scheduler utilized for allocating resources. Consequent to submitting user data through the job scheduler for data analysis, 25.6% of jobs failed because of program errors, scheduler errors, or I/O errors. Based on this analysis, we propose a K-hook method for scheduling to increase the success rate of job submissions and improve the availability of supercomputing services. By applying this method, the job-submission success rate was improved by 15% without negatively affecting users’ waiting time. We also achieved a mean time between interrupts (MTBI) of 24.3 days and maintained average system availability at 97%. As this research was verified on the Nurion supercomputer in a real service environment, the value of the research is expected to be found in significant service improvements.

WebMO: Web‐based computational chemistry calculations in education and research

AbstractWebMO is a web‐based interface for all major quantum chemistry programs. WebMO uses a server–client architecture that installs on a single server or cluster computer and provides access to state‐of‐the‐art computational chemistry programs from a standard web browser. The web interface provides a 3‐D molecular editor, pre‐defined calculations types, job submission and monitoring, visualization of results, and user management tools. Barriers to using state‐of‐the‐art computational chemistry in teaching and research are minimized through WebMO's universal accessibility, its intuitive and uniform interface to all programs, no software to install on client computers, and support for multiple users with a single instance. Applications of WebMO throughout the undergraduate curriculum are provided. The extensible open‐architecture design allows for collaboration among educators, researchers, quantum chemistry program developers, and the WebMO interface developers.This article is categorized under: Computer and Information Science > Visualization Electronic Structure Theory > Ab Initio Electronic Structure Methods Software > Quantum Chemistry

Research on the Revolution of Multidimensional Learning Space in the Big Data Environment

Multiuser fair sharing of clusters is a classic problem in cluster construction. However, the cluster computing system for hybrid big data applications has the characteristics of heterogeneous requirements, which makes more and more cluster resource managers support fine-grained multidimensional learning resource management. In this context, it is oriented to multiusers of multidimensional learning resources. Shared clusters have become a new topic. A single consideration of a fair-shared cluster will result in a huge waste of resources in the context of discrete and dynamic resource allocation. Fairness and efficiency of cluster resource sharing for multidimensional learning resources are equally important. This paper studies big data processing technology and representative systems and analyzes multidimensional analysis and performance optimization technology. This article discusses the importance of discrete multidimensional learning resource allocation optimization in dynamic scenarios. At the same time, in view of the fact that most of the resources of the big data application cluster system are supplied to large jobs that account for a small proportion of job submissions, while the small jobs that account for a large proportion only use the characteristics of a small part of the system’s resources, the expected residual multidimensionality of large-scale work is proposed. The server with the least learning resources is allocated first, and only fair strategies are considered for small assignments. The topic index is distributed and stored on the system to realize the parallel processing of search to improve the efficiency of search processing. The effectiveness of RDIBT is verified through experimental simulation. The results show that RDIBT has higher performance than LSII index technology in index creation speed and search response speed. In addition, RDIBT can also ensure the scalability of the index system.

DeepRefiner: high-accuracy protein structure refinement by deep network calibration.

The DeepRefiner webserver, freely available at http://watson.cse.eng.auburn.edu/DeepRefiner/, is an interactive and fully configurable online system for high-accuracy protein structure refinement. Fuelled by deep learning, DeepRefiner offers the ability to leverage cutting-edge deep neural network architectures which can be calibrated for on-demand selection of adventurous or conservative refinement modes targeted at degree or consistency of refinement. The method has been extensively tested in the Critical Assessment of Techniques for Protein Structure Prediction (CASP) experiments under the group name ‘Bhattacharya-Server’ and was officially ranked as the No. 2 refinement server in CASP13 (second only to ‘Seok-server’ and outperforming all other refinement servers) and No. 2 refinement server in CASP14 (second only to ‘FEIG-S’ and outperforming all other refinement servers including ‘Seok-server’). The DeepRefiner web interface offers a number of convenient features, including (i) fully customizable refinement job submission and validation; (ii) automated job status update, tracking, and notifications; (ii) interactive and interpretable web-based results retrieval with quantitative and visual analysis and (iv) extensive help information on job submission and results interpretation via web-based tutorial and help tooltips.

An Abaqus plug-in to simulate fatigue crack growth

Fatigue crack propagation is an important consideration in evaluating the design life of engineering components, especially in the energy and transport industries. Despite its importance, fatigue analyses are not usually supported by commercial Finite Element (FE) codes; in fact, most FE codes require the addition of costly plug-ins to perform fatigue crack growth simulations. Therefore, this paper introduces a new, freely distributed plug-in to simulate fatigue crack growth with the commercial FE code Abaqus. The plug-in includes five different fatigue crack growth models and relies on the extended FE method to simulate crack propagation. The plug-in is limited to 2D analyses, but covers all necessary steps for fatigue crack growth simulations, from creating the geometry to job submission and post-processing. The implementation of the plug-in is validated by comparing its predictions to analytical and experimental results. Finally, we hope that the simplicity of the plug-in and the fact that it is distributed freely will make it a useful simulation tool for industrial, research and educational purposes.

Improving the automated calibration at Belle II

In 2019 a Python plugin package, b2cal, based on the Apache Airflow workflow management platform was developed to automate the calibration at Belle II. It uses Directed Acyclic Graphs to describe the ordering of processes and Flask to provide administration and job submission web pages. This system was hosted in Melbourne, Australia and submitted calibration jobs to the High Energy Accelerator Research Organization in Japan. In 2020 the b2cal was dockerised and deployed at the Deutsches Elektronen-Synchrotron Laboratory in Germany. Improvements have been implemented that allow jobs to be submitted to multiple calibration centers and for the amount of required human interactions to be greatly reduced. All job submissions and validation of calibration constants now occur as soon as possible. In this paper, we describe the upgrades to the automated calibration at Belle II.

DMRlib: Easy-Coding and Efficient Resource Management for Job Malleability

Process malleability has proved to have a highly positive impact on the resource utilization and global productivity in data centers compared with the conventional static resource allocation policy. However, the non-negligible additional development effort this solution imposes has constrained its adoption by the scientific programming community. In this work, we present DMRlib, a library designed to offer the global advantages of process malleability while providing a minimalist MPI-like syntax. The library includes a series of predefined communication patterns that greatly ease the development of malleable applications. In addition, we deploy several scenarios to demonstrate the positive impact of process malleability featuring different scalability patterns. Concretely, we study two job submission modes (rigid and moldable) in order to identify the best-case scenarios for malleability using metrics such as resource allocation rate, completed jobs per second, and energy consumption. The experiments prove that our elastic approach may improve global throughput by a factor higher than 3x compared to the traditional workloads of non-malleable jobs.

Monte Carlo calculations of radiotherapy dose in "homogeneous" anatomy.

Given the substantial literature on the use of Monte Carlo (MC) simulations to verify treatment planning system (TPS) calculations of radiotherapy dose in heterogeneous regions, such as head and neck and lung, this study investigated the potential value of running MC simulations of radiotherapy treatments of nominally homogeneous pelvic anatomy. A pre-existing in-house MC job submission and analysis system, built around BEAMnrc and DOSXYZnrc, was used to evaluate the dosimetric accuracy of a sample of 12 pelvic volumetric arc therapy (VMAT) treatments, planned using the Varian Eclipse TPS, where dose was calculated with both the Analytical Anisotropic Algorithm (AAA) and the Acuros (AXB) algorithm. In-house TADA (Treatment And Dose Assessor) software was used to evaluate treatment plan complexity, in terms of the small aperture score (SAS), modulation index (MI) and a novel exposed leaf score (ELS/ELA). Results showed that the TPS generally achieved closer agreement with the MC dose distribution when treatments were planned for smaller (single-organ) targets rather than larger targets that included nodes or metastases. Analysis of these MC results with reference to the complexity metrics indicated that while AXB was useful for reducing dosimetric uncertainties associated with density heterogeneity, the residual TPS dose calculation uncertainties resulted from treatment plan complexity and TPS model simplicity. The results of this study demonstrate the value of using MC methods to recalculate and check the dose calculations provided by commercial radiotherapy TPSs, even when the treated anatomy is assumed to be comparatively homogeneous, such as in the pelvic region.

Difficult texts: The submission of Job (Job 42.6)

Interpreting this text as an admission of sin on Job’s part makes nonsense of the book for Job is specifically rewarded by God for what he has said. Rather, it is to be seen as recognition by Job of what it means to be human: faith must of necessity be agnostic. This is in the end a comfort.

Optimization of Metascheduler for Cloud Machine Learning Services

Cloud meta scheduler act as a global resource manager that discover the resource mapping, job submission and monitoring services in a distributed environment. Machine learning services are offered as pay-per-use by the leading cloud providers such as Amazon, Microsoft and IBM etc., Most of the cloud machine learning service architecture had a constraint that cloud users are allowed to choose the predefined listed algorithms for their experimentation and architecture like Amazon uses a single classification or regression algorithm irrespective of the input dataset. The resource utilization by cloud service user (CSU) is too low with respect to time. Here we proposed and construct the economic metascheduler in cloud machine learning architecture using a decision tree algorithm that selects the suitable classification or prediction algorithm based on the users need and achieve maximum profit for both cloud machine learning service providers (CMLSP) and cloud service user (CSU). The experimentation is carried out using Amazon AWS, Microsoft Azure, WEKA and the number results show that the proposed metascheduler architecture is superior to the real time CMLSP.

The open EEGLAB portal Interface: High-Performance computing with EEGLAB

EEGLAB signal processing environment is currently the leading open-source software for processing electroencephalographic (EEG) data. The Neuroscience Gateway (NSG, nsgportal.org) is a web and API-based portal allowing users to easily run a variety of neuroscience-related software on high-performance computing (HPC) resources in the U.S. XSEDE network. We have reported recently (Delorme et al., 2019) on the Open EEGLAB Portal expansion of the free NSG services to allow the neuroscience community to build and run MATLAB pipelines using the EEGLAB tool environment. We are now releasing an EEGLAB plug-in, nsgportal, that interfaces EEGLAB with NSG directly from within EEGLAB running on MATLAB on any personal lab computer. The plug-in features a flexible MATLAB graphical user interface (GUI) that allows users to easily submit, interact with, and manage NSG jobs, and to retrieve and examine their results. Command line nsgportal tools supporting these GUI functionalities allow EEGLAB users and plug-in tool developers to build largely automated functions and workflows that include optional NSG job submission and processing. Here we present details on nsgportal implementation and documentation, provide user tutorials on example applications, and show sample test results comparing computation times using HPC versus laptop processing.

Enhancing Hadoop Performance in Homogenous and Heterogeneous Big Data Environments by Dynamic Slot Configuration

Hadoop is one of the most famous platform solutions for processing large volume and scale of data in parallel processing in Cloud computing. A Hadoop system can be characterized based on three main factors: cluster, workload and user. Each of these factors can be described as either heterogeneous or homogenous, which reflects the heterogeneity degree of the Hadoop systemThe objective of this proposed research work is to investigate the degree of influence of heterogeneity for each of these factors on the performance of Hadoop based on different schedulers. Three schedulers are considered with different levels of Hadoop heterogeneity and are tested and analyzed: the first algorithm considered is the FIFO (First in First out), the second is the Fair sharing, and the final is the COSHH (Classification and Optimization based Scheduler for Heterogeneous Hadoop). Performance issues are related to Hadoop schedulers and comparative performance analysis between different cases of jobs submission. These jobs are processed in different homogenous or heterogeneous data environments and under fixed or reconfigurable slot between map and reduce tasks for Hadoop MapReduce java programming clustering model. The results showed that when assigning tunable knob between map and reduce tasks under certain schedulers like FIFO algorithm, the performance enhanced significantly especially in cases of heterogeneity environment where the workload decreased significantly and the utilization of computational resources increase was obvious.

Development of Computational Pipeline Software for Genome/Exome Analysis on the K Computer

Pipeline software that comprise tool and application chains for specific data processing have found extensive utilization in the analysis of several data types, such as genome, in bioinformatics research. Recent trends in genome analysis require use of pipeline software for optimum utilization of computational resources, thereby facilitating efficient handling of large-scale biological data accumulated on a daily basis. However, use of pipeline software in bioinformatics tends to be problematic owing to their large memory and storage capacity requirements, increasing number of job submissions, and a wide range of software dependencies. This paper presents a massive parallel genome/exome analysis pipeline software that addresses these difficulties. Additionally, it can be executed on a large number of K computer nodes. The proposed pipeline incorporates workflow management functionality that performs effectively when considering the task-dependency graph of internal executions via extension of the dynamic task distribution framework. Performance results pertaining to the core pipeline functionality, obtained via evaluation experiments performed using an actual exome dataset, demonstrate good scalability when using over a thousand nodes. Additionally, this study proposes several approaches to resolve performance bottlenecks of a pipeline by considering the domain knowledge pertaining to internal pipeline executions as a major challenge facing pipeline parallelization.

DP-GEN: A concurrent learning platform for the generation of reliable deep learning based potential energy models

In recent years, promising deep learning based interatomic potential energy surface (PES) models have been proposed that can potentially allow us to perform molecular dynamics simulations for large scale systems with quantum accuracy. However, making these models truly reliable and practically useful is still a very non-trivial task. A key component in this task is the generation of datasets used in model training. In this paper, we introduce the Deep Potential GENerator (DP-GEN), an open-source software platform that implements the recently proposed ”on-the-fly” learning procedure (Zhang et al. 2019) and is capable of generating uniformly accurate deep learning based PES models in a way that minimizes human intervention and the computational cost for data generation and model training. DP-GEN automatically and iteratively performs three steps: exploration, labeling, and training. It supports various popular packages for these three steps: LAMMPS for exploration, Quantum Espresso, VASP, CP2K, etc. for labeling, and DeePMD-kit for training. It also allows automatic job submission and result collection on different types of machines, such as high performance clusters and cloud machines, and is adaptive to different job management tools, including Slurm, PBS, and LSF. As a concrete example, we illustrate the details of the process for generating a general-purpose PES model for Cu using DP-GEN. Program summaryProgram Title: DP-GENProgram Files doi:http://dx.doi.org/10.17632/sxybkgc5xc.1Licensing provisions: LGPLProgramming language: PythonNature of problem: Generating reliable deep learning based potential energy models with minimal human intervention and computational cost.Solution method: The concurrent learning scheme is implemented. Supports for sampling configuration space with LAMMPS, generating ab initio data with Quantum Espresso, VASP, CP2K and training potential models with DeePMD-kit are provided. Supports for different machines including workstations, high performance clusters and cloud machines are provided. Supports for job management tools including Slurm, PBS, LSF are provided.

Lightweight job submission and file sharing schemes for a teaching ecosystem for parallel computing courses

Parallel computing has been widely used in various industries. Many colleges and universities have launched courses on parallel computing and can build teaching ecosystems to serve these courses. Job submission and file sharing are two essential functionalities. This paper builds a teaching ecosystem for parallel computing courses and proposes lightweight job submission and file sharing schemes for the ecosystem. The job submission scheme supports Pthread, OpenMP, and MPI (message passing interface) jobs. It has five working modes: demonstration, experiment, examination, research, and maintenance. In particular, the examination mode can be applied for online examinations. For the fair use of resources, the job submission scheme not only limits the submission times of each student and the run time of each job but also designs a fair job scheduling algorithm. The ecosystem adopts Samba to supply file sharing services. To ensure the safety of the shared files, this paper proposes a lightweight attribute-based access control mechanism. This mechanism uses Python logical expressions to describe access rules; thus, it is not only easy to use but also able to describe complicated rules. In addition, the ecosystem has a well-structured module design that facilitates functionality expansion and modification. Experiments verify the feasibility and validity of the ecosystem.

Designing a MapReduce performance model in distributed heterogeneous platforms based on benchmarking approach

MapReduce framework is an effective method for big data parallel processing. Enhancing the performance of MapReduce clusters, along with reducing their job execution time, is a fundamental challenge to this approach. In fact, one is faced with two challenges here: how to maximize the execution overlap between jobs and how to create an optimum job scheduling. Accordingly, one of the most critical challenges to achieving these goals is developing a precise model to estimate the job execution time due to the large number and high volume of the submitted jobs, limited consumable resources, and the need for proper Hadoop configuration. This paper presents a model based on MapReduce phases for predicting the execution time of jobs in a heterogeneous cluster. Moreover, a novel heuristic method is designed, which significantly reduces the makespan of the jobs. In this method, first by providing the job profiling tool, we obtain the execution details of the MapReduce phases through log analysis. Then, using machine learning methods and statistical analysis, we propose a relevant model to predict runtime. Finally, another tool called job submission and monitoring tool is used for calculating makespan. Different experiments were conducted on the benchmarks under identical conditions for all jobs. The results show that the average makespan speedup for the proposed method was higher than an unoptimized case.

Production Operations Management System (POMS) for Fermilab Experiments

The Production Operations Management System (POMS) software allows production teams and analysis groups across multiple Fermilab experiments to launch, modify and monitor large scale campaigns of related Monte Carlo or data processing jobs, and currently manages the majority of production computing of experiments at Fermilab. POMS provides a web service interface that enables automated job submission on distributed resources according to customers’ requests, as well as subsequent monitoring of those submissions as well as recovery of failed submissions, debugging and record keeping. POMS interfaces with existing HEP data access, processing, movement and monitoring tools at Fermilab, including Jobsub (for job submission), SAM (for data management), and Landscape (for monitoring), and in combination with them handles the creation of intermediate datasets of files for multistage campaigns. POMS visualizes these campaigns with an interactive GUI interface. An important feature of POMS is a one-to-one connection between the GUI campaign visualizer and a text based representation of the complete campaign configuration. An extensible library of template campaign configurations is available. The templates are user modifiable and map cleanly to and from the GUI description of the campaign. This paper will discuss the current usage, history, and some technical details of POMS, as well as planned future extensions, particularly interfacing with Rucio as well as SAM for file movement.

ART ATLAS Release Tester using the Grid

The ART (ATLAS Release Tester) system is designed to run test jobs on the Grid after a nightly release of the ATLAS offline software has been built. The choice was taken to exploit the Grid as a backend as it offers a huge resource pool, suitable for a deep set of integration tests, and running the tests could be delegated to the highly scalable ATLAS production system (PanDA). The challenge of enabling the Grid as a test environment is met through the use of the CVMFS file system for the software and input data files. Test jobs are submitted to the Grid by the GitLab Continuous Integration (gitlab-ci) system, which itself is triggered at end of a release build. Jobs can be adorned with special headers that inform the system how to run the specific test, allowing many options to be customised. The gitlab-ci system waits for exit status and output files are copied back from the Grid to an EOS area accessible by users. All gitlab-ci jobs run in ART’s virtual machines, using docker images for their ATLAS setup. ART jobs can be tracked by using the PanDA system. ART can also be used to run short test jobs locally. It uses the same ART command-line interface, where the backend is replaced to access a local machine for job submission rather than the Grid. This allows developers to ensure their tests work correctly before adding them to the system. In both the Grid and local machine options, running and result copying are completely parallelized. ART is written in python, complete with its own local and Grid tests to give approximately 90% code coverage of the ART tool itself. ART has been in production for one year and fully replaces and augments the former ATLAS testing system.

Evolution and Perspectives of the Service for Parallel Applications Running at JINR Multifunctional Information and Computing Complex

Nowadays scientists use cloud computing as a routine tool in a lot of fields of their research. Various Multifunctional Information and Computing Complex (MICC) resources are provided for JINR users to perform a wide range of scientific computations. The JINR cloud service for parallel applications was developed in order to simplify scientists’ work on running similar tasks but on different MICC resources and also to speed up the process of reaching significant results. There are several components with a flexible and modular architecture that allow running a various number of applications using different types of computational resources. The service is constantly developing and improving with the help of the users’ feedback. Some changes in web-interface were made to improve users’ experience: there was added the possibility to choose a certain type of particular application, to set a description for a job, to run multiple tasks, to notify a user about successful job submission and its completion. Moreover, accessibility of job results was reworked: when the job is done, its output is uploaded at the external file storage, where it becomes available at the auto-generated unique URL for downloading by the user and further analysis and/or visualization.