Distributed Environment Research Articles

Optimal Data File Allocation for All-to-All Comparison in Distributed System: A Case Study on Genetic Sequence Comparison

In order to solve the problem of unbalanced load of data les in large-scale data all-to-all comparison under distributed system environment, the differences of les themselves arefully considered. This paper aims to fully utilize the advantages of distributed system to enhance the le allocation of all-to-all comparison between the data les in a large dataset. For this purpose, the author formally described the all-to-all comparison problem, and con-structed a data allocation model via mixed integer linear programming (MILP). Meanwhile, a data allocation algorithm was developed on the Matlab using the intlinprog function of branch-and-bound method. Finally, our model and algorithm were veried through several experiments. The results show that the proposed le allocation strategy can achieve the basic load balance of each node in the distributed system without exceeding the storage capacity of any node, and completely localize the data le. The research ndings can be applied to such elds as bioinformatics, biometrics and data mining.

Distributed Deep Learning Model for Intelligent Video Surveillance Systems with Edge Computing

In this paper, we propose a Distributed Intelligent Video Surveillance (DIVS) system using Deep Learning (DL) algorithms and deploy it in an edge computing environment. We establish a multi-layer edge computing architecture and a distributed DL training model for the DIVS system. The DIVS system can migrate computing workloads from the network center to network edges to reduce huge network communication overhead and provide low-latency and accurate video analysis solutions. We implement the proposed DIVS system and address the problems of parallel training, model synchronization, and workload balancing. Task-level parallel and model-level parallel training methods are proposed to further accelerate the video analysis process. In addition, we propose a model parameter updating method to achieve model synchronization of the global DL model in a distributed EC environment. Moreover, a dynamic data migration approach is proposed to address the imbalance of workload and computational power of edge nodes. Experimental results showed that the EC architecture can provide elastic and scalable computing power, and the proposed DIVS system can efficiently handle video surveillance and analysis tasks.

A novel directional and non-local-convergent particle swarm optimization based workflow scheduling in cloud–edge environment

With the increasing popularity of Internet of Things (IoT), edge computing has become the key driving force to provide computing resources, storage and network services closer to the edge on the basis of cloud computing. Workflow scheduling in such distributed environment is regarded as an NP-hard problem, and the existing approaches may not work well for task scheduling with multiple optimization goals in complex applications. As an intelligent algorithm, particle swarm optimization (PSO) has the advantages of fewer parameters, simpler algorithm and faster convergence speed, which is widely applied to workflow scheduling. However, there are also some shortcomings such as easy to fall into local optimum and sometimes difficult to obtain real optimal solution. To address this issue, first, the scheduling problem of workflow applications and objective function based on two optimized factors are clearly formalized, which can provide a theoretical foundation for workflow scheduling strategy. Then this paper proposes a novel directional and non-local-convergent particle swarm optimization (DNCPSO) that employs non-linear inertia weight with selection and mutation operations by directional search process, which can reduce the makespan and cost dramatically and obtain a compromising result. The results of simulation experiments based on various real and random workflow examples show that our DNCPSO can achieve better performance than other classical and improved algorithms, which sufficiently demonstrate the effectiveness and efficiency of DNCPSO.

Developing student writing in higher education: digital third-party products in distributed learning environments

ABSTRACT Academic literacies research emphasizes the importance of social context for understanding student writing development in higher education. In particular, students’ choices of textual practices are shaped by perceptions of disciplinary norms and institutional expectations. In contemporary online learning environments, however, student writing is increasingly guided by advice from digital products – referred to in this paper as ‘third-party products’. These are developed by commercial providers external to students’ disciplines and institutions and include machine-based tools for analyzing texts, content-based courses on writing, and person-based services connecting students to writing tutors via online platforms. These products appear to offer universities the opportunity to outsource the task of developing student writing, however, they position writing as a technical process and elide the role of social context in determining what ‘good’ writing is. This paper locates third-party products within theories of academic writing, discussing implications for student writing development in ‘distributed’ university learning environments.

Lightweight data management with dtool.

The explosion in volumes and types of data has led to substantial challenges in data management. These challenges are often faced by front-line researchers who are already dealing with rapidly changing technologies and have limited time to devote to data management. There are good high-level guidelines for managing and processing scientific data. However, there is a lack of simple, practical tools to implement these guidelines. This is particularly problematic in a highly distributed research environment where needs differ substantially from group to group and centralised solutions are difficult to implement and storage technologies change rapidly. To meet these challenges we have developed dtool, a command line tool for managing data. The tool packages data and metadata into a unified whole, which we call a dataset. The dataset provides consistency checking and the ability to access metadata for both the whole dataset and individual files. The tool can store these datasets on several different storage systems, including a traditional file system, object store (S3 and Azure) and iRODS. It includes an application programming interface that can be used to incorporate it into existing pipelines and workflows. The tool has provided substantial process, cost, and peace-of-mind benefits to our data management practices and we want to share these benefits. The tool is open source and available freely online at http://dtool.readthedocs.io.

Fruitful student interactions and perceived learning improvement in DSLab: A dynamic assessment tool for distributed programming

AbstractThis work presents an automated assessment tool for online distributed programming, called DSLab. It is a web‐based environment that provides a transparent deployment and execution of assignments in remote computers, a transparent initialization and the possibility to add new logs by the students. DSLab has been evaluated in a real distributed learning environment by analysing students' own perceptions on their learning improvement and exploring whether students' interactions with the tool yielded a fruitful learning experience. Current research provides no evidence that the effects of such a tool have been investigated in the field of online distributed programming. Two types of analysis were performed: a quantitative analysis of data coming from students answering a questionnaire and an analysis of the log files of students' interactions with the tool. Our results showed that students perceived substantial learning improvement from the use of the automated assessment tool. Moreover, students produced fruitful interaction with the tool as soon as they achieved high familiarization and constant activity with it, which ultimately helped them improve their academic performance. Finally, the limitations of the current study and directions for future research are presented.

An extended access control model for permissioned blockchain frameworks

In distributed environment, a digital transaction or operation requires transparency and trust among multiple stakeholders. Several approches address such issues however, among these blockchain provides a viable solution which has received wide acceptance in the recent past. Permissioned blockchain solutions adopt more efficient consensus algorithms and smart contracts. There are many smart-contract solutions exists (such as, etherium, IBM blockchain, hyperledger fabric), however, much of them mainly follow traditional access control models. A role-based access control model provides controlled access of resources to members. This research work presents an extended usage control model known as DistU (Distributed Usage Control). DistU is proposed to capture all possible access control models required by a business for permissioned blockchain frameworks. DistU can monitor a resource continuously during the operation and update the attributes accordingly, performing different actions, such as denying or revoking permissions. We believe that the proposed DistU usage control model can provide a fine-grained control for blockchain resource management. The paper also contributes to provide a protoype implementation of fine-grained permission model on Hyperledger Fabric. The reason of selecting Fabric for this research is that, it is the first execute-order achitecture blockchain that provides a platform to develop general business applciations. Secondly, it is an opensource operating system of permissioned blockchain with huge industry support.

CauseInfer: Automated End-to-End Performance Diagnosis with Hierarchical Causality Graph in Cloud Environment

Modern computing systems especially cloud-based and cloud-centric systems always consist of a mass of components running in large distributed environments with complicated interactions. They are vulnerable to performance problems due to the highly dynamic runtime environment changes (e.g., overload and resource contention) or software bugs (e.g., memory leak). Unfortunately, it is notoriously difficult to diagnose the root causes of these performance problems in a fine granularity due to complicated interactions and a large cardinality of potential cause set. In this paper, we build an automated, black-box and end-to-end cause inference system named CauseInfer to pinpoint the root causes or at least provide some hints. CauseInfer can automatically map a distributed system to a two-layer hierarchical causality graph and infer the root causes along the causal paths in the causality graph. CauseInfer models the fault propagation paths in an explicit way and works without instrumentation to the running production system, which makes CauseInfer more effective and practical than previous approaches. The experimental evaluations in two benchmark systems show that CauseInfer can identify the root causes in a high accuracy. Compared to several state-of-the-art approaches, CauseInfer can achieve over 10 percent improvement. Moreover, CauseInfer is lightweight and flexible enough to readily scale out in large distributed systems. With CauseInfer, the mean time to recovery (MTTR) of the cloud systems can be significantly reduced.

A post-quantum light weight 1-out-n oblivious transfer protocol

Security is a key concern in any IoT deployment, particularly if we have to take into consideration future attacks facilitated by the use of quantum computers. Therefore, in this paper, we present a post-quantum lightweight 1-out-n oblivious transfer (OT) protocol, based on the NTRU cryptographic primitive. Compared to the OT scheme proposed by Chou and Orlandi in LATINCRYPT 2015, our protocol is more suitable for deployment in heterogeneous and distributed environment, due to the reusability of encoded data set. Findings from the performance evaluation indicate that the proposed protocol outperforms that of Chou and Orlandi [13] protocol, in terms of computation and communication costs.

Preparing communication design students as facilitators

Building from previous work by Lauren and Schreiber (2017) and research individually conducted by the author (Lauren, 2018), this brief teaching case provides a rationale for coursework in project management that draws from experiential learning to teach facilitation. The case begins by providing a research context for how communication designers are increasingly focused on practices of facilitation in their work, particularly in fast-paced, distributed work environments. The case presents two metaphors (gardening and cooking) for helping students think about facilitation techniques. Then, the article describes a project management course that emphasizes the importance of facilitation in classroom exercises and major assignments by developing skills in three foundational areas: improvisation, document design, and systems design. Each area is described with examples to help instructors of project management adapt or use similar approaches at their own unique institutional, programmatic, and classroom contexts. The article concludes with four suggestions, such as partnering with industry practitioners and arranging site visits to see project management in action. As well, the concluding suggestions explain recent iterations of the course's design.

Exploring and Expanding GSE Education with Open Source Software Development

Global software engineering (GSE) courses traditionally require cooperation between at least two universities so as to provide a distributed development environment to the students. In this study, we explore an alternative way to organize a global software engineering course where students work on open source software development (OSSD) projects rather than in a multi-university collaboration setting. The results show that the new setup may provide core GSE challenges as well as challenges associated with software development outsourcing and challenges related to working on large open source software. The present article compares the experiences gained from running a combined GSE and OSSD course against the experiences gained from running a traditional GSE course. The two alternatives are compared in terms of students’ learning outcomes and course organization. The authors found that a combined GSE and OSSD course provides learning opportunities that are partly overlapping with, and partly complementary to, a traditional GSE course. The authors also found that the combined OSSD and GSE course was somewhat easier to organize because most of the activities took place in a single university setting. The authors used the extended GSE taxonomy for the comparison and found it to be a useful tool for this, although it had some limitations in expressive power. Therefore, two additional relationship dimensions are proposed that will further enrich the extended taxonomy in classifying GSE (and OSSD) projects.

CRIC: a unified information system for WLCG and beyond

The Worldwide LHC Computing Grid (WLCG) is an innovative distributed environment which is deployed through the use of grid computing technologiesin order to provide computing and storage resources to the LHC experimentsfor data processing and physics analysis. Following increasing demands of LHC computing needs toward high luminosity era, the experiments are engagdin an ambitious program to extend the capability of WLCG distributed environment, for instance including opportunistically used resources such as High-Performance Computers (HPCs), cloud platforms and volunteer computer. norder to be effectively used by the LHC experiments, all these diverse distributed resources should be described in detail. This implies easy service discovery of shared physical resources, detailed description of service configurations and experiment-specific data structures is needed. In this contribution, we present a high-level information component of a distributed computing environment, the Computing Resource Information Catalogue (CRIC) which aims to facilitate distributed computing operations for the LHC experiments and consolidate WLCG topology information. In addition, CRIC performs data validation and provides coherent view and topology descriptinto the LHC VOs for service discovery and configuration. CRIC represents teevolution of ATLAS Grid Information System (AGIS) into the common experiment independent high-level information framework. CRIC’s mission is to serve not just ATLAS Collaboration needs for the description of the distributed environment but any other virtual organization relying on large scale distributed infrastructure as well as the WLCG on the global scope. The contribution describes CRIC architecture, implementation of data model,collectors, user interfaces, advanced authentication and access control components of the system.

TEPaaS: test execution platform as-a-service applied in the context of e-health

Runtime testing of large-scale systems running in dynamic and distributed environments is a costly and a resource consuming task. With the aim of handling such runtime validation activity in a cost...

A Real-Time Trace-Level Root-Cause Diagnosis System in Alibaba Datacenters

Root-cause analysis (RCA) for service performance degradation can be a challenging exercise given the increasingly complex, inter-related, distributed infrastructure environment in today's enterprises. Many approaches have been applied into enterprise datacenters to improve the maintenance efficiency. A novel graph-level RCA approach is introduced in this paper, including tracing, weighted graph matching and suspicion ranking. The approach is developed based on performance profiling, tracing, and logging systems in Alibaba datacenters to speed up the real-time root-cause diagnosis. Our system allows the discovery of normative patterns and the corresponding key graph properties, which are stored and updated offline as a knowledge base for subsequently being used in trace-level risk estimation and identification of transitions that are unexpected deviations from the normative patterns. Through testing in production, we show the effectiveness of applying the graph-level RCA to discover the origins of problems and generate real-time operational support. It greatly decreases the workload for locating the root-cause of the anomaly.

CDLB: a cross-domain load balancing mechanism for software defined networks in cloud data centre

Currently, cross-domain load balancing is one of the core issues for software defined networks (SDN) in cloud data centre, which can optimise resource allocation. In this paper, we propose a cross-domain load balancing mechanism, CDLB, based on Extensive Messaging and Presence Protocol (XMPP) for SDN in cloud data centre. Different from poll method, XMPP based push model is introduced in the proposed scheme, which can avoid wasting network and computing resources in large-scale distributed network environment. The proposed scheme enables all the controllers in the flat distributed control plane to share the same consistent global-view network information in real time through XMPP and XMPP publish/subscribe extension. Thus, the problem of non-real time information synchronisation can be resolved and cross-domain load balancing can be realised. The simulations show the efficiency of the proposed scheme.

CDLB: a cross-domain load balancing mechanism for software defined networks in cloud data centre

Currently, cross-domain load balancing is one of the core issues for software defined networks (SDN) in cloud data centre, which can optimise resource allocation. In this paper, we propose a cross-domain load balancing mechanism, CDLB, based on Extensive Messaging and Presence Protocol (XMPP) for SDN in cloud data centre. Different from poll method, XMPP based push model is introduced in the proposed scheme, which can avoid wasting network and computing resources in large-scale distributed network environment. The proposed scheme enables all the controllers in the flat distributed control plane to share the same consistent global-view network information in real time through XMPP and XMPP publish/subscribe extension. Thus, the problem of non-real time information synchronisation can be resolved and cross-domain load balancing can be realised. The simulations show the efficiency of the proposed scheme.

Secure and Efficient Multi-Authority Attribute-Based Encryption Scheme From Lattices

Attribute-based encryption (ABE) mechanism on lattices can against quantum attack and guarantees fine-grained access control of encrypted data, and it has been widely used in cloud computing and privacy protection. However, the high complexity of the trapdoor generation and delegation and preimage sampling are the main barriers for the development of ABE schemes in a post-quantum world. Thus, we proposed a security and efficient multi-authority ABE scheme from lattices in this paper. The scheme is based on the theory of MP12 and is proved to be secure under the standard model. In this scheme, we construct an optimized sampling algorithm to generate a $\mathbb {Z}_{q}$ -invertible matrix with a lower runtime. Based on it, combining the trapdoor generation and delegation algorithm in MP12, we designed multiple attribute authorities which can manage different attribute sets and generate private keys for the user independently. In addition, the Shamir’s secret sharing technique is introduced to support policies expressed in any monotone access structures. Compared with the existing related schemes, the proposed scheme can improve the functional agility and the flexibility of the access policy prominently, and it also can achieve a better performance with less lattice dimension, trapdoor storage occupation, and ciphertext expansion rate. The analysis shows that our scheme is feasible and superior in the large-scale distributed environment.

A New Method to Compute Ratio of Secure Summations and Its Application in Privacy Preserving Distributed Data Mining

Computing the ratio of secure summations (RSS) is one of the most important tools for privacy preserving distributed data mining. It refers to such a problem; given the n parties and their respective secret values (x i ,y i ), where i=1, ..., n , how can they get the value of Σ i=1 n x i / Σ i=1 n y i without disclosing (x i ,y i ), and Σ i=1 n x i and Σ i=1 n y i In this paper, we propose a new method to solve this problem. Compared with the existing methods, our method is not only secure under the assumption of semi-honest but also can resist collusion attacks-even when all but one party are corrupt. At the same time, the computational complexity and communication complexity of our method are both O(n) . Therefore, it can meet the needs of the practical application in terms of both security and efficiency. In addition to the theoretical guarantee, the practical usability of our method is also verified through experiments by constructing Naive Bayes classifiers in a privacy-preserving distributed environment.

Using Machine Learning Ensemble Methods to Predict Execution Time of e-Science Workflows in Heterogeneous Distributed Systems

Effective planning and optimized execution of the e-Science workflows in distributed systems, such as the Grid, need predictions of execution times of the workflows. However, predicting the execution times of e-Science workflows in heterogeneous distributed systems is a challenging job due to the complex structure of workflows, variations due to input problem-sizes, and heterogeneous and dynamic nature of the shared resources. To this end, we propose two novel workflow execution time-prediction methods based on the machine learning ensemble models. In this paper, we showcase our approach for two different real Grid environments. Our approach can effectively predict the execution time of the scientific workflow applications in the Grid for various problem sizes, Grid sites, and runtime environments. We characterized the workflow performance in the Grid using the attributes that define structure of workflow as well as the execution environment. Contrary to common ensembles, our ensemble systems employed three strong learners, which balance the weaknesses of each other by their strengths to model the workflow execution times. The proposed methods have been thoroughly evaluated for three real-world e-science workflow applications. The experimental results demonstrated that our proposed multi-model ensemble models can significantly decrease the prediction error (by 50%, on average) as compared with methods based on the radial basis function neural network, local learning, and performance templates. The proposed methods can also be applied with similar effectiveness and without any major modification for other heterogeneous distributed environments, such as the Cloud.

A modular cloud-based ontology framework for context-aware EHR services

Healthcare providers in a heterogeneous distributed environment depend on many non-functional requirements or cross-cutting concerns that could affect the entire Healthcare Information System (HIS). The implementation of crosscutting concerns could potentially lead to a significant impact on the integrity of Electronic Health Records (EHRs), as well as code quality. Aspect-Oriented Programming (AOP) encapsulates the crosscutting concerns to improve the overall quality of software development by reducing the required code to implement collection modules. In this paper, we propose an AOP-based novel approach to improve the elasticity of cloud-based EHR services. We propose an abstract framework for cloud-based HIS, which provides EHR interoperability through incorporating crosscutting concerns. The proposed framework eliminates the need to change the core code by adding high-level abstractions to complex cloud applications. We implement a prototype of the proposed framework to validate its ability in both separation of concerns and the ability to reuse EHR services.