Multiprocessor Networks Research Articles

OpenMP for Networks of SMPs

In this paper, we present the first system that implements OpenMP on a network of shared-memory multiprocessors. This system enables the programmer to rely on a single, standard, shared-memory API for parallelization within a multiprocessor and between multiprocessors. It is implemented via a translator that converts OpenMP directives to appropriate calls to a modified version of the TreadMarks software distributed shared-memory (SDSM) system. In contrast to previous SDSM systems for SMPs, the modified TreadMarks system uses POSIX threads for parallelism within an SMP node. This approach greatly simplifies the changes required to the SDSM in order to exploit the intranode hardware shared memory. We present performance results for seven applications (Barnes-Hut, CLU, and Water from SPLASH-2, 3D-FFT from NAS, Red-Black SOR, TSP, and MGS) running on an SP2 with four four-processor SMP nodes. A comparison between the thread implementation and the original implementation of TreadMarks shows that using the hardware shared memory within an SMP node significantly reduces the amount of data and the number of messages transmitted between nodes and consequently achieves speedups that are up to 30% better than the original versions. We also compare SDSM against message passing. Overall, the speedups of multithreaded TreadMarks programs are within 7–30% of the MPI versions.

Software-based rerouting for fault-tolerant pipelined communication

This paper presents a software-based approach to fault-tolerant routing in networks using wormhole or virtual cut-through switching. When a message encounters a faulty output link, it is removed from the network by the local router and delivered to the messaging layer of the local node's operating system. The message passing software can reroute this message, possibly along nonminimal paths. Alternatively, the message may be addressed to an intermediate node, which will forward the message to the destination. A message may encounter multiple faults and pass through multiple intermediate nodes. The proposed techniques are applicable to both obliviously and adaptively routed networks. The techniques are specifically targeted toward commercial multiprocessors where the mean time to repair (MTTR) is much smaller than the mean time between router failures (MTBF), i.e., it is sufficient to tolerate a maximum of three failures. This paper presents requirements for buffer management, deadlock freedom, and livelock freedom. Simulation results are presented to evaluate the degradation in latency and throughput as a function of the number and distribution of faults. There are several advantages of such an approach. Router designs are minimally impacted, and thus remain compact and fast. Only messages that encounter faulty components are affected, while the machine is ensured of continued operation until the faulty components can be replaced. The technique leverages existing network technology, and the concepts are portable across evolving switch and router designs. Therefore, we feel that the technique is a good candidate for incorporation into the next generation of multiprocessor networks.

Guest editors' introduction: challenges in designing fault-tolerant routing in networks

+I INRE arc several means nf intercomiecting the compoT nents of a computer system. Components can be connccted through direct or indirect connections. In a direct network, channels directly connect each processor. Neighboring nodes communicate with each other by scnding messages across the channcls. In an indirect network, processors are cnnnccted through switches and/or buses. Depending on how components are connected, we have bus-based networks, router-based networks, switcli-bascd networks, and cluster-based (hybrid of the other thrce networks) nctworks. Nctworks can also be classified bascd on topologies, which cau be either replnr or irregulnr. Anothcr classification is based on applications-there are local arca networks (LANs), networks of workstations (NOWs), multiprocessors, and mobilc networks. In a computer network, the efficiency of a routing process is critical to the system performance. A routing process deals with moving data between processors in a given network. When a routing process involves oiic source and onc destination, it is called unicnst or poirit-lopoint communication. When such a proccss involves more than one source and/or one destination, it is called a collective comrnlrnicntion process. Examples of collective communication include iirulticnst (one-to-many, one source and many destinations), brundcnst (one-to-all), and gossip (all-to-all).

High Performance Computing in Multibody System Design

This paper presents recent developments of high performance computing and networking techniques in the field of computer-aided multibody analysis and design. The authors describe the main achievements obtained in the development of a tool to aid in the design of new industrial mechanical systems by performing parallel parametric multibody simulations. The parallel software is composed of four main modules: two user-friendly interfaces to input the data and visualize the results, a simulation module, and a parallel manager module that is the main focus of this paper. The authors show that the implementation, using PVM, of simple and well-known ideas leads to efficient and flexible parallel software targeted for heterogeneous networks of nondedicated workstations, which is the parallel platform available in most mechanical design departments. The authors also describe the main features of this module that implements, for the sake of efficiency and robustness, a load-balancing strategy and fault tolerance capabilities. One of the main results of this development has been to demonstrate that high performance computing can sometimes be easily and efficiently introduced into industrial companies without requiring major investments. Pictures of the user interfaces are depicted to illustrate the usability of the parallel software. Performance observed on single and multiprocessor workstation networks is given to assess the relevance of using high performance computing techniques in an industrial environment for the design of mechanical systems.

Routing in wormhole-switched clustered networks with applications to fault tolerance

This paper presents a novel technique for routing in wormhole-switched multiprocessor interconnection networks with clustered configuration. The network model used here consists of a set of clusters interfaced through a common central network. We assume that the central network and the clusters use independent algorithms to route messages between their internal nodes. A technique for deriving a global routing algorithm based on the local algorithms is presented, which allows the transfer of messages between any pair of nodes in the network. This proposed method is shown to be deadlock-free with two virtual channels. The clustered network model and the proposed routing technique can be used to enhance the fault tolerance capability of existing routing algorithms. In particular, we describe fault-tolerant routing methods for meshes, which can tolerate any arbitrary fault distribution without disabling connected healthy nodes.

Task spreading and shrinking on multiprocessor systems and networks of workstations

In this paper, we describe how our computational model can be used for the problems of processor allocation and task mapping. The intended applications for this model include the dynamic mapping problems of shrinking or spreading an existing mapping when the available pool of processors changes during execution of the problem. The concept of problem edge class and other features of our model are developed to realistically and efficiently support task partitioning and merging for static and dynamic mapping. The model dictates realistic changes in the computation and communication characteristics of a problem when the problem partitioning is modified dynamically. This model forms the basis of our algorithms for shrinking and spreading, and yields realistic results for a variety of problems mapped onto real systems. An emulation program running on a network of workstations under PVM is used to measure execution times for the mapping solutions found by the algorithms. The results indicate that the problem edge class is a crucial consideration for processor allocation and task mapping.

A testbed for evaluation of fault-tolerant routing in multiprocessor interconnection networks

This paper presents a comprehensive evaluation testbed for interconnection networks and routing algorithms using real applications. The testbed is flexible enough to implement any network topology and fault-tolerant routing algorithm, and allows the system architect to study the cost versus performance trade-offs for a range of network parameters. We illustrate its use with one fault-tolerant algorithm and analyze the performance of four shared memory applications with different fault conditions. We also show how the testbed can be used to drive future research in fault-tolerant routing algorithms and architectures by proposing and evaluating novel architectural enhancements to the network router, called path selection heuristics (PSH). We propose three such schemes and the Least Recently Used (LRU) PSH is shown to give the best performance in the presence of faults.

On communications performance and processor efficiency

Efficient applications running on large multiprocessor machines require the support of an effective communications fabric. Contemporary multiprocessor networks based on dedicated VLSI switching components have dramatically improved performance over store-and-forward techniques, however communications latency is still very significant compared with computation speed. This limitation places constraints on the design of efficient parallel applications. This paper demonstrates the viability of an approach to enhancing communications performance and thus run-time efficiency of network-bounded parallel applications. A bound is derived on the maximum obtainable efficiency for applications on point-to-point connected architectures. This motivates an optimisation strategy based on controlling network bandwidth requirements, by constraining the dilation of the mapped interprocess communications graph. The effectiveness of the optimisation strategy is demonstrated through simulation of realistic network-bounded application loads.

Comparisons between mesh and hierarchical ring networks for shared-memory multiprocessors

Communication performance comparisons, in terms of message latency, between mesh and hierarchical ring interconnection networks for shared-memory multiprocessors are developed. Traffic in the networks is assumed to consist of the short, fixed-length messages needed for remote-memory read/write operations and cache coherency control. Square-mesh networks with bidirectional links and no end-around connections are compared to two- and three-level hierarchical rings. Wormhole routing is used in the meshes, with messages consisting of / flits, with each flit containing b bits. In the ring networks, each ring segment can contain a complete message of f × b bits. Meshes and three-level rings provide comparable delays for system sizes of up to 200 or 300 processor clusters. The delay comparisons are valid only for the lightly loaded traffic case in which contention and blocking are low. This is the situation in practical shared-memory multiprocessor systems with a low cache miss rate and a high physical locality of reference in the global address space.

A priority-driven flow control mechanism for real-time traffic in multiprocessor networks

Real-time applications when mapped to distributed memory multiprocessors produce periodic messages with an associated deadline and priority. Real-time messages may be hard or soft deadline. Real-time extensions to wormhole routing (WR) with multiple virtual channels (VCs) and priority-based physical link arbitration and VC allocation have been proposed in the literature. With a fixed number of VCs/link, a message can face an unbounded priority inversion, rendering the global priority ineffective. In this paper, we propose a new flow control mechanism called Preemptive Pipelined Circuit Switching for Real-Time messages (PPCS-RT) to reduce the priority inversion problem. For the proposed model, with some architectural support, we present an off-line approach to compute delivery guarantees of hard deadline real-time messages. We also perform a comparison of real-time WR and PPCS-RT in terms of performance with soft deadline traffic. The overall miss ratio percentage is over 30 percent higher for WR than PPCS-RT with one VC/link at high traffic loads. Finally, we compare the architectural complexity of a PPCS-RT router and other real-time routers.

A theory for total exchange in multidimensional interconnection networks

Total exchange (or multiscattering) is one of the important collective communication problems in multiprocessor interconnection networks. It involves the dissemination of distinct messages from every node to every other node. We present a novel theory for solving the problem in any multidimensional (cartesian product) network. These networks have been adopted as cost-effective interconnection structures for distributed-memory multiprocessors. We construct a general algorithm for single-port networks and provide conditions under which it behaves optimally. It is seen that many of the popular topologies, including hypercubes, k-ary n-cubes, and general tori satisfy these conditions. The algorithm is also extended to homogeneous networks with 2/sup k/ dimensions and with multiport capabilities. Optimality conditions are also given for this model. In the course of our analysis, we also derive a formula for the average distance of nodes in multidimensional networks; it can be used to obtain almost closed-form results for many interesting networks.

A DYNAMIC SCHEDULING COMMUNICATION PROTOCOL AND ITS ANALYSIS FOR HYPERCUBE NETWORKS

We propose a new protocol for one-to-one communication in multiprocessor networks, which we call the Dynamic Scheduling Communication (or DSC) protocol. In the DSC protocol, the capacity of a link is partitioned into two channels: a data channel, used to transmit packets, and a control channel used to make reservations. We initially describe the DSC protocol and the data structures needed to implement it for a general network topology. We then analyze the steady-state throughput of the DSC protocol for random node-to-node communication in a hypercube topology. The analytical results obtained are in very close agreement with corresponding simulation results. For the hypercube topology, and under the same set of assumptions on the node architecture and the routing algorithm used, the DSC protocol is found to achieve higher throughput than packet switching, provided that the size of the network is sufficiently large. We also investigate the relationship between the achievable throughput and the fraction of network capacity dedicated to the control channel, and present a method to select this fraction so as to optimize throughput.

Ring Embedding in Hypercubes with Faculty Nodes

Hypercube is an attractive structure for parellel processing due to its symmetry and regularity. To increase the reliability of hypercube based systems and to allow their use in the presence of faulty nodes, efficient fault-tolerant schemes in hypercubes are necessary. In this paper, we present an algorithm for embedding rings in hypercubes based multiprocessor network in the event of node failures. The algorithm can tolerate up to θ(2n/2) faults, and guarantee that given any f < (n - 2k)2k faulty nodes, it can find a ring of size at least 2n - 2f for k = 0 and 2n - 2k f - 22k for k ≥ 1 in an n-dimensional hypercube. It improves over existing algorithms in the size of ring.

Embedding of rings in 2-D meshes and tori with faulty nodes

To increase the reliability of networks embedded in meshes or tori and to allow their use in the presence of faulty nodes, efficient fault-tolerant schemes in meshes and tort are necessary. In this paper, we present algorithms for embedding a ring in a mesh-type and torus-type multiprocessor network in the event of node failures. In our approach, a mesh or torus is partitioned into 4×4 submeshes. Subgraphs (rings and linear arrays) are mapped in each submesh and then merged to form a large ring. Using this approach, we obtain the following results in a torus. Given any f ≤ 2 faulty nodes, a ring which is optimal can be found in a mesh. Also, given any f ≤ 4 faulty nodes, a ring which is optimal can be found in a torus. Furthermore, given f ≤ 4 faulty nodes, a Hamiltonian cycle can be found in a torus if and only if the number of even faulty nodes is equal to the number of odd faulty nodes. Even and odd refer to the sum of the node's coordinates.

Smart-pixel-based network interface chip

We present the design and experimental setup of an optically interconnected smart-pixel network interface chip designed to implement a collisionless multichannel-access control protocol. The design demonstrates the cointegration of optoelectronic pixel modules of various levels of complexity for dense, high-speed interconnection of highly functional digital logic components typical of multiprocessor network routers.

Circuit switching with input queuing: an analysis for the d-dimensional wraparound mesh and the hypercube

We analyze circuit switching in a multiprocessor network, where connection requests (or sessions) arrive at each node of the network according to a Poisson process with rate /spl lambda/. Each session joins the appropriate input-queue at its source node, and, upon advancing to the head of the queue, transmits a setup packet to establish a connection. If the setup packet is successful, it reserves the links on the path for the duration of the session, and the session is served without interruptions. Otherwise, the connection request remains queued at the source, and subsequent attempts are made to establish the circuit. We analyze the queue of connection requests at the input-buffer of a network link, and obtain analytic expressions for the stability region, the average queuing delay, the average connection time, the average waiting time, and the average total delay, which show how these parameters depend on system variables, such as network dimension and session arrival rate. The queuing analysis focuses on the input-queue of a particular link, and accounts for the interactions with queues of other links through the retrial attempts and the associated probability of success. The queuing analysis is independent of the particular network topology under consideration, as long as the probability that a session arriving at a random time successfully establishes a connection can be calculated for that network. Simulations demonstrate the close agreement between the observed network behavior and that predicted by the analysis.

The performance of the Cedar multistage switching network

While multistage switching networks for vector multiprocessors have been studied extensively, detailed evaluations of their performance are rare. Indeed, analytical models, simulations with pseudosynthetic loads, studies focused on average-value parameters, and measurements of networks disconnected from the machine, all provide limited information. In this paper, instead, we present an in-depth empirical analysis of a multistage switching network in a realistic setting: We use hardware probes to examine the performance of the omega network of the Cedar shared-memory machine executing real applications. The machine is configured with 16 vector processors. The analysis suggests that the performance of multistage switching networks is limited by traffic nonuniformities. We identify two major nonuniformities that degrade Cedar's performance and are likely to slow down other networks too. The first one is the contention caused by the return messages in a vector access as they converge from the memories to one processor port. This traffic convergence penalizes vector reads and, more importantly, causes tree saturation. The second nonuniformity is the uneven contention delays induced by a relatively fair scheme to resolve message collisions. Based on our observations, we argue that intuitive optimizations for multistage switching networks may not be the most cost-effective ones. Instead, we suggest changes to increase the network bandwidth at the root of the traffic convergence tree and to delay traffic convergence up until the final stages of the network.

Parallel computer vision on a reconfigurable multiprocessor network

A novel reconfigurable architecture based on a multiring multiprocessor network is described. The reconfigurability of the architecture is shown to result in a low network diameter and also a low degree of connectivity for each node in the network. The mathematical properties of the network topology and the hardware for the reconfiguration switch are described. Primitive parallel operations on the network topology are described and analyzed. The architecture is shown to contain 2D mesh topologies of varying sizes and also a single one factor of the Boolean hypercube in any given configuration. A large class of algorithms for the 2D mesh and the Boolean n-cube are shown to map efficiently on the proposed architecture without loss of performance. The architecture is shown to be well suited for a number of problems in low and intermediate level computer vision such as the FFT, edge detection, template matching, and the Hough transform. Timing results for typical low and intermediate level vision algorithms on a transputer based prototype are presented.

Performance Evaluation of Wire-Limited Hierarchical Networks

In the study of multiprocessor networks, attention must be paid to the effects of packaging constraints on cost and performance as well as the physical hierarchy imposed by packaging. Given wiring costs and constraints, hierarchical or clustered networks are a promising approach to building multiprocessor interconnects. We examine a simple approach to clustering: local buses connect processors within a cluster, and intercluster networks such ask-aryn-cubes and generalized shuffle–exchange networks provide connections between clusters. We perform detailed queueing analysis and simulations and compare the performance of a variety of flat and clustered networks. Previously, hierarchical networks were believed to be beneficial only for traffic with good locality. Under wiring cost models, we find that hierarchical networks may have superior performance to nonhierarchical networks, even under uniform traffic (which previously was believed to be a “worst case” scenario for hierarchical topologies). Relative performance is dependent on the cost constraint used, the system configuration and message granularity. The choice of a multiprocessor interconnect must take into consideration all these factors.

Distributed ring embedding in faulty De Bruijn networks

We present a distributed network-level algorithm that constructs a cycle in a d-ary De Bruijn multiprocessor network in the presence of an arbitrary number of node failures. When the number of faults f does not exceed d-1 a cycle of length at least d/sup n/-nf-1 can always be found in O(n) steps in a network of size d/sup n/.