Large-scale Multiprocessors Research Articles

Key Multimode Silicon Photonic Devices Inspired by Geometrical Optics

On-chip optical interconnect has been widely accepted as a promising technology to realize future large-scale multiprocessors. Mode-division multiplexing (MDM) provides a new degree of freedom for ...

The g-good-neighbor diagnosability of (n,k)-star graphs

Many large-scale multiprocessor or multi-computer systems take interconnection networks as underlying topologies. Fault diagnosis is especially important to identify fault tolerability of such systems. The g-good-neighbor (conditional) diagnosability such that every fault-free node has at least g fault-free neighbors is a novel measure of diagnosability. In this paper, we show that the g-good-neighbor diagnosability of the (n,k)-star graph Sn,k under the PMC model (2≤k≤n−1 and 1≤g≤n−k) and the comparison model (2≤k≤n−1 and 2≤g≤n−k) is n+g(k−1)−1, respectively. In addition, we derive that 1-good-neighbor diagnosability of Sn,k under the comparison model is n+k−2 for 3≤k≤n−1 and n≥4. As a supplement, we also derive that the g-good-neighbor diagnosability of the (n,1)-star graph Sn,1 (1≤g≤⌊n/2⌋−1 and n≥4) under the PMC model and the comparison model is ⌈n/2⌉−1, respectively.

A load-balanced routing method for large-scale mesh multiprocessors

Research on message routing in large-scale mesh multiprocessors usually focuses on routing in light traffic. Studies employ simple routing methods that are not load-balanced but do not need route initiation before message transmission. In non-light and/or non-uniform input traffic situations, maximum bandwidth utilization and average latency of such routing methods can be poor due to congestion caused by unbalanced load in global mesh regions. Existing load-balanced routing methods applied in mobile ad hoc mesh networks require route initiation before message transmission, which is not acceptable in mesh multiprocessors. In this research, we designed a load-balanced hierarchical mesh-routing method without route initiation. We organize a large-scale mesh into a virtually structured hierarchical mesh structure. Global mesh routing can thus be transformed into a number of load-balanced routing subtasks at levels of hierarchical submesh regions. In such a structure, we designed periodical hierarchical traffic state polling and distribution with very low overhead. We developed two congestion-avoidance routing methods for load-balanced routing at levels of submesh regions with polled traffic states. Experimental results show that maximum bandwidth utilization and average latency in non-light and/or non-uniform input traffic situations can be improved significantly in comparison with ordinary mesh multiprocessor routing methods.

Performance analysis in the real world

What issues are on the minds of industrial performance analysts? Four representatives of world-class product organizations will describe their work at the front lines of measurement, modeling, and performance tuning. Topics will include performance engineering of middleware at IBM, tools for detecting false sharing in large-scale multiprocessors at Hewlett-Packard, kernel thread-scheduling performance in multiprocessors at Microsoft, and low-overhead instrumentation for profiling large-scale services at Google. Plenty of time will be available to ask questions about how to direct our research to have the greatest impact on industrial practice.

Servo

Conventional programming models were designed to be used by expert programmers for programming for large-scale multiprocessors, distributed computational clusters, or specialized parallel machines. These models, therefore, are deemed either too difficult for an "average" programmer (who will be expected to do parallel programming in the many-core world) or too inefficient to use for many-core architectures [3]. Similarly, conventional execution models were designed for performance, not scalability. To address the challenge of performance scalability for many-core architectures, we introduce Servo, a Service Oriented programming model that decomposes every program into a set of components (each with its own local memory, mutable state, and a program counter) that either request services or deliver services. Servo is characterized by the decoupling of logical communication mapping between program modules (or services) from physical communication mapping between modules. This allows the services to be migrated and replicated during execution. The proposed model also allows the granularity of data parallel operations to be changed dynamically. This allows runtime variation in the locking granularity which in turn enables higher write-parallelism. Finally, the models partition even data into services. This can significantly enhance locality and makes it possible to have superlinear speedups with increasing number of cores. Our preliminary investigations demonstrate significant performance scalability advantages for Servo.

ON A NETWORK SENSING PROBLEM

Live traffic flow information can help improve the efficiency of a communication network. There are many ways available to monitor the traffic flow of a network. In this paper, we propose a very efficient monitoring strategy. This strategy not only reduces the number of nodes to be monitored but also determines the complete traffic information of the entire network using the information from the monitored nodes. The strategy is optimal for monitoring a network because it reduces the number of monitored nodes to a minimum. Fast algorithms are also presented in this paper to compute the traffic information for the entire network based on the information collected from the monitored nodes. The monitoring scheme discussed in this paper can be applied to the internet, telecommunication networks, wireless ad hoc networks, large scale multiprocessor computing systems, and other information systems where the transmission of information needs to be monitored.

Reliable Omega Interconnected Network for Large-Scale Multiprocessor Systems

The omega network has various attractive topological properties. It supports both one-to-one message routing and broadcast routing. Independent of the system size, every node in the network has a fixed size; therefore, it is used intensively in large-sized systems. In this paper, we examine a reliable omega-based multiprocessor system that preserves its full rigid omega configuration even in the presence of faults. The proposed omega interconnection network can tolerate any single and many multiple node failures, giving rise to significantly improved reliability. Reconfiguration in response to a single or multiple faults in the new design is easy and may be performed in a distributed manner. Unlike the reliable butterfly network, in the proposed reliable omega network, if a node at stage zero fails, the system will not lose a connection to one of its input/output ports. Reliability results show that our design compares favorably with an earlier one.

Modeling and evaluating the time overhead induced by BER in COMA multiprocessors

Designing multiprocessors based on distributed shared memory (DSM) architecture considerably increases their scalability. But as the number of nodes in a multiprocessor increases, the probability of encountering failures in one or more nodes of the system raises as a serious problem. Thus, every large-scale multiprocessor should be equipped with mechanisms that tolerate node failures. Backward error recovery (BER) is one of the most feasible strategies to build fault tolerant multiprocessors and it can be shown that among various DSM-based architectures, cache only memory architecture (COMA) is the most suitable for implementing BER. The main reason is the existence of built-in mechanisms for data replication in COMA memory system. BER is applicable to COMA multiprocessors with minor hardware redundancy, but it will obviously cause some other kinds of overheads. The most important overhead induced by BER is the time required to produce and store recovery data. This paper introduces an analytical model for predicting the amount of this time overhead and then verifies the correctness of the model through comparing the results predicted from this model with the previously published simulation results. Both the analytical model and simulation results show that the overhead is nearly independent of the number of nodes. The immediate result is that BER is a cost-effective strategy for tolerating node failures in large-scale COMA multiprocessors with large numbers of nodes.

Adaptive software prefetching in scalable multiprocessors using cache information

Scalable multiprocessors present special challenges to static software prefetching because on these systems the memory access latency is not completely determined at compile time. Furthermore, dynamic software prefetching cannot do much better because individual nodes on large-scale multiprocessors would tend to experience different remote memory delays over time. A fixed prefetch distance, even when computed at run-time, cannot perform well for the whole duration of a software pipeline. Here we present an adaptive scheme for software prefetching that makes it possible for nodes to dynamically change, not only the amount of prefetching, but the prefetch distance as well. We show how simple performance data collected by hardware monitors can allow programs to observe, evaluate and change their prefetching policies. Our results show that adaptive prefetching (APF) was capable of improving performance over static and dynamic prefetching by 10%–60%.

ADir/sub p/NB: a cost-effective way to implement full map directory-based cache coherence protocols

Directories have been used to maintain cache coherency in shared memory multiprocessors with private caches. The traditional full map directory tracks the exact caching status for each shared memory block and is designed to be efficient and simple. Unfortunately, the inherent directory size explosion makes it unsuitable for large-scale multiprocessors. In this paper, we propose new directory scheme, dubbed associative full map directory (ADir/sub p/NB) which reduces the directory storage requirement. The proposed ADir/sub p/NB uses one directory entry to maintain the sharing information for set of exclusively cached memory blocks in centralized linked list style. By implementing dynamic cache pointer allocation, reclamation, and replacement hints, ADir/sub p/NB can be implemented as a full map directory with lower directory memory cost. Our analysis indicates that, on typical architectural paradigm, ADir/sub p/NB reduces memory overhead of traditional full map directory by up to 70-80 percent. In addition to the low memory overhead, we show that the proposed scheme can be implemented with appropriate protocol modification and hardware addition. Simulation studies indicate that ADir/sub p/NB can achieve competitive performance with the Dir/sub p/NB. Compared with limited directory schemes, ADir/sub p/NB shows more stable and robust performance results on applications across spectrum of memory sharing and access patterns due to the elimination of directory overflows. We believe that ADir/sub p/NB can be employed as design alternative of full map directory for moderately large-scale and fine-grain shared memory multiprocessors.

Broadcast directory: A scalable cache coherent architecture for mesh-connected multiprocessors

Abstract Large-scale shared memory multiprocessors favor a directory-based cache coherence scheme for its scalability. The directory space needed to record the information for sharers has a complexity of Θ(N2) when a full-mapped vector is used for an N-node system. Although this overhead can be reduced by limiting the directory size assuming that the sharing degree is small, it will experience significant inefficiency when data is widely shared. In this paper, we propose a new directory scheme and a cache coherence scheme based on it for a mesh interconnection. Deterministic and wormhole routing enables a pointer to represent a set of nodes. Also a message traversing on the mesh performs a broadcast mission to a set of nodes without extra traffic, which can be utilized for the cache coherence protocol. Only a slight change on a typical router is needed to implement our scheme. This scheme is also applicable to any k-ary n-cube networks including a mesh. The Splash-2 parallel program suite is used in the simulation study, where our scheme is compared with other directory-based schemes. Our scheme is proved to generate much less traffic for cache coherence while the space complexity is more scalable Θ(N 3/2 log N) .

Hardware and compiler-directed cache coherence in large-scale multiprocessors: Design considerations and performance study

In this paper, we study a hardware-supported, compiler-directed (HSCD) cache coherence scheme, which can be implemented on a large-scale multiprocessor using off-the-shelf microprocessors, such as the Cray T3D. The scheme can be adapted to various cache organizations, including multiword cache lines and byte-addressable architectures. Several system related issues, including critical sections, interthread communication, and task migration have also been addressed. The cost of the required hardware support is minimal and proportional to the cache size. The necessary compiler algorithms, including intra- and interprocedural array data flow analysis, have been implemented on the Polaris parallelizing compiler. From our simulation study using the Perfect Club benchmarks, we found that in spite of the conservative analysis made by the compiler, for four of six benchmark programs tested, the proposed HSCD scheme outperforms the full-map hardware directory scheme up to 70 percent while the hardware scheme outperforms the HSCD scheme in the remaining two applications up to 89 percent. Given its comparable performance and reduced hardware cost, the proposed scheme can be a viable alternative for large-scale multiprocessors such as the Cray T3D, which rely on users to maintain data coherence.

Limited Combining Strategies for Large-Scale Shared-Memory Multiprocessors

In a large-scale shared-memory multiprocessor, there is a possibility of serious contention due to many requests issued concurrently for the same memory location. A multistage combining network, in which each switch is enhanced with combining so that multiple requests directed to the same memory location can form a single request, significantly reduces the amount of the contention. However, employing combining in every switch of a multistage interconnection network tends to increase the cost and to slow down the network. In this paper, assuming a single-job environment, we investigate some simple strategies that allow only a limited portion of a network to have a combining capability. We show that for situations with a limited number of hot spot locations, these simple strategies can provide performance comparable to a complete combining network in which every switch is enhanced with combining

A lock-based cache coherence protocol for scope consistency

Directory protocols are widely adopted to maintain cache coherence of distributed shared memory multiprocessors. Although scalable to a certain extent, directory protocols are complex enough to prevent it from being used in very large scale multiprocessors with tens of thousands of nodes. This paper proposes a lock-based cache coherence protocol for scope consistency. It does not rely on directory information to maintain cache coherence. Instead, cache coherence is maintained through requiring the releasing processor of a lock to store all write-notices generated in the associated critical section to the lock and the acquiring processor invalidates or updates its locally cached data copies according to the write notices of the lock. To evaluate the performance of the lock-based cache coherence protocol, a software DSM system named JIAJIA is built on network of workstations. Besides the lock-based cache coherence protocol, JIAJIA also characterizes itself with its shared memory organization scheme which combines the physical memories of multiple workstations to form a large shared space. Performance measurements with SPLASH2 program suite and NAS benchmarks indicate that, compared to recent SVM systems such as CVM, higher speedup is achieved by JIAJIA. Besides, JIAJIA can solve large scale problems that cannot be solved by other SVM systems due to memory size limitation.

An innovative implementation for directory-based cache coherence in shared memory multiprocessors

Directory-based cache coherence protocol is accepted as the common technique in large scale shared memory multiprocessors because of its scalability. Although it was extensively studied in the past, however, the memory overhead and long miss penalty entailed by this protocol are the major obstacles to scale for large scale multiprocessors. On the other hand, the ever-increasing cache line size makes the false sharing problem more serious than before, which will lead to high miss rate. Based on the scope consistency, we propose a lock-specific home-based cache coherence protocol. In this new cache coherence protocol, all directory memory overhead are eleminated completely and false sharing problem and high miss rate will be sloved greatly at the cost of small write notice buffer in each processor.

Performance impact of run queue organization and synchronization on large-scale NUMA multiprocessor systems

Abstract The goal of this paper is to study the impact of run queue organization on the performance of synchronization methods in multiprocessor systems. Two run queue organizations are considered: distributed and hierarchical organizations. The performance impact of spinning and blocking synchronization methods on these two run queue organizations is studied. We use two canonical workload types that require task synchronization: lock accessing and barrier synchronization workloads. The results presented here show that, when fine grain synchronization is required, the distributed organization is better. However, for large granularity tasks, the performance of the distributed organization is unacceptable and the hierarchical organization should be used. Note that the distributed organization is embedded into the hierarchical organization. Thus, for coarse granularity parallel applications, the hierarchical organization with its load sharing feature can be used; for fine-granularity parallel applications, the hierarchy of queues can be circumvented and the round robin task assignment can be done on processor's local queues as in the distributed organization. Therefore, the hierarchical organization is useful in general-purpose large-scale shared-memory multiprocessors.

Scheduler-conscious synchronization

Efficient synchronization is important for achieving good performance in parallel programs, especially on large-scale multiprocessors. Most synchronization algorithms have been designed to run on a dedicated machine, with one application process per processor, and can suffer serious performance degradation in the presence of multiprogramming. Problems arise when running processes block or, worse, busy-wait for action on the part of a process that the scheduler has chosen not to run. We show that these problems are particularly severe for scalable synchronization algorithms based on distributed data structures. We then describe and evaluate a set of algorithms that perform well in the presence of multiprogramming while maintaining good performance on dedicated machines. We consider both large and small machines, with a particular focus on scalability, and examine mutual-exclusion locks, reader-writer locks, and barriers. Our algorithms vary in the degree of support required from the kernel scheduler. We find that while it is possible to avoid pathological performance problems using previously proposed kernel mechanisms, a modest additional widening of the kernel/user interface can make scheduler-conscious synchronization algorithms significantly simpler and faster, with performance on dedicated machines comparable to that of scheduler-oblivious algorithms.

Evaluation of a Competitive-Update Cache Coherence Protocol with Migratory Data Detection

Although directory-based write-invalidate cache coherence protocols have a potential to improve the performance of large-scale multiprocessors, coherence misses limit the processor utilization. Therefore, so-called competitive-update protocols—hybrid protocols that on a per-block basis dynamically switch between write-invalidate and write-update—have been considered as a means to reduce the coherence miss rate and have been shown to be a better coherence policy for a wide range of applications. Unfortunately, such protocols may cause high traffic peaks for applications with extensive use of migratory objects. These traffic peaks can offset the performance gain of a reduced miss rate if the network bandwidth is not sufficient. We propose in this study to extend a competitive-update protocol with a previously published adaptive mechanism that can dynamically detect migratory objects and reduce the coherence traffic they cause. Detailed architectural simulations based on five scientific and engineering applications show that this adaptive protocol outperforms a write-invalidate protocol by reducing the miss rate and bandwidth needed by up to 71 and 26%, respectively.

Parallel Satisfiability Test with Synchronous Simulated Annealing on Distributed-Memory Multiprocessor

Solving the hard Satisfiability Problem is time consuming even for modest-sized problem instances. Solving the Random L-SAT Problem is especially difficult due to the ratio of clauses to variables. This report presents a practical approach to solving the Random L-SAT Problem on a large-scale distributed-memory multiprocessor. In particular, we use a parallel synchronous simulated annealing procedure, called Generalized Speculative Computation, which guarantees the same decision sequence as sequential simulated annealing. We have selected problem instances varying in size from 100-variable/425-clause to 5000-variable/21,250-clause. Experimental results on the AP1000 distributed-memory multiprocessor indicate that Generalized Speculative Computation of synchronous simulated annealing can satisfy 99.9% of the clauses while giving almost a 70-fold speedup on 500 processors.

Compiler and hardware support for cache coherence in large-scale multiprocessors

In this paper, we study a hardware-supported, compiler directed (HSCD) cache coherence scheme, which can be implemented on a large-scale multiprocessor using off-the-shelf microprocessors, such as the Cray T3D. It can be adapted to various cache organizations, including multi-word cache lines and byte-addressable architectures. Several system related issues, including critical sections, inter-thread communication, and task migration have also been addressed. The cost of the required hardware support is small and proportional to the cache size. The necessary compiler algorithms, including intra- and interprocedural array data-flow analysis, have been implemented on the Polaris compiler [17].From our simulation study using the Perfect Club benchmarks, we found that, in spite of the conservative analysis made by the compiler, the performance of the proposed HSCD scheme can be comparable to that of a full-map hardware directory scheme. With its comparable performance and reduced hardware cost, the scheme can be a viable alternative for large-scale multiprocessors, such as the Cray T3D, that rely on users to maintain data coherence.