Self-timed Interface Research Articles

IBM System z10 I/O subsystem

The performance, reliability, and functionality of a large server are greatly influenced by the design characteristics of its I/O subsystem. The critical components of the IBM System z10™ I/O subsystem have, therefore, been significantly improved in terms of performance, capability, and cost. The first-order network has been redesigned from the long-evolved enhanced self-timed interface (eSTI) links to utilize InfiniBand™ links. A redesign of the host logic of I/O chips and the fiberoptic interfaces within the links made it possible to introduce InfiniBand-based IBM Parallel Sysplex® links. A broad range of legacy I/O channels have been carried forward to connect through InfiniBand, and a foundation has been laid for new channel types of improved functionality and performance. The first such hardware channel to be introduced is the next generation of Ethernet-virtualization data routers. A new and methodical recovery structure has been designed to ensure consistent, extensive support of reliability, availability, and serviceability. A building-block-oriented design process has been developed to enable the innovations that made these advances possible. Finally, a new performance verification methodology has been introduced to ensure that the system and subsystem designs are balanced to make effective use of the increased capacity.

Functional verification of a frequency-programmable switch chip with asynchronous clock sections

An integral part of the IBM eServer™ z990 I/O subsystem is the self-timed interface (STI) switch chip. The STI switch is an application-specific integrated circuit (ASIC) designed to provide high I/O connectivity and high bandwidth within the system. The complexity of the functional verification of the STI switch chip is inherent in the implementation of seventeen logical clock domains and the support of six different STI interfaces with programmable frequencies. The logic within these clock domains is connected via asynchronous interfaces. This paper describes the methodology to verify the functionality of the switch chip with various STIs by introducing a combination of verification techniques. This involves random biased stimulus generation, automated result prediction checking, and the use of cycle simulation to stress the logical design. The cycle simulation required new techniques to model equivalent behavior in order to verify the correct integration of nondigital components on the chip. Advanced methods were implemented to ensure correctness of the frequency-dependent design units and functionality across the asynchronous interfaces. A single verification environment was developed, providing the flexibility to seamlessly support the different levels of design abstraction and uncover the design errors at the appropriate level.

First- and second-level packaging of the z990 processor cage

In this paper, we describe the challenging first- and second-level packaging technology of a new system packaging architecture for the IBM eServer™ z990. The z990 dramatically increases the volumetric processor density over that of the predecessor z900 by implementing a super-blade design comprising four node cards. Each blade is plugged into a common center board, and a blade contains the node with up to sixteen processor cores on the multichip module (MCM), up to 64 GB of memory on two memory cards, and up to twelve self-timed interface (STI) cables plugged into the front of the node. Each glass-ceramic MCM carries 16 chips dissipating a maximum power of 800 W. In this super-blade design, the packaging complexity is increased dramatically over that of the previous zSeries® eServer z900 to achieve increased volumetric density, processor performance, and system scalability. This approach permits the system to be scaled from one to four nodes, with full interaction between all nodes using a ring structure for the wiring between the four nodes. The processor frequencies are increased to 1.2 GHz, with a 0.6-GHz nest with synchronous double-data-rate interchip and interblade communication. This data rate over these package connections demands an electrical verification methodology that includes all of the different relevant system components to ensure that the proper signal and power distribution operation is achieved. The signal integrity analysis verifies that crosstalk limits are not exceeded and proper timing relationships are maintained. The power integrity simulations are performed to optimize the hierarchical decoupling in order to maintain the voltage on the power distribution networks within prescribed limits.

The structure of chips and links comprising the IBM eServer z990 I/O subsystem

The performance of large servers is to a high degree determined by their I/O subsystems. In the z990 server, nearly all of the components in the I/O path have been considerably improved in performance, capability, and cost. A 2-GB/s enhanced self-timed interface (eSTI) was introduced which is capable of absorbing the ever-increasing data rates of modern high-speed adapters. The I/O bandwidth available from a single node (three memory bus adapter, or MBA, chips, each with four eSTI ports) now equals 48 GB/s. As a consequence, both the MBA chip and the STI multiplexer switch (STI switch) chip had to be completely redesigned. In addition to these two chips, this paper describes the eSTI design itself and the Sweep chip, which integrates the function of four bidirectional adapter chips, one switch chip, and a clock chip.

Self-timed interface of the input/output subsystem of the IBM eServer z900

The self-timed interface (STI), used in several generations of IBM's most powerful servers, has evolved to provide a greater I/O subsystem bandwidth than ever before. The STI of the IBM eServer z900 is capable of transferring data between the processor complex and the I/O subsystem at a rate of up to 1 GB/s compared with the 333MB/s rate of the IBM S/390® G3/G4- and G5/G6-class large servers—an improvement of up to a factor of 3. Additionally, in the eServer z900, STI links suitable for transferring data at a rate of 1 GB/s are used for the third iteration of the integrated cluster bus (ICB-3), thus providing direct links to other eServer z900s. Also, the role of the STI has increased in the eServer z900, becoming the interface used on the I/O subsystem backplane. The majority of the STI logic (both physical and logical layers) has remained essentially unchanged, with advances resulting from the use of improved cables and connectors, and from the introduction of a multilevel, pre-distortion differential off-chip “super-driver.”

Hierarchical indexing data structure method for verifying the functionality of the STI-to-PCI bridge chips of the IBM eServer z900

The IBM eServer z900 has an overall system I/O bandwidth which is three times that of IBM S/390® G5/G6 servers, necessitating the use of Self-Timed-Interface (STI)-to-Peripheral-Component-Interface (PCI) bridge chips to exploit this bandwidth. The chips are used to form a layer between the networking attachments of the z900 and its main storage complex. The layer adapts the high-speed point-to-point packet-oriented STI interface of the z900 to its multi-drop PCI bus structure. This paper describes a method for verifying the functionality of the STI-to-PCI bridge chips by implementing a hierarchical indexing method to support all address dispatching, data management, and data integrity checking. The method is at the core of the random-element-level verification methodology to support all data movement mainline testing of the z900. Monitors, checkers, and drivers were developed and integrated as part of the overall methodology to verify all external interfaces.

IBM eServer z900 I/O subsystem

The IBM eServer z900 is the first in a generation of future eServers that continues its leadership via a new I/O subsystem with enhancements in capability, performance, configuration management, and qualities of service. Significant features of the I/O subsystem are included to support the 64-bit z/Architecture™ and configuration-management enhancements [e.g., assignable channel path identifiers (CHPIDs) and dynamic channel path management (DCM)]. A 1GB/s self-timed interface (STI), I/O infrastructure, and I/O card cage are described which support the high-bandwidth I/O (e.g., FICON™, Ethernet). These improvements yield enhanced configuration flexibility and connectivity, as well as reliability, availability, and serviceability (RAS), while providing for future bandwidth growth. The various types of I/O ports supported by the IBM eServer z900 platform are also discussed. A common I/O platform is discussed which has been used to provide a uniform, high-bandwidth attachment of industry-standard peripheral computer interface (PCI) cards, while maintaining the leadership functionality and RAS of the eServer zSeries™. A high-density (16-port) ESCON® I/O card has been designed by exploiting IBM advanced CMOS and state-of-the-art fiber optic technologies. Finally, a high-performance, high-density intersystem channel (ISC-3) coupling link I/O card has been developed for the IBM eServer z900 by leveraging advanced technology and packaging techniques.

Self-timed interface for S/390 I/O subsystem interconnection

A high-speed interface has been designed for interconnection of the S/390® I/O subsystem to the IBM S/390 G5 processor. The self-timed interface (STI) provides high bandwidth, greater communication distances, and simpler timing within the S/390 servers than traditional interfaces. The STI communicates between the memory bus adapter and the expanded S/390 I/O subsystem, which now includes the new S/390 fiber channel offering (FICON™) and other network-based protocols such as ATM, Fast Ethernet, and Gigabit Ethernet. Also new for G5 is the use of STI for the integrated cluster bus (ICB), providing direct links among multiple G5 servers. The STI communicates over cables up to ten meters in length at a clock frequency of 167 MHz. Data is sent at twice the clock frequency (333 MB/s). The hardware implementation comprises specially designed logic macros, differential drivers and receivers, and the cables and connectors. The receive macro accommodates up to three bittimes of skew by retiming each data bit of the interface to the transmitted clock. This paper describes the STI logic, the characteristics of the link, and the transmission and reception of the data (and clock).

The Integrated Cluster Bus for the IBM S/390 Parallel Sysplex

IBM has developed a new S/390® Parallel Sysplex® coupling interface for the G5 server called the Integrated Cluster Bus (ICB). This interface improves the coupling efficiency by greatly reducing message-passing latency. Using the transport layer of the S/390 self-timed interface (STI) introduced in the G3 server, ICB adds channel function to the hub chip to allow a more direct interconnection between S/390 servers. This new channel has the same function as the present intersystem channel (ISC), but because it is integrated into the hub chip and therefore requires no additional components, its reliability is much better than that of the ISC. Since the ISC transmits data at a peak rate of 106 MB/s over distances exceeding ten kilometers and the ICB transmits data at a peak rate of 333 MB/s at distances of ten meters, the ISC is still required for the more geographically dispersed Parallel Sysplexes, whereas the ICB is well suited to the machine room, where multiple servers ca n be interconnected by ten-meter cables. This paper describes the design approach for the ICB. It describes the fundamental message-passing requirements of the Parallel Sysplex and how they are implemented in very complex yet compact hardware in the server's hub chip.