PCI Express Research Articles

A high-speed data acquisition system based on FPGA for tokamak.

The Experimental Advanced Superconducting Tokamak (EAST) device aims to achieve a steady-state and long-pulse discharge over 1000 s. An embedded high-speed data acquisition system based on a field-programmable gate array (FPGA) for EAST is designed in this study. A cyclone FPGA is used as the master chip, and a TI's analog-to-digital conversion (ADC) chip is used to complete ADC. One acquisition system board consists of four ADC chips. The acquired data are compressed and stored into a disk array through a Peripheral Component Interconnect (PCI) Express interface and then uploaded to the data server. One board can collect the signals of eight channels synchronously. A number of such boards can be used to collect additional channel signals. Experimental results show that the system can reach 80 MSps and the sampling precision can reach 12 bits with 1250 s continuous sampling. The system integrates signal conditioning, data acquisition, and data processing into a single board and provides an architecture with high integration and portability levels.

The ALICE O2 common driver for the C-RORC and CRU read-out cards

ALICE (A Large Ion Collider Experiment) is the heavy-ion detector designed to study the strongly interacting state of matter realized in relativistic heavy-ion collisions at the CERN Large Hadron Collider (LHC). A major upgrade of the experiment is planned during the 2019-2020 long shutdown. In order to cope with a data rate 100 times higher than during LHC Run 1 and with the continuous read-out of the Time Projection Chamber (TPC), it is necessary to upgrade the Online and Offline Computing to a new common system called O2. The O2 read-out chain will use commodity x86 Linux servers equipped with custom PCIe FPGA-based read-out cards. This paper discusses the driver architecture for the cards that will be used in O2: the PCIe v2 x8, Xilinx Virtex 6 based C-RORC (Common Readout Receiver Card) and the PCIe v3 x16, Intel Arria 10 based CRU (Common Readout Unit). Access to the PCIe cards is provided via three layers of software. Firstly, the low-level PCIe (PCI Express) layer responsible for the userspace interface for low-level operations such as memory mapping the PCIe BAR (Base Address Registers) and creating scatter-gather lists, which is provided by the PDA (Portable Driver Architecture) library developed by the Frankfurt Institute for Advanced Studies (FIAS). Above that sits our userspace driver which implements synchronization, controls the read-out card – e.g. resetting and configuring the card, providing it with bus addresses to transfer data to and checking for data arrival – and presents a uniform, high-level C++ interface that abstracts over the differences between the C-RORC and CRU. This interface – of which direct usage is principally intended for high-performance read-out processes – allows users to configure and use the various aspects of the read-out cards, such as configuration, DMA transfers and commands to the front-end. The top layer consists of a Python wrapper and command-line utilities that are provided to facilitate scripting and executing tasks from a shell, such as card resetting; performing benchmarks; reading or writing registers; and running test suites. Additionally, the paper presents the results of benchmarks in various test environments. Finally, we present our plans for future development, testing and integration.

Performance assessment of a software-based coincidence processor for the EXPLORER total-body PET scanner

Coincidence processing in positron emission tomography (PET) is typically done during acquisition of the data. However, on the EXPLORER total-body PET scanner we plan, in addition, to store unpaired single events (i.e. singles) for post-acquisition coincidence processing. A software-based coincidence processor was developed for EXPLORER and its performance was assessed. Our results showed that the performance of the coincidence processor could be significantly impacted by the type of data storage (Peripheral Component Interconnect Express (PCIe)-attached solid state drive (SSD) versus RAID 6 hard disk drives (HDDs)) especially when multiple data files were processed in parallel. We showed that a 48-thread computer node with dual Intel Xeon E5-2650 v4 central processing units (CPUs) and a PCIe SSD was sufficient to process approximately 120 M singles s−1 at an incoming singles rate of approximately 150 Mcps. With two computer nodes, near real-time coincidence processing became possible at this incoming singles rate.

A multi-channel PCI express readout board for fast readout of large pixel detectors

Recently the ATLAS Pixel Detector at CERN has been upgraded by inserting an additional layer of pixels, the Insertable B-Layer (IBL). In addition, the off-detector readout electronics of the other layers that composed the Pixel Detector (the B-Layer, the Layer 1, the Layer 2 and the Disks) were updated using the IBL readout boards. The system has been updated, one layer at a time, giving priority to the next critical layer as the luminosity and level-1 trigger frequency increased. Hence, after IBL, the first critical layer was the Layer 2, then the Layer 1 and finally the B-Layer and the Disks. Eventually, after the technical stop in 2018 the entire ATLAS Pixel Detector will share the same off-detector readout electronics.In parallel with the commissioning of the upgrade of the current ATLAS Pixel Detector we have designed and fabricated a new readout electronic board to address the requirements of the LHC Phase-2 upgrade. Two batches of prototypes of a Peripheral Component Interconnect Express (PCIe) Gen. 2 boards have been designed and fabricated, the second being a patched version of the first. The first batch was composed of two boards, called Pixel ReadOut Driver (Pixel_ROD) and the second batch was made of five cards called π-LUP. All the boards feature many of the input–output ports and interfaces to address the requirements of the future front-end electronics being developed for the Large Hadron Collider (LHC) Phase-2 upgrade. Thus, the current VERSABUS Module Eurocard (VME) bus will be replaced with the PCIe bus to accommodate the huge increase of throughput (data to be transferred to the DAQ). In this new scenario, the GigaBit Transceiver (GBT) and Aurora protocols are compatible with our boards and the GBTx and RD53A chips will be the first components to be interfaced with. Some laboratory results and measurements are presented here.

FELIX-Based Readout of the Single-Phase ProtoDUNE Detector

Large liquid argon (LAr) time projection chambers (TPCs) have been adopted for the Deep Underground Neutrino Experiment (DUNE) experiment's far detector, which will be composed of four 17-kton detectors situated 1.5 km underground at the Sanford Underground Research Facility. This represents a large increase in scale compared to existing experiments. Both single- and dual-phase technologies will be validated at CERN, in cryostats capable of accommodating full-size detector modules, and exposed to low-energy charged particle beams. This program, called ProtoDUNE, also allows for extensive tests of data acquisition strategies. The Front-End LInk eXchange (FELIX) readout system was initially developed within the ATLAS collaboration and is based on custom field-programmable gate array (FPGA)-based Peripheral Component Interconnect Express input/output cards, connected through point-to-point links to the detector front end and hosted in commodity servers. FELIX will be used in the single-phase ProtoDUNE setup to read the data coming from 2560 anode wires organized in a single anode plane assembly (APS) structure. With a continuous readout at a sampling rate of 2 MHz, the system must deal with an input rate of 96 Gb/s. An external trigger will preselect time windows of 5 ms with interesting activity expected inside the detector. Event building will occur for triggered events, at a target rate of 25 Hz; the readout system will form fragments from the data samples matching the time window, carry out lossless compression, and forward the data to event building nodes over 10-Gb/s Ethernet. This paper discusses the design and implementation of this readout system as well as the first operational experience.

Limitations of intra datacentre communications exploiting optical PCIe interconnections

Abstract To support the huge intra-datacentre traffic growth, network architectures based on Peripheral Component Interconnect Express (PCIe) communication protocol have been proposed in alternative to Ethernet ones. To cover the entire datacentre area and to promote the recently released PCIe Gen4, operating at 16 Gb/s, the employment of optical fibre communication is mandatory. We demonstrate, by physical layer measurements, the compliance to the PCIe standard in terms of eye diagram mask and jitter for both Gen3 and Gen4 optical PCIe links, considering up to 10-km standard single mode fibre (SMF) propagation and transmission in the second window of optical communications. On the other hand, we perform an analysis to evidence that the maximum fibre reach is limited by the propagation latency due to the PCIe protocol ACK/NACK operation. For 1-lane PCIe Gen3 link we experimentally verify, through a protocol analysis, the reach limits expected by our evaluation; moreover, for longer distances, we detect retransmissions which reduce the actual transmitted capacity, without error introduction.

EPICS data acquisition system for FPGA-based plasma electron density measurement on KSTAR

The Korea Superconducting Tokamak Advanced Research (KSTAR) has several diagnostics devices, including a millimeter-wave interferometer (MMWI), far-infrared interferometer (FIR), two-color interferometer, etc., for measuring plasma electron densities. The MMWI diagnostic device measures the electron density of the plasma from the phase difference generated by transmitting a 1 mm (280 GHz) electromagnetic wave into it. The FIR device is similar to the MMWI, but uses a far-infrared (119 um) laser as its electromagnetic wave source. The MMWI is the primary plasma density measurement system in KSTAR. However, its signal contains significant noise and hundreds of fringe-jump events from the phase comparator circuit. The original analogue circuit of the phase comparator is limited in noise handling since it needs to average the raw IF signal to be treated in analogue circuit with limited bandwidth. To handle noise with more flexibility and to reduce occurrence of fringe-jump errors, we developed a new data acquisition (DAQ) system using a high-speed FPGA-based digitizer. The MMWI signal resembles a 10 MHz sine wave. We acquired data at 100 MSPS using a new digitizer. The phase difference per sample was 36°, which lead to inaccurate results due to noise. To increase the accuracy, the new DAQ system has interpolation logic within the FPGA that increases the phase resolution. In addition, the phase comparative value is transmitted to two channels by inverting the signal. The signal that does not travel through the plasma has a phase difference of 180°. The FPGA-based system was constructed using the same algorithm as existing phase comparator circuit [2]. The detected phase was averaged to obtain a phase value, and the result was transmitted as 1280 phase values per msec. In the EPICS Input/Output Controller (IOC), the density is calculated using data received from the direct memory access (DMA) via peripheral component interconnect (PCI) Express. The above FPGA logic was successfully used for real-time feedback control in the ITER CODAC environment. This paper introduces a new, detailed MMWI DAQ system design concept using FPGA, and its operational results.

Taking advantage of the intercommunication features of IPMCs in ATCA CDAQ systems

The PCI Industrial Computer Manufacturers Group (PICMG) 3.x AdvancedTCA Base Specification and Intelligent Platform Management Interface (IPMI) standards specify a modular system for Control and Data Acquisition (CDAQ) where the boards can communicate through the Intelligent Platform Management Bus-0 (IPMB-0).CDAQ systems applied to large physics experiments like ITER, are designed, among other features, for High-Availability (HA). A CDAQ system based on the PICMG 3.x AdvancedTCA Base Specification and IPMI standards grants these features. One of the key functions of the HA is the hot swapping possibility of the CDAQ system elements which is granted by those standards. An Advanced Telecommunications Computers Architecture (ATCA) CDAQ system can be designed implementing the Peripheral Component Interconnect Express (PCIe) standard communication protocol in their fabric lines. Despite the fact that this protocol has its own rules for hot swapping, defined as hot-plug, ATCA and PCIe standards are not entirely compatible in what concerns this aspect.Instituto de Plasmas e Fusão Nuclear (IPFN) has developed and implemented a CDAQ system using the ATCA and PCIe standards.This paper describes the software architecture and functions implemented in the microcontroller of the Intelligent Platform Management Controller (IPMC) of the ATCA boards in order to render those standards compatible in what concerns hot-swap and hot-plug processes as an example of the possibilities that can be achieved taking advantage of the intercommunication features of IPMCs through the Intelligent Platform Management Bus-0 (IPMB-0).

Design and implementation of initial OpenSHMEM on PCIe NTB based cloud computing

Cloud computing services are provided by key roles of data centers in which homogeneous and heterogeneous computation nodes are connected by high speed interconnection network. The rapid development of cloud-based services and applications has made data center networks more difficult. The PCI Express is a widely used system bus technology that connects processors and peripheral I/O devices. So, the PCI Express is regarded as a de-facto standard in system area interconnection network. It is currently validating the possibility of using PCI Express as a system interconnection network in areas such as high-performance computers and cluster/cloud computing. With the development of PCI Express non-transparent bridge (NTB) technology, the PCI Express has become available as a system interconnection network. NTB allows two PCI Express subsystems to be interconnected and, if necessary, isolated from each other. Partitioned global address space (PGAS) is one of the shared address space programming models. Due to the recent spread of multicore processors, PGAS has been attracting attention as a parallel computing framework. We make use of the PCI Express NTB to realize the PGAS shared address space model. In this paper, we designed and implemented the interconnection network using PCI Express x8 using a RDK, the PEX8749 based PCI Express evaluation board. We performed some Openshmem applications from Github to verify the accuracy of our initial OpenSHMEM API implementation.

MBSPEX and PEXORNET—Linux Device Drivers for PCIe Optical Receiver DAQ and Control

The GSI PCI express (PCIe) optical receiver (PEXOR) family PCIe boards are used as an interface for data acquisition (DAQ) from various detector front ends, linked by up to four chains of optical fiber connections. Communication with the front-end hardware is handled by the proprietary gigabit optical serial interface protocol. A trigger module trigger for pexor extends the PEXOR by additional signal connections for triggered DAQ. For several years, the PEXOR boards have been used with the DAQ framework multibranch system (MBS). On a Linux x-86 platform, the device driver software mbspex implements concurrent access to the PEXOR front ends from MBS DAQ and from separate control applications, such as the command line tool gosipcmd or hardware specific configuration graphical user interfaces. Besides the established character driver mbspex , a network driver pexornet has been developed to evaluate a lightweight DAQ system with readout from PEXOR via a user datagram protocol (UDP) socket. Therefore, common network tools can be applied for driver configuration and data debugging. Moreover, the gosipcmd tool and its adjusted application programming interface library are fully applicable also for pexornet . A simple example DAQ application with pexornet UDP readout has been implemented with the software framework DAQ backbone core (DABC), delivering the same data file format and online monitoring capabilities as MBS. Readout performance of a test setup has been measured both with MBS/ mbspex and with DABC/ pexornet .

IOMMU protection against I/O attacks: a vulnerability and a proof of concept

Input/output (I/O) attacks have received increasing attention during the last decade. These attacks are performed by malicious peripherals that make read or write accesses to DRAM memory or to memory embedded in other peripherals, through DMA (Direct Memory Access) requests. Some protection mechanisms have been implemented in modern architectures to face these attacks. A typical example is the IOMMU (Input-Output Memory Management Unit). However, such mechanisms may not be properly configured and used by the firmware and the operating system. This paper describes a design weakness that we discovered in the configuration of an IOMMU and a possible exploitation scenario that would allow a malicious peripheral to bypass the underlying protection mechanism. The exploitation scenario is implemented for Intel architectures, with a PCI Express peripheral Field Programmable Gate Array, based on Intel specifications and Linux source code analysis. Finally, as a proof of concept, a Linux rootkit based on the attack presented in this paper is implemented.

FPGA Implementations of Kernel Normalised Least Mean Squares Processors

Kernel adaptive filters (KAFs) are online machine learning algorithms which are amenable to highly efficient streaming implementations. They require only a single pass through the data and can act as universal approximators, i.e. approximate any continuous function with arbitrary accuracy. KAFs are members of a family of kernel methods which apply an implicit non-linear mapping of input data to a high dimensional feature space, permitting learning algorithms to be expressed entirely as inner products. Such an approach avoids explicit projection into the feature space, enabling computational efficiency. In this paper, we propose the first fully pipelined implementation of the kernel normalised least mean squares algorithm for regression. Independent training tasks necessary for hyperparameter optimisation fill pipeline stages, so no stall cycles to resolve dependencies are required. Together with other optimisations to reduce resource utilisation and latency, our core achieves 161 GFLOPS on a Virtex 7 XC7VX485T FPGA for a floating point implementation and 211 GOPS for fixed point. Our PCI Express based floating-point system implementation achieves 80% of the core’s speed, this being a speedup of 10× over an optimised implementation on a desktop processor and 2.66× over a GPU.

A High-Performance and Hardware-Efficient PCIe Transmission for a Multi-Channel Video Using Command Caching and Dynamic Splicing on FPGA

With the increasing requirements of transmitting larger amounts of diverse video and image data, the PCI express (PCIe) interface has been extensively used in high-speed digital systems for multimedia processing and communication. However, the existing PCIe transmission methods are not efficient to transmit multi-channel videos on FPGA processors, as they waste PCIe bandwidth and hardware resources. In this paper, an efficient PCIe transmission method for multi-channel video is presented on an FPGA processor. First, to reduce the programmed I/O (PIO) write latency, the command caching method is adopted. The traditional DMA descriptors are transformed into DMA commands, which are smaller, and the command buffer is used to store these commands. Second, a dynamic splicing mechanism is proposed for data transmission and commands the scheduling separately. The multi-type data is transmitted concurrently in this dynamic splicing mechanism. Moreover, the adjacent commands are spliced into one command, which reduces the DMA initiation times. Finally, the direct kernel memory access technique and timer self-feedback monitor technique are applied to improve the transmission efficiency and enhance the system reliability. The experimental results show that the transmission efficiency of the proposed method is increased by nearly 30% compared with other conventional methods, and the highest measured transfer rate is 1631 Mbytes/s on write and 1582 Mbytes/s on read, which reaches 84.4% of the theoretical maximum.

1 Gbit/s UDP/IP Offload Engine IP Core with PCIe Interface

A significant amount of processor power is required to handle packet processing in high speed data networks and taking it to the hardware helps processor to save its energy for other processes. In this study, an Offload Engine IP core that provides the hardware acceleration of UDP/IP protocol stack together with a few other network protocols is introduced. Furthermore, the IP core is equipped with PCI Express (PCIe) interface so as to communicate with applications running on a host PC. Consequently, a processor core deals with only the data processing, while the IP core takes care of the packet processing as per the protocol. The design and implementation of the IP core are verified and tested on an FPGA board; its area utilization and supported features are compared against several competitive designs from the literature. According to these results, the IP core is proved to be a useful one for those network applications that require a hardware-accelerated network protocol stack.

GPU-accelerated track reconstruction in the ALICE High Level Trigger

ALICE (A Large Heavy Ion Experiment) is one of the four major experiments at the Large Hadron Collider (LHC) at CERN. The High Level Trigger (HLT) is an online compute farm which reconstructs events measured by the ALICE detector in real-time. The most compute-intensive part is the reconstruction of particle trajectories called tracking and the most important detector for tracking is the Time Projection Chamber (TPC). The HLT uses a GPU-accelerated algorithm for TPC tracking that is based on the Cellular Automaton principle and on the Kalman filter. The GPU tracking has been running in 24/7 operation since 2012 in LHC Run 1 and 2. In order to better leverage the potential of the GPUs, and speed up the overall HLT reconstruction, we plan to bring more reconstruction steps (e.g. the tracking for other detectors) onto the GPUs. There are several tasks running so far on the CPU that could benefit from cooperation with the tracking, which is hardly feasible at the moment due to the delay of the PCI Express transfers. Moving more steps onto the GPU, and processing them on the GPU at once, will reduce PCI Express transfers and free up CPU resources. On top of that, modern GPUs and GPU programming APIs provide new features which are not yet exploited by the TPC tracking. We present our new developments for GPU reconstruction, both with a focus on the online reconstruction on GPU for the online offline computing upgrade in ALICE during LHC Run 3, and also taking into account how the current HLT in Run 2 can profit from these improvements.

A Hypervisor Approach to Enable Live Migration with Passthrough SR-IOV Network Devices

Single-Root I/O Virtualization (SR-IOV) is a specification that allows a single PCI Express (PCIe) device (physical function or PF) to be used as multiple PCIe devices (virtual functions or VF). In a virtualization system, each VF can be directly assigned to a virtual machine (VM) in passthrough mode to significantly improve the network performance. However, VF passthrough mode is not compatible with live migration, which is an essential capability that enables many advanced virtualization features such as high availability and resource provisioning. To solve this problem, we design SRVM which provides hypervisor support to ensure the VF device can be correctly used by the migrated VM and the applications. SRVM is implemented in the hypervisor without modification in guest operating systems or guest VM drivers. SRVM does not increase VM downtime. It only costs limited resources (an extra CPU core only during the live migration pre-copy phase), and there is no significant runtime overhead in VM network performance.

A wide range real-time synchronous demodulation system for the dispersion interferometer on HL-2M.

A real-time synchronous demodulation system has been developed for the dispersion interferometer on a HL-2M tokamak. The system is based on the phase extraction method which uses a ratio of modulation amplitudes. A high-performance field programmable gate array with pipeline process capabilities is used to realize the real time synchronous demodulation algorithm. A fringe jump correction algorithm is applied to follow the fast density changes of the plasma. By using the Peripheral Component Interconnect Express protocol, the electronics can perform real-time density feedback with a temporal resolution of 100 ns. Some experimental results presented show that the electronics can obtain a wide measurement range of 2.28 × 1022 m-2 with high precision.

Design on PCIe Real-time Measurement and Control Platform for MEMS Gyroscope

This paper introduces a PCIe (PCI Express) real-time measurement and control platform for MEMS (Micro-Electro-Mechanical System) gyroscopes. Through optimization of the PCIe and FPGA (Field Programmable Gate Array) drivers as well as the delay link, the platform realized the closed-loop control of gyroscope in both the driving frequency and amplitude. Consequently, the linear vibration gyroscope could work at the resonant frequency and the amplitude was stable. Considering the real-time performance of the platform and stability of data transmission, analysis from the aspects of software and hardware are presented here. For the PCIe bus transmission, the hardware of FPGA was optimized. The readout time of single data was reduced to 1.8 μs by accelerating controls of bus transmission and PCIe IP core interface. For the system software, the underlying driver and algorithm were streamlined so that the data transmission delay between kernel layer and user layer was decreased. The development of highly-efficient timing control proposal ensured stabilization in data transmission. The proposed platform could reach a 100 kHz sampling and processing rate in PC system, which is enough for the current requirements. Furthermore, the proposed high-rate real-time platform can also be used in more real-time high-demanding systems.

Readout electronics for CBM-TOF super module quality evaluation based on 10 Gbps ethernet

The Compressed Baryonic Matter-Time of Flight (CBM-TOF) wall uses high performance of Multi-gap Resistive Plate Chambers (MRPC) assembled in super modules to identify charged particles with high channel density and high measurement precision at high event rate. Electronics meet the challenge for reading data out from a super module at high speed of about 6 Gbps in real time. In this paper, the readout electronics for CBM-TOF super module quality evaluation is proposed based on 10 Gigabit Ethernet. The digitized TOF data from one super module will be concentrated at the front-end electronics residing on the side of the super module and transmitted to an extreme speed readout module (XSRM) housed in the backend crate through the PCI Express (PCIe) protocol via optic channels. Eventually, the XSRM transmits data to the data acquisition (DAQ) system through four 10 Gbps Ethernet ports in real time. This readout structure has advantages of high performance and expansibility. Furthermore, it is easy to operate. Test results on the prototype show that the overall data readout performance for each XSRM can reach up to 28.8 Gbps, which means XSRM can meet the requirement of reading data out from 4 super modules with 1280 channels in real time.

PCIe hot-plug event handling tasks using PICMG standard interrupt mechanism for ATCA based instrumentation

The Advanced Telecommunications Computing Architecture (AdvancedTCA®) specification implements important key features such as high reliability, high availability, redundancy and serviceability for control and data acquisition instrumentation fault condition, hardware malfunction, firmware updates and hardware reconfiguration. Red Hat Enterprise Linux and corresponding kernels already have built-in mechanisms and embedded software for Peripheral Component Interconnect Express (PCIe) hot-plug support that allows automatic removal of PCIe device nodes and associated device files from the system providing a fast replacement strategy for damaged cards without requiring an entire system shutdown. This paper describes handling of PCIe hot-plug events at a software middle level using the PCI Industrial Computer Manufacturers Group (PICMG) standard interrupt mechanism. The handling tasks can be accomplished by AdvancedTCA® cards chipsets with support for PCIe hot-plug features, Linux hot-plug embedded controller and Red Hat built-in device manager module. The goal is to implement a fast hardware replacement solution without system shutdown providing high availability capabilities to AdvancedTCA® control and data acquisition instrumentation specially directed for large fusion experiments such as International Thermonuclear Experiment Reactor (ITER).