Hardware Timestamping Research Articles

High-performance wide-band open-source system for acoustic stimulation

The design and characterization of a low-cost, open-source auditory delivery system to deliver high performance auditory stimuli is presented. The system includes a high-fidelity sound card and audio amplifier devices with low-latency and wide bandwidth targeted for behavioral neuroscience research. The characterization of the individual devices and the entire system is performed, providing a thorough audio characterization data for varying frequencies and sound levels. The system implements the open-source Harp protocol, enabling the hardware timestamping of devices and seamless synchronization with other Harp devices.

MUN-FRL: A Visual-Inertial-LiDAR Dataset for Aerial Autonomous Navigation and Mapping

This paper presents a unique outdoor aerial visual-inertial-LiDAR dataset captured using a multi-sensor payload to promote the global navigation satellite system (GNSS)-denied navigation research. The dataset features flight distances ranging from 300 m to 5 km, collected using a DJI-M600 hexacopter drone and the National Research Council (NRC) Bell412 Advanced Systems Research Aircraft (ASRA). The dataset consists of hardware-synchronized monocular images, inertial measurement unit (IMU) measurements, 3D light detection and ranging (LiDAR) point-clouds, and high-precision real-time kinematic (RTK)-GNSS based ground truth. Nine data sequences were collected as robot operating system (ROS) bags over 100 mins of outdoor environment footage ranging from urban areas, highways, airports, hillsides, prairies, and waterfronts. The dataset was collected to facilitate the development of visual-inertial-LiDAR odometry and mapping algorithms, visual-inertial navigation algorithms, object detection, segmentation, and landing zone detection algorithms based on real-world drone and full-scale helicopter data. All the data sequences contain raw sensor measurements, hardware timestamps, and spatio-temporally aligned ground truth. The intrinsic and extrinsic calibrations of the sensors are also provided, along with raw calibration datasets. A performance summary of state-of-the-art methods applied on the data sequences is also provided.

Ethernet-based timing system for accelerator facilities: The IFMIF-DONES case

This article presents the design of a timing system for accelerator facilities, which relies on a general networking approach based on standard Ethernet protocols that keeps all the devices synchronized to a common time reference. The case of the IFMIF-DONES infrastructure is studied in detail, providing a framework for the implementation of the timing system. The network time protocol (NTP) with software timestamping and the precision time protocol (PTP) with hardware timestamping are used to synchronize devices with sub-millisecond and sub-microsecond accuracy requirements, respectively. The design also considers the utilization of IEEE 1588 high accuracy default PTP profile (PTP-HA) to provide sub-nanosecond accuracy for the most demanding components. Three different solutions for the design of the timing system are discussed in detail. The first solution considers the deployment of one time-dedicated network for each synchronization protocol, while the second one proposes the integration of the synchronization data of NTP and PTP into the networks of the facility. The third solution relies on the single distribution of PTP-HA to all the systems. The final design aims to be fully based on standard technologies and to be cost-efficient, seeking for interoperability and scalability, and minimizing the impact on other systems in the facility. An experimental setup has been implemented to evaluate and discuss the suitability of the solutions for the timing system by studying the synchronization accuracy obtained with NTP, PTP and PTP-HA under different network conditions. It includes a timing evaluation platform that tries to resemble the network architecture foreseen in the facility. The measured results revealed that PTP is the most limiting protocol for the second solution. Using the default PTP configuration, it tolerates less than 20% of maximum bandwidth utilization for symmetric bidirectional flows, and around 30% in the case of unidirectional flows (server to client or client to server), with the current setup and using switches without enabled timing support. This case study provides a better understanding of the trade-off between bandwidth utilization, synchronization accuracy and cost in these kinds of facilities.

Development Board Implementation and Chip Design of IEEE 1588 Clock Synchronization System Applied to Computer Networking

With the vigorous development of industrial automation and the Internet of things, the transmission of data is more dependent on immediacy, so network devices have higher and higher requirements for time synchronization accuracy. The clock source of common network devices is provided by the transistor oscillator in the server, but the oscillator will change with factors such as aging and temperature, and it cannot be guaranteed that the oscillator in the server will work at the same frequency. Time synchronization can be achieved by technologies such as IRIG-B, NTP, or IEEE 1588 (PTP), but the hardware cost of building IRIG-B is high, and NTP has the lowest cost, but it can only provide time accuracy from milliseconds to microseconds. PTP can provide sub-microsecond or even nanosecond time precision. It is a system of time synchronization mechanisms through Ethernet transmission. In this article, we first propose a time synchronization system using the development board and PDP protocol. On the Xilinx Zynq-7000 SOC platform of Petalinux, we implement the hardware solution of Linux PTP. The hardware time stamp is 20 ns. To improve the accuracy, the congenital frequency error between the oscillators must be considered. Therefore, a PTP auxiliary time stamp with dynamic frequency compensation is proposed and designed into a chip. Experimental results show that at 45 nm (TN40G) it can operate at 370 MHz and achieve 2.7 ns resolution, which can be applied to more demands.

Implementing Time-triggered Communication for Embedded Ethernet Real-time Network using IEEE 1588 Time Synchronization

A time-triggered embedded system has become a ubiquitous technique in safety-critical systems, such as automotive, aerospace, and medical sectors. Thus, precision is of utmost importance to determine if the application is synchronized correctly. The Network Time Protocol is the traditional protocol used for clock synchronization in wired networks. On the other hand, the protocol lacks hardware timestamping features, and it collects many timestamping errors because of the diversification of the queuing time in the switches and router. This drawback was implemented into a new protocol, and new features on the Precision Time Protocol, known as IEEE 1588-2008, were added. This paper proposes a real-time network for a time-triggered Embedded system, the Precision Time Protocol synchronize, on a Spartan 3E Field Programmable Gate Array (FPGA). Two approaches were also introduced to evaluate the performance using Precision Time Protocol: hardware and software implementations. MATLAB Simulink software was used to evaluate the software approaches. The result showed that the Precision Time Protocol used 0.83μs to complete the clock synchronization compared to the 4.65μs used by the Network Time Protocol, which was six times faster.

HyperTester: High-Performance Network Testing Driven by Programmable Switches

Modern network devices and systems are raising higher requirements on network testers that are regularly used to evaluate performance and assess correctness. These requirements include high scale, high accuracy, flexibility and low cost, which existing testers cannot fulfill at the same time. In this paper, we propose HyperTester, a network tester leveraging new-generation programmable switches and achieving all of the above goals simultaneously. Programmable switches are born with features like high throughput and linerate, deterministic processing pipelines and nanosecond-level hardware timestamps, the P4 programming model as well as comparable pricing with commodity servers, but they come with limited programmability and memory resources. HyperTester uses template-based packet generation to overcome the limitations of the switch ASIC in programmability and designs a stateless connection mechanism as well as counter-based state compression algorithms to overcome the memory resource constraints in the data plane. We have implemented HyperTester on Tofino, and the evaluations on the hardware testbed show that HyperTester supports high-scale packet generation (more than 1.6Tbps) and achieves highly accurate rate control and timestamping. We demonstrate that programmable switches can be potential and attractive targets for realizing network testers.

TS-Perf: General Performance Measurement of Trusted Execution Environment and Rich Execution Environment on Intel SGX, Arm TrustZone, and RISC-V Keystone

A trusted execution environment (TEE) is a new hardware security feature that is isolated from a normal OS (i.e., rich execution environment (REE)). The TEE enables us to run a critical process, but the behavior is invisible from the normal OS, which makes it difficult to debug and tune the performance. In addition, the hardware/software architectures of TEE are different on CPUs. For example, Intel SGX allows user-mode only, although Arm TrustZone and RISC-V Keystone run a trusted OS. In addition, each TEE has each SDK for programming. Each SDK offers own APIs and makes difficult to write a common program. These features make it difficult to compare the performance fairly between TEE and REE on different CPUs. To obtain precise performance and behavior in TEE, we propose TS-perf which is a compiler-based performance measurement method. TS-perf accesses the hardware timestamp counter in TEE as well as REE and keeps a precise log. The codes for measurement are inserted in a TEE binary by the compiler options (i.e., profile option, constructor, and destructor). Furthermore, we utilize the separate compilation technique, and the same benchmark binary is used for a fair comparison between TEE and REE. The architecture of TS-perf is general and implemented for three TEE architectures (Arm TrustZone, Intel SGX, and RISC-V Keystone). TS-perf measures the performance of GlobalPlatform’s TEE internal APIs, matrix multiplication, memory access, and storage access. The comparisons show the difference in performance between TEE and REE and the unusual behavior of trusted applications (TAs).

Subnanosecond Time Synchronization Using a 100Base-TX Ethernet Transceiver and an Optimized PI-Clock Servo

This work presents a real-time 100Base-TX Ethernet physical layer (PHY) transceiver chip implemented in a 180-nm technology. The PHY chip implements highly accurate hardware timestamping by using an 8-bit digital-to-phase converter (DPC) generating 256 phases of the 125 MHz system clock. Using these clock phases, spaced 31.25 ps apart from each other, phase relationships can be evaluated to timestamp ingress and egress frames with improved resolution. Connected by up to 120-m-long category five unshielded twisted-pair cables, two of these PHY chips are used to demonstrate the synchronization in a two-node scenario. A proportional-integral (PI)-clock servo is used at the slave for the synchronization. To optimize the performance, the parameters of the controller need to be chosen carefully. To do so, a simple model is used to find a suitable bandwidth of the controller as a tradeoff between the noise of the timestamping and the noise of the oscillator. All in all, a synchronization accuracy with a standard deviation of only 64 ps and a mean offset of well below 100 ps is achieved in the given scenario. To the best of our knowledge, this is the highest synchronization accuracy over copper-based Ethernet reported to date.

How to Measure the Killer Microsecond

Datacenter-networking research requires tools to both generate traffic and accurately measure latency and throughput. While hardware-based tools have long existed commercially, they are primarily used to validate ASICs and lack flexibility, e.g. to study new protocols. They are also too expensive for academics. The recent development of kernel-bypass networking and advanced NIC features such as hardware timestamping have created new opportunities for accurate latency measurements. This paper compares these two approaches, and in particular whether commodity servers and NICs, when properly configured, can measure the latency distributions as precisely as specialized hardware. Our work shows that well-designed commodity solutions can capture subtle differences in the tail latency of stateless UDP traffic. We use hardware devices as the ground truth, both to measure latency and to forward traffic. We compare the ground truth with observations that combine five latency-measuring clients and five different port forwarding solutions and configurations. State-of-the-art software such as MoonGen that uses NIC hardware timestamping provides sufficient visibility into tail latencies to study the effect of subtle operating system configuration changes. We also observe that the kernel-bypass-based TRex software, that only relies on the CPU to timestamp traffic, can also provide solid results when NIC timestamps are not available for a particular protocol or device.

Hardware Timestamping for an Image Acquisition System Based on FlexRIO and IEEE 1588 v2 Standard

Current fusion devices usually implement distributed acquisition systems for the multiple diagnostics of their experiments. However, each diagnostic is composed by hundreds or even thousands of signals, including images from the vessel interior. These signals and images must be correctly timestamped, because all the information will be analyzed to identify plasma behavior using temporal correlations. For acquisition devices without synchronization mechanisms the timestamp is given by another device with timing capabilities when signaled by the first device. Later, each data should be related with its timestamp, usually via software. This critical action is unfeasible for software applications when sampling rates are high. In order to solve this problem this paper presents the implementation of an image acquisition system with real-time hardware timestamping mechanism. This is synchronized with a master clock using the IEEE 1588 v2 Precision Time Protocol (PTP). Synchronization, image acquisition and processing, and timestamping mechanisms are implemented using Field Programmable Gate Array (FPGA) and a timing card -PTP v2 synchronized. The system has been validated using a camera simulator streaming videos from fusion databases. The developed architecture is fully compatible with ITER Fast Controllers and has been integrated with EPICS to control and monitor the whole system.

TS2: a realistic IEEE1588 time-synchronization simulator for mobile wireless sensor networks

This paper presents the development of a simulator, TS2 (time-synchronization simulator), for realistically simulating and studying the performance of the IEEE 1588 precise time protocol (PTP) in IEEE 802.15.4 (TI CC2420 chip)-based mobile wireless sensor networks (WSNs). The PTP has the advantage of achieving high time precision at a low cost. It was designed for a wired Ethernet with requirements of symmetric communication paths and accurate time stamping, which is difficult for the low-cost WSNs to meet, as WSNs suffer more from overwhelming transmission delay jitters. An analytic solution to the performance of the PTP in WSNs is not possible, and it is of importance to evaluate the performance by realistic simulation. Based on the open-source OMNeT++ simulation engine, a realistic simulator was developed to simulate the PTP in IEEE 802.15.4 networks. There are two main contributions and benefits of the developed simulator. (1) Reality and fidelity: the WSN node’s various layers (including radio channels and TI CC2420 chip), drifting oscillator clocks and PTPs are simulated realistically. The drifting clock is simulated at an adjustable and higher resolution. (2) Support to both hardware and software time stamping, and the time-stamping uncertainties, by using a separated modular time-stamping module. It also has other features (such as extendibility and code-reusability, mobile WSN nodes, scalability for multi-node, multi-hop simulation). Finally, to demonstrate the simulator’s application to evaluating a PTP-based clock correction algorithm, a direct servo clock adjustment algorithm (i.e., a P controller) for a TI CC2420-based WSN was simulated and its performance was analyzed.

Asymmetry Mitigation in IEEE 802.3 Ethernet for High-Accuracy Clock Synchronization

Clock synchronization is one of the basic services in a distributed network as it enables a palette of services, such as synchronized measurements and actions, or time-based access to shared communication media. The IEEE 1588 standard defines the precision time protocol (PTP) that is capable of synchronizing multiple slave clocks to a master by means of synchronization event messages. Supported by the recent advances in hardware timestamping, PTP devices are ready for achieving synchronization accuracies in the subnanosecond range. The accuracy of practical synchronization systems is, however, often bounded by the inability to determine and compensate for asymmetric line delays leading to unresolvable clock offsets. Although IEEE 1588 version 2008 is able to compensate for known asymmetry, no specific measures to estimate the asymmetry are defined in the standard. In this paper, we present a solution to determine the asymmetry for 100Base-TX networks based on line swapping and highly accurate timestamping. When the presented approach is used within the startup procedure of an Ethernet link, the synchronization offsets can be minimized while the operation of the network is not impaired. We show by an FPGA-based prototype system that our approach is able to reduce the clock offset from multiple nanoseconds to below 120 ps.

A packet I/O architecture for shell script-based packet processing

We propose a new scripting model for rapid and easier development of packet processing using shell scripts. In this paper we present EtherPIPE, a character network I/O device, that allows the programmer to access network traffic data as a file through UNIX commands. By setting a UNIX pipe ?|? from or to EtherPIPE's output or input with UNIX commands, packets can be easily processed, executing functions such as packet filtering, packet capturing, generating arbitrary packets, and rewriting header information. In order to prove the utilities of our model, we have developed FPGA-based EtherPIPE adapter using a commodity FPGA card and a character device driver featuring new offloading functions. With our prototype implementation, packet scripting works at 1Gbps wire-speed, receiving packets with precise hardware timestamps. This paper argues for use cases of the EtherPIPE, and discusses enhanced formats of character devices for easier network scripting.)

The Research of IEEE 1588 Interface Module Based on LXI Instrument

LXI, which is generally recognized as the new generation of instrument bus technology, not only has developed rapidly in the field of test and measurement, but also has a vast potential for future development. At present, the LXI trigger requires the clock synchronization precision to reach a sub microsecond level or even higher one. Therefore, in order to meet this accuracy requirement, a scheme based on the embedded Linux and IEEE 1588 protocol is proposed in this paper on the basis of studying LXI so as to realize high accurate clock synchronization. This scheme acquires hardware timestamp by utilizing PHY chip DP83640 and realizes PTP hardware clock driving based on Linux. A synchronization test with Agilent B LXI trigger box under the current condition is conducted and about 100ns of synchronization precision can be reached when connected with cable directly. The result demonstrates the feasibility of the scheme and that the clock synchronization precision can reach a nanosecond level.

A Hardware Time Stamping Method for PTP Messages Based on Linux system

In order to achieve high-precision clock synchronization among instrument nodes in distributed LXI bus test systems. A Hardware time stamping method for PTP messages is introduced in this paper. It is implemented in Linux systems, which runs on ARM9 S3C2440 processor. The special PHYTER DP83640 is used to mark the time of PTP messages in physical layer. The new socket option SO_TIMESTAMPING is used to obtain the hardware timestamps in user space programming. A clock synchronization test has been done after transplanting the open source protocol software PTPd on this platform. The results indicate that this method is feasible. It is able to achieve nanosecond level clock synchronization accuracy. DOI: http://dx.doi.org/10.11591/telkomnika.v11i9.3255

Wait-n-GoTM

Transactional memory (TM) has been proposed to alleviate some key programmability problems in chip multiprocessors. Most TMs optimistically allow concurrent transactions, detecting read-write or write-write conflicts. Upon conflicts, existing hardware TMs (HTMs) use one of three conflict-resolution policies: (1) always-abort, (2) always-wait for some conflicting transactions to complete, or (3) always-go past conflicts and resolve acyclic conflicts at commit or abort upon cyclic dependencies. While each policy has advantages, the policies degrade performance under contention by limiting concurrency (always-abort, always-wait) or incurring late aborts due to cyclic dependencies (always-go). Thus, while always-go avoids acyclic aborts, no policy avoids cyclic aborts. We propose Wait-n-GoTM (WnGTM) to increase concurrency while avoiding cyclic aborts. We observe that most cyclic dependencies are caused by threads interleaving multiple accesses to a few heavily-read-write-shared delinquent data cache blocks. These accesses occur in code sections called cycle inducer sections (CISTs) . Accordingly, we propose Wait-n-Go (WnG) conflict-resolution to avoid many cyclic aborts by predicting and serializing the CISTs. To support the WnG policy, we extend previous HTMs to (1) allow multiple readers and writers, (2) scalably identify dependencies, and (3) detect cyclic dependencies via new mechanisms, namely, conflict transactional state, order-capture, and hardware timestamps, respectively. In 16-core simulations of STAMP, WnGTM achieves average speedups of 46% for higher-contention benchmarks and 28% for all benchmarks over always-abort (TokenTM) with low-contention benchmarks remaining unchanged, compared to always-go (DATM) and always-wait (LogTM-SE), which perform worse than and 6% better than TokenTM, respectively.

Wait-n-GoTM

Transactional memory (TM) has been proposed to alleviate some key programmability problems in chip multiprocessors. Most TMs optimistically allow concurrent transactions, detecting read-write or write-write conflicts. Upon conflicts, existing hardware TMs (HTMs) use one of three conflict-resolution policies: (1) always-abort, (2) always-wait for some conflicting transactions to complete, or (3) always-go past conflicts and resolve acyclic conflicts at commit or abort upon cyclic dependencies. While each policy has advantages, the policies degrade performance under contention by limiting concurrency (always-abort, always-wait) or incurring late aborts due to cyclic dependencies (always-go). Thus, while always-go avoids acyclic aborts, no policy avoids cyclic aborts. We propose Wait-n-GoTM (WnGTM) to increase concurrency while avoiding cyclic aborts. We observe that most cyclic dependencies are caused by threads interleaving multiple accesses to a few heavily-read-write-shared delinquent data cache blocks. These accesses occur in code sections called cycle inducer sections (CISTs) . Accordingly, we propose Wait-n-Go (WnG) conflict-resolution to avoid many cyclic aborts by predicting and serializing the CISTs. To support the WnG policy, we extend previous HTMs to (1) allow multiple readers and writers, (2) scalably identify dependencies, and (3) detect cyclic dependencies via new mechanisms, namely, conflict transactional state, order-capture, and hardware timestamps, respectively. In 16-core simulations of STAMP, WnGTM achieves average speedups of 46% for higher-contention benchmarks and 28% for all benchmarks over always-abort (TokenTM) with low-contention benchmarks remaining unchanged, compared to always-go (DATM) and always-wait (LogTM-SE), which perform worse than and 6% better than TokenTM, respectively.

Research on Synchronization Technology Based on IEEE1588 Protocol

With the rapid development of network technology, Distributed test system is widely applied in the test field, and clock synchronization in the system is the key issues in distributed test system. Meanwhile, with the development of testing techniques, distributed test has higher requirement for precision of the synchronization. Since the IEEE1588 protocol promulgated, it provides a feasible way to realize the requirement. This paper firstly analyzes the basic principle of IEEE1588, and then puts forward a scheme to implement IEEE1588 by using hardware timestamp. Finally, optimizes the method of PTP (precision time protocol) synchronization adjustment and builds experimental platform to verify the effect of optimization. Experiments prove that the scheme is feasible, and the clock synchronization precision can achieve a level of sub-microsecond.

PIXHAWK: A micro aerial vehicle design for autonomous flight using onboard computer vision

We describe a novel quadrotor Micro Air Vehicle (MAV) system that is designed to use computer vision algorithms within the flight control loop. The main contribution is a MAV system that is able to run both the vision-based flight control and stereo-vision-based obstacle detection parallelly on an embedded computer onboard the MAV. The system design features the integration of a powerful onboard computer and the synchronization of IMU-Vision measurements by hardware timestamping which allows tight integration of IMU measurements into the computer vision pipeline. We evaluate the accuracy of marker-based visual pose estimation for flight control and demonstrate marker-based autonomous flight including obstacle detection using stereo vision. We also show the benefits of our IMU-Vision synchronization for egomotion estimation in additional experiments where we use the synchronized measurements for pose estimation using the 2pt+gravity formulation of the PnP problem.