Frame Drops Research Articles

Robust marker-based projector–camera synchronization

Recording clean pictures of projected images requires the projector and camera to be synchronized. This task usually requires additional hardware or imposes major restrictions on the devices with software-based approaches, e.g., a specific frame rate of the camera. We present a novel software-based synchronization technique that supports projectors and cameras with different frame rates and at the same time tolerates camera frame drops. We focus on the special needs of LCD projectors and the effect of their liquid crystal response time on the projected image. By relying on visible marker detection we entirely refrain from taking time measurements, allowing for a robust and fast synchronization.

Synchronous Remote Rendering for VR

Remote rendering for VR is a technology that enables high-quality VR on low-powered devices. This is realized by offloading heavy computation and rendering to high-powered servers that stream VR as video to the clients. This article focuses on one specific issue in remote rendering when imperfect frame timing between client and server may cause recurring frame drops. We propose a system design that executes synchronously and eliminates the aforementioned problem. The design is presented, and an implementation is tested using various networks and hardware. The design cannot drop frames due to synchronization issues but may on the other hand stall if temporal disturbances occur, e.g., due to network delay spikes or loss. However, experiments confirm that such events can remain rare given an appropriate environment. For example, remote rendering on an intranet at 90 fps with a server located approximately 50 km away yielded just 0.002% stalled frames while rendering with extra latency corresponding to the duration of exactly one frame (11.1 ms at 90 fps). In a LAN without extra latency setting, i.e., with latency equal to locally rendered VR, 0.009% stalls were observed while using a wired Ethernet connection and 0.058% stalls when using 5 GHz wireless IEEE 802.11 ac.

IoTorch: Reliable LED-to-Camera Communication Against Inter-Frame Gaps and Frame Drops

LED-to-camera communication for smartphones promises to enable low-cost, small-size, and intuitive data access to Internet-of-things (IoT) devices. However, for fast and reliable communication under the rolling shutter effect, the packet losses caused by inter-frame gaps and frame drops must be dealt with. In this paper, we introduce LED-to-camera communication between a commercial smartphone and an IoT device with LED flashlight, where the user can intuitively acquire the desired data transmitted from the IoT device to the smartphone. Specifically, we present IoTorch, a fast and reliable LED-to-camera communication that efficiently prevents the packet losses. IoTorch consists of two core mechanisms: 1) a minimum-repetition one-way reliable transmission focusing on the periodicity of inter-frame gaps and 2) an acknowledgement mechanism to overcome frame drops by using a smartphone's built-in flash focusing on its delay characteristics. Additionally, we propose an optimization method to increase the throughput up to 2.92 kbps (i.e., 2.43 times faster than that of the state-of-the-art method that overcomes the inter-frame gaps). Furthermore, we realize a remote wake-up function by using the smartphone's flash as the data transmission trigger. To demonstrate the benefit of IoTorch, we present a sensor data viewer using Arduino and a smartphone console system on TelosB.

Behavior of traditional Chinese mortise-tenon joints: Experimental and numerical insight for coupled vertical and reversed cyclic horizontal loads

Mortise-tenon joints in traditional Chinese architecture are of major importance. Studying their performance under horizontal loads loads is paramount for China, where there are more than ten thousand historical pavilions and buildings –some of them 1000 years old-built with such a technology and where seismicity is high. To probe the impact of damage to mortise-tenon joints on the seismic performance of unidirectional straight-tenon wooden frames, this paper uses the so-called artificial damage simulation method. Under the condition of considering damage depth and degree of surface damage of the tenon, one intact wooden frame and 3 mortise-tenon wooden frames (having joints with different decay degrees of damage) are built in the laboratory using a 1:3.52 scale ratio. To obtain failure modes and hysteresis performance of the specimens, low-cycle reciprocating loading tests were conducted. This paper compares and analyzes the characteristics of edge point strain change of tenon and socket joints, the variation law of seismic performance such as hysteresis characteristics of the wood frame, the stiffness degradation and energy dissipation ability, and the accumulation of damage during repeated loading of the wood frame. Results show that, with an increase of the degree of decadent damage, the peak of the hysteresis curve of the wood frame decreases gradually, the tendency of the skeleton curve is flattened, the equivalent viscous damping coefficient decreases, and the seismic behavior of the wood frame drops. A sophisticated FE approach with contact-frictional interfaces at the mortise-tenon joints and with beams exhibiting a non-linear behavior is adopted to fit data obtained experimentally in the lab. A satisfactory agreement is found between numerical simulations and experimental results, with the comparison giving interesting insight into the expected behavior of such connections in case of an earthquake.

Delivering High-Frame-Rate Video to Mobile Devices in Heterogeneous Wireless Networks

High-frame rate (HFR) video streaming is emerging as a new paradigm in popular multimedia applications (e.g., mobile cloud gaming) to achieve smooth viewing experience perceived by end users. However, it is severely challenging to guarantee the delivery quality of HFR video over wireless platforms with regard to the high transmission rate and limited network resources. Multihoming capability enables mobile devices to concurrently receive video data with different radio interfaces (e.g., cellular and Wi-Fi). To effectively deliver mobile HFR video over multiple wireless access networks, this paper develops an application-layer transmission scheme dubbed joint FRAme Scheduling and Error Resilience (FRASER). First, we propose an unequal frame scheduling approach by taking advantage of interlaced forward error correction coding and reliability-aware data allocation to minimize the total distortion. Second, we introduce an error resilience scheme at the receiver side to proactively leverage the out-of-order and overdue video packets to mitigate the error propagations. The proposed FRASER is able to substantially reduce the probability of I (Intra) frame loss and consecutive P (Predicted) frame drops. We conduct the performance evaluation through extensive emulations in Exata involving HFR video streaming encoded with H.264 codec. Evaluation results show that FRASER outperforms the reference transmission schemes in terms of video peak signal-to-noise ratio, end-to-end delay, and received frame rate. Thus, FRASER is recommended for delivering HFR video streaming to multihomed mobile devices in heterogeneous wireless networks.

GemDroid

As the demand for feature-rich mobile systems such as smartphones and tablets has outpaced other computing systems and is expected to continue at a faster rate, it is projected that SoCs with tens of cores and hundreds of IPs (or accelerator) will be designed to provide unprecedented level of features and functionality in future. Design of such mobile systems with required QoS and power budgets along with other design constraints will be a daunting task for computer architects since any ad hoc, piece-meal solution is unlikely to result in an optimal design. This requires early exploration of the complete design space to understand the system-level design trade-offs. To the best of our knowledge, there is no such publicly available tool to conduct a holistic evaluation of mobile platforms consisting of cores, IPs and system software. This paper presents GemDroid, a comprehensive simulation infrastructure to address these concerns. GemDroid has been designed by integrating the Android open-source emulator for facilitating execution of mobile applications, the GEM5 core simulator for analyzing the CPU and memory centric designs, and models for several IPs to collectively study their impact on system-level performance and power. Analyzing a spectrum of applications with GemDroid, we observed that the memory subsystem is a vital cog in the mobile platform because, it needs to handle both core and IP traffic, which have very different characteristics. Consequently, we present a heterogeneous memory controller (HMC) design, where we divide the memory physically into two address regions, where the first region with one memory controller (MC) handles core-specific application data and the second region with another MC handles all IP related data. The proposed modifications to the memory controller design results in an average 25% reduction in execution time for CPU bound applications, up to 11% reduction in frame drops, and on average 17% reduction in CPU busy time for on-screen (IP bound) applications.

An Evaluation of Motion JPEG 2000 for Video Archiving

Motion JPEG 2000 (MJ2) is one potential format for long-term video preservation. The format is attractive as an open standard with a truly lossless compression mode.Currently, three software-only MJ2 implementations are readily available, from the Open JPEG 2000 project, from the Kakadu project, and (incorporating Kakadu) from vendor Morgan Multimedia. These are given a snapshot evaluation here. Among the findings: on a modern desktop machine, the Kakadu-based implementations can decode and deliver quarter-screen or smaller lossless-MJ2-encoded videos without frame drops. The newer Open JPEG 2000, while improving, is not yet competitive. All the implementations have practical limitations on acceptable input formats, and inadequate or missing audio support.At higher image resolutions, playback without frame drops or reversion to lossy mode currently suggests hardware-based implementations. A practical impediment is limited availability of off-the-shelf board-level products.Competing candidate file formats for video-editing, archiving, and delivery currently offer better-defined storage of metadata. Some formats, such as MPEG4/AVC, achieve better compression at the expense of some lossiness.

Video quality and traffic QoS in learning-based subsampled and receiver-interpolated video sequences

Sources of real-time traffic are generally highly unpredictable with respect to the instantaneous and average load which they create. Yet such sources will provide a significant portion of traffic in future networks, and will significantly affect the overall performance of and quality of service. Clearly high levels of compression are desirable as long as video quality remains satisfactory, and our research addresses this key issue with a novel learning-based approach. We propose the use of neural networks (NNs) as post-processors for any existing video compression scheme. The approach is to interpolate video sequences and compensate for frames which may have been lost or deliberately dropped. We show that deliberately dropping frames will significantly reduce the amount of offered traffic in the network, and hence the cell loss probability and network congestion, while the NN post-processor will preserve most of the desired video quality. Dropping frames at the sender or in the network is also a fast way to react to network overload and reduce congestion. Our interpolation techniques at the receiver, including neural network-based algorithms, provide output frame rates which are identical to (or possibly higher than) the original video sequence's frame rate. The resulting video quality is essentially equivalent to the sequence without frame drops, despite the loss of a significant fraction of the frames. Experimental evaluation using real video sequences is provided or interpolation with a connectionist NN using the backpropagation learning algorithm, the random NN (RNN) in a feed-forward configuration with its associated learning algorithm, and cubic spline interpolation. The experiments show that when more frames are being dropped or lost, the RNN performs generally better than the other techniques in terms of resulting video quality and overall performance. When the fraction of dropped frames is small, cubic splines offer better performance. Experimental data shows that this receiver-reconstructed subsampling technique significantly reduces the cell loss rates in an asynchronous transfer mode switch for different buffer sizes and service rates.