Interactive Video Applications Research Articles

A Tutorial on Immersive Video Delivery: From Omnidirectional Video to Holography

Video services are evolving from traditional two-dimensional video to virtual reality and holograms, which offer six degrees of freedom to users, enabling them to freely move around in a scene and change focus as desired. However, this increase in freedom translates into stringent requirements in terms of ultra-high bandwidth (in the order of Gigabits per second) and minimal latency (in the order of milliseconds). To realize such immersive services, the network transport, as well as the video representation and encoding, have to be fundamentally enhanced. The purpose of this tutorial article is to provide an elaborate introduction to the creation, streaming, and evaluation of immersive video. Moreover, it aims to provide lessons learned and to point at promising research paths to enable truly interactive immersive video applications toward holography.

Effects of Learning Using EDPUZZLE Interactive Video Application on Students' Interest, Engagement and Achievement in Science Subjects

This study was conducted to identify the effect of learning methods using the Edpuzzle interactive video application on students' interest, engagement, and achievement in science subjects. The purposive sampling technique was used in this study to select the sample. A total of 40 students were selected as a sample in this study. The selected sample is students from a primary school in the Batu Pahat Johor district. The sample was selected from two classes of year 2 students who had similar ICT backgrounds in primary school. Pre-tests, post-tests, checklists, and questionnaires were used as instruments in this study. This study's objectives are to identify the effectiveness of the use of the interactive video application, Edpuzzle in increasing students' interest, engagement, and achievement in science subjects. The data were analysed through inferential statistics using Statistical Package for Social Science (SPSS) software. The data analysis results show significant differences in the level of interest, engagement, and achievement of primary school students in science subjects when using the EdPuzzle application. This suggests that the use of interactive video in the classroom has a promising approach to improve the quality of teaching and learning (TnL) among science teachers. However, considering the limitations of the research, specifically in relation to the context and the period of application, more studies are needed with different methodological approaches and in extended periods and samples.

Ultra-Low Latency, Stable, and Scalable Video Transmission for Free-Viewpoint Video Services

With IT and video technologies evolving rapidly, video services with user interaction emerge, and bring about attracting experiences. The growing bandwidth and latency demands call for advanced transmission techniques which utilize wired, wireless, and 5G cellular network, and cooperate unicast, multicast, and broadcast. Free-viewpoint video, as one of the interactive video applications, can provide users with immersion of real world scenes, and shows potential to be an essential application in future digital life. However, such an application is different from other video services in aspects of computation and communication features, and existing researches are not up to the low frame latency, smooth playback, and high service scalability challenges. In this paper, we target on the above problems, and achieve ultra-low latency and stable video transmission for free-viewpoint video services by improving the performance of multiple system components accordingly. Firstly, the key performance indicator is formulated for such services, and an edge computing based video transmission schema is proposed to achieve ultra-low frame latency. Secondly, a pair of stabilization algorithms at both server and client sides based on application features are introduced to ensure stable encoded video streams before transmission, and smooth playback after that. Thirdly, a cloud-edge-client hybrid free-viewpoint service architecture is proposed, which can provide hierarchical experiences and enlarge the service scale with restricted resources. Experimental results prove the effectiveness of the proposed transmission schema and scheduling algorithms, and feasibility of the proposed service architecture is revealed by qualitative analyses.

An Elastic System Architecture for Edge Based Low Latency Interactive Video Applications

5G and edge computing have brought great changes to video industry. Interactive video is becoming an emerging application form of multimedia service, which provides attraction beyond typical scenarios like cloud gaming and remote virtual reality (VR), and puts forward great challenges in resource capacity, response latency, and function flexibility to its service system. In this paper, we propose an elastic system architecture with low latency features to accommodate generic interactive video applications on near user edges. To increase system flexibility, we firstly design a dynamic Directed Acyclic Graph (dDAG) model for efficient task representation. Secondly, based on the model, we present the elastic architecture together with its scalable workflow pipeline. Thirdly, we propose a set of novel latency measurement metrics to analyze and optimize the performance of an interactive video system. Based on the proposed approaches, we disassemble a real world free-viewpoint synthesis application and benchmark its performance with the metrics. Extensive experimental results show the flexibility of our system to handle the stochastic human interactions during a video service session, with less than 5 ms additional scheduling latency introduced. End to end latency is kept within 43 ms for complex functions, and 28 ms for simpler scenarios, which satisfies the restrictions of most interactive video applications provided by an edge. Client of the architecture serves as a pure video player, which is also friendly to power limited terminals such as 5G phones. Efficiency and stability analyses of the system show superiorities over existing work, and also reveal potential optimization directions for future research.

Statistical Learning Based Congestion Control for Real-Time Video Communication

With the increasing demands on interactive video applications, how to adapt video bit rate to avoid network congestion has become critical, since congestion results in self-inflicted delay and packet loss which deteriorate the quality of real-time video service. The existing congestion control is hard to simultaneously achieve low latency, high throughput, good adaptability and fair bandwidth allocation, mainly because of the hardwired control strategy and egocentric convergence objective. To address these issues, we propose an end-to-end statistical learning based congestion control, named Iris. By exploring the underlying principles of self-inflicted delay, we reveal that congestion delay is determined by sending rate, receiving rate and network status, which inspires us to control video bit rate using a statistical-learning congestion control model. The key idea of Iris is to force all flows to converge to the same queue load, and adjust the bit rate by the model. All flows keep a small and fixed number of packets queuing in the network, thus the fair bandwidth allocation and low latency are both achieved. Besides, the adjustment step size of sending rate is updated by online learning, to better adapt to dynamically changing networks. We carried out extensive experiments to evaluate the performance of Iris, with the implementations of transport layer (UDP) and application layer (QUIC) respectively. The testing environment includes emulated network, real-world Internet and commercial LTE networks. Compared against TCP flavors and state-of-the-art protocols, Iris is able to achieve high bandwidth utilization, low latency and good fairness concurrently. Especially over QUIC, Iris is able to increase the video bitrate up to 25%, and PSNR up to 1dB.

Robotic Glove for Rehabilitation Purpose: Review

Rehabilitation robots have become one of the main technical instruments that Treat disorder patients in the biomedical engineering field. The robotic glove for the rehabilitation is basically made of specialized materials which can be designed to help the post-stroke patients. In this paper, a review of the different types of robotic glove for Rehabilitation have been discussed and summarized. This study reviews a different mechanical system of robotic gloves in previous years. The selected studies have been classified into four types according to the Mechanical Design: The first type is a tendon-driven robotic glove. The second type of robotic glove works with a soft actuator as a pneumatic which is operated by air pressure that passes through a plastic pipe, pressure valves, and air compressor. The third type is the exoskeleton robotic gloves this type consists of a wearable mechanical design that can used a finger-based sensor to measure grip strength or is used in interactive video applications. And the fourth type is the robotic glove with a liner actuator this type consists of a tape placed on the fingers and connected to linear actuators to open and close the fingers during the rehabilitation process.

Responsive high throughput congestion control for interactive applications over SDN-enabled networks

New interactive video applications are increasingly emerging over the Internet; these interactive applications are characterized by high bandwidth requirements that fluctuates depending on end-user actions (e.g. less bandwidth is usually needed for stationary scenes). More importantly, this interactive class of services also involves a requirement for high responsiveness (i.e. low latency) from the network, in order to respond in real-time to end-user actions. One emerging service of this nature is 360° video streaming; another example is cloud-based gaming services. In this paper, we focus specifically on JPIP (JPEG 2000 Interactive Protocol) applications that support remote interactive video browsing with dynamic pan and zoom capabilities, as a highly representative example of the interactive service class. Existing network communication services are mostly agnostic to latency implications, and hence are not well adapted to such interactive applications. Meanwhile, explicit resource reservation protocols have not been widely deployed, and do not consider the time-varying dependencies that naturally arise in interactive applications. In this work, we leverage software defined networking (SDN) principles to support a proposed “interactive service” class. The main contributions of this work are a network-exposed application programming interface (API) that provides visibility into the state of the network, an SDN-assisted congestion control algorithm that utilizes network state information to achieve the desired low latency and high bandwidth utilization requirements, and a fair resource assignment algorithm that shares available bandwidth among interactive and non-interactive traffic dynamically – all without a reservation protocol.

First We Feel Then We Fall: James Joyce’s Finnegans Wake as an interactive video application

First We Feel Then We Fall: James Joyce’s Finnegans Wake as an interactive video application

QoS, Energy and Cost Efficient Resource Allocation for Cloud-Based Interactive TV Applications

Internet-based social and interactive video applications have become major constituents of the envisaged applications for next-generation multimedia networks. However, inherently dynamic network conditions, together with varying user expectations, pose many challenges for resource allocation mechanisms for such applications. Yet, in addition to addressing these challenges, service providers must also consider how to mitigate their operational costs (e.g., energy costs, equipment costs) while satisfying the end-user quality of service (QoS) expectations. This paper proposes a heuristic solution to the problem, where the energy incurred by the applications, and the monetary costs associated with the service infrastructure, are minimized while simultaneously maximizing the average end-user QoS. We evaluate the performance of the proposed solution in terms of serving probability, i.e., the likelihood of being able to allocate resources to groups of users, the computation time of the resource allocation process, and the adaptability and sensitivity to dynamic network conditions. The proposed method demonstrates improvements in serving probability of up to 27%, in comparison with greedy resource allocation schemes, and a several-orders-of-magnitude reduction in computation time, compared to the linear programming approach, which significantly reduces the service-interrupted user percentage when operating under variable network conditions.

24-1 自由視聴点技術に基づくスマートフォン向けミュージックビデオアプリケーション(第24部門デモセッション)

24-1 自由視聴点技術に基づくスマートフォン向けミュージックビデオアプリケーション(第24部門デモセッション)

Large-scale MIMO is capable of eliminating power-thirsty channel coding for wireless transmission of HEVC/H.265 video

A wireless video transmission architecture relying on the emerging large-scale multiple-input--multiple-output (LS-MIMO) technique is proposed. Upon using the most advanced High Efficiency Video Coding (HEVC) (also known as H.265), we demonstrate that the proposed architecture invoking the low-complexity linear zero-forcing (ZF) detector and dispensing with any channel coding is capable of significantly outperforming the conventional small-scale MIMO based architecture, even if the latter employs the high-complexity optimal maximum-likelihood (ML) detector and a rate-$1/3$ recursive systematic convolutional (RSC) channel codec. Specifically, compared to the conventional small-scale MIMO system, the effective system throughput of the proposed LS-MIMO based scheme is increased by a factor of up to three and the quality of reconstructed video quantified in terms of the peak signal-to-noise ratio (PSNR) is improved by about $22.5\, \text{dB}$ at a channel-SNR of $E_b/N_0 \approx 6\,\text{dB}$ for delay-tolerant video-file delivery applications, and about $20\,\text{dB}$ for lip-synchronized real-time interactive video applications. Alternatively, viewing the attainable improvement from a power-saving perspective, a channel-SNR gain as high as $\Delta_{E_b/N_0}\approx 5\,\text{dB}$ is observed at a PSNR of $36\, \text{dB}$ for the scenario of delay-tolerant video applications and again, an even higher gain is achieved in the real-time video application scenario. Therefore, we envisage that LS-MIMO aided wireless multimedia communications is capable of dispensing with power-thirsty channel codec altogether!

DIY multimedia: A review of a step-by-step handbook

In "Develop Your Own Multimedia Application: How to Create Interactive Video Applications in Flash Format" Alexis Aronis demonstrates just how simple it is to create a multimedia application from scratch these days. Readers learn how to create unique video projects using Camtasia Studio and Power Point.

Video over Software-Defined Networking (VSDN)

Supporting end-to-end quality of service (QoS) for real-time interactive video applications, such as video conferencing and distance learning, requires the network to select an optimum path from among multiple paths. Multiple network paths may be available between the source and the destination, but finding an alternative path is difficult because of the high coupling and complex design of the network. Network architectures such as Integrated Services (IntServ) install the path selected by the routing protocol, which may not deliver best-case performance. Furthermore, it is an arduous task for the video application developer to request service from IntServ.We provide three contributions to the literature on providing end-to-end QoS for real-time interactive video applications. First, we present Video over Software-Defined Networking (VSDN), a network architecture that selects the optimum path using a network-wide view. Second, we describe the VSDN protocol that is used by the developer to request QoS from the network. Finally, we present the results of implementing a VSDN prototype and evaluate the behavior of VSDN using message complexity, network throughput, network delay, and network jitter. The message complexity of VSDN is linear.

Wireless Multicast for Zoomable Video Streaming

Zoomable video streaming refers to a new class of interactive video applications, where users can zoom into a video stream to view a selected region of interest in higher resolutions and pan around to move the region of interest. The zoom and pan effects are typically achieved by breaking the source video into a grid of independently decodable tiles. Streaming the tiles to a set of heterogeneous users using broadcast is challenging, as users have different link rates and different regions of interest at different resolution levels. In this article, we consider the following problem: Given the subset of tiles that each user requested, the link rate of each user, and the available time slots, at which resolution should each tile be sent, to maximize the overall video quality received by all users. We design an efficient algorithm to solve this problem and evaluate the solution on a testbed using 10 mobile devices. Our method is able to achieve up to 12dB improvements over other heuristic methods.

QoE prediction model for mobile video telephony

Interactive online video applications, such as video telephony, are known for their vulnerability to network condition. With the increasing usage of hand-held wireless mobile devices, which are capable of capturing and processing good quality videos, combined with the flexibility in an end-user movements have added new challenging factors for application providers and network operators. These factors affect the perceived video quality of mobile video telephony applications, unlike conventional video telephony over desktop computers. We investigate this impact on video quality of mobile video telephony in varying network conditions and end-users movement scenarios. Based on 312 live traces, we quantitatively derive the correlation between the perceived video quality and the network Quality of Service (QoS) and user mobility. With the results, we develop a Quality of Experience (QoE) prediction model for mobile video telephony using Support Vector Regression techniques. The prediction models display ? 0.8 pearson correlation with experimental data. Our methodology and findings can be used to guide the video telephony application providers and network operators to work towards satisfying end-user experience.

ALP

There has been an increasing demand for interactive video transmission over the Internet for applications such as video conferencing, video calls, and telepresence applications. These applications are increasingly moving towards providing High Definition (HD) video quality to users. A key challenge in these applications is to preserve the quality of video when it is transported over best-effort networks that do not guarantee lossless transport of video packets. In such conditions, it is important to protect the transmitted video by using intelligent and adaptive protection schemes. Applications such as HD video conferencing require live interaction among participants, which limits the overall delay the system can tolerate. Therefore, the protection scheme should add little or no extra delay to video transport. We propose a novel Adaptive Loss Protection (ALP) scheme for interactive HD video applications such as video conferencing and video chats. This scheme adds negligible delay to the transmission process and is shown to achieve better quality than other schemes in lossy networks. The proposed ALP scheme adaptively applies four different protection modes to cope with the dynamic network conditions, which results in high video quality in all network conditions. Our ALP scheme consists of four protection modes ; each of these modes utilizes a protection method . Two of the modes rely on the state-of-the-art protection methods, and we propose a new Integrated Loss Protection (ILP) method for the other two modes. In the ILP method we integrate three factors for distributing the protection among packets. These three factors are error propagation, region of interest and header information. In order to decide when to switch between the protection modes, a new metric is proposed based on the effectiveness of each mode in performing protection, rather than just considering network statistics such as packet loss rate. Results show that by using this metric not only the overall quality will be improved but also the variance of quality will decrease. One of the main advantages of the proposed ALP scheme is that it does not increase the bit rate overhead in poor network conditions. Our results show a significant gain in video quality, up to 3dB PSNR improvement is achieved using our scheme, compared to protecting all packets equally with the same amount of overhead.

Mobile video chat: issues and challenges

The next generation of cellular communication needs to endure a higher demand for enhanced real-time multimedia support to end users. Among interactive video applications, use of mobile video chat is on the rise in both the enterprise and consumer worlds. Mobile video chats are resource-constrained and heterogenous with varying display sizes, processing powers, network conditions, and battery levels, and pose real-time delivery constraints unlike traditional video streaming applications. With the demand for anywhere any time connectivity for video chat, it becomes quintessential to develop techniques that adapt to the underlying network, optimize the device architectures, and provide the best possible video chat experience to end users. In this article, we identify main limitations and challenges in mobile video chat. We further discuss possible solutions and propose research directions to make video chat a good experience.

QoS‐aware and autonomic‐oriented multi‐path TCP extensions for mobile and multimedia applications

PurposeThe goal of this paper is to provide contributions aimed at illustrating how the autonomic computing concept may be applied within a multi‐path transport protocol called multi‐path TCP (MPTCP) with the aim of providing as high as possible a quality of service (QoS) for mobile and multimedia applications.Design/methodology/approachThe paper's general vision is to define how to use the general IBM Autonomic framework for the design of a new and generic autonomous transport layer aimed at fitting as best as possible both QoS requirements and network constraints.FindingsSeveral QoS‐aware mechanisms are proposed and evaluated and an ontology‐based semantic model for the dynamic discovery of MPTCP mechanisms is also presented. This work was done within the A2NETS project funded by the ITEA 2 program, a strategic pan‐European program for advanced pre‐competitive research and development (R&D). A2NETS aims at developing a set of common services for M2M devices communicating within heterogeneous networks.Originality/valueThe originality of this paper is the study of the QoS benefits induced by the implementation of the “partial reliability” concept in MPTCP for interactive video applications based on the codec H.264. The second contribution of the paper is that it deals with the decision process that has to be enforced to provide such an adaptable MPTCP with self‐autonomic capabilities. Towards this goal, a semantic ontology‐based model has been proposed allowing to classify and to define clearly the semantic of the different QoS‐oriented mechanisms that have been proposed for MPTCP. This ontology is aimed at being used by the autonomic manager during the decision process step.

The PatchMatch randomized matching algorithm for image manipulation

This paper presents a new randomized algorithm for quickly finding approximate nearest neighbor matches between image patches. Our algorithm offers substantial performance improvements over the previous state of the art (20--100×), enabling its use in new interactive image editing tools, computer vision, and video applications. Previously, the cost of computing such matches for an entire image had eluded efforts to provide interactive performance. The key insight driving our algorithm is that the elements of our search domain---patches of image pixels---are correlated, and thus the search strategy takes advantage of these statistics. Our algorithm uses two principles: first, that good patch matches can be found via random sampling, and second, that natural coherence in the imagery allows us to propagate such matches quickly to surrounding areas. Our simple algorithm allows finding a single nearest neighbor match across translations only, whereas our general algorithm additionally allows matching of k -nearest neighbors, across all rotations and scales, and matching arbitrary descriptors. This one simple algorithm forms the basis for a variety of applications including image retargeting, completion, reshuffling, object detection, digital forgery detection, and video summarization.

A Free-Viewpoint Video Renderer

Free-viewpoint video allows interactive control of the camera viewpoint in video playback using a three-dimensional (3D) scene representation. This paper describes the challenges and the state-of-the-art techniques adopted in developing a free-viewpoint renderer for 3D video production of people. The system requires video streams from a set of fixed or dynamic real-world video cameras and synthesizes arbitrary viewpoints on the GPU using a dynamic 3D scene model. We have released the renderer as an open source application for the research community (see the web information at the end of this paper). The renderer provides (i) the facility to simulate camera viewpoints from publicly available multiple-view video datasets and (ii) a baseline technique for free-viewpoint video synthesis in the development of interactive 3D video and 3DTV applications.