Measuring Objective Visual Quality of Real-Time Communication Systems in the Wild

Abstract

During the outbreak of epidemic diseases, the importance of real-time communication (RTC) systems dramatically increases. People use RTC systems for communicating with others, presenting projects, attending online courses, and sharing videos. With different network conditions, applications, and scenarios, how to choose an appropriate system for high-quality RTC is an open question. To the best of our knowledge, there is no general and unified method to comprehensively evaluate the performance of the publicly available RTC systems. In this paper, we systematically evaluate several performances of RTC systems. Our method treats systems as a black-box, which can be easily adapted to other systems. Our method is also available for other video transmission systems, such as streaming and live broadcasting systems. According to our measurement method, we evaluate three web-based and three software-based RTC systems on two video conferencing (VC)-based and two screen sharing (SS)-based scenarios. We measure the received video quality (graphical quality and frame rate) at the receiver, the upload bitrate at the sender, and four usages of local resource. Furthermore, we propose a new metric to measure the ability of the system to handle insufficient bandwidth situations. Our proposed metric is the first one directly measure the ability of the rate adaptation mechanism for RTC systems. We expect the measurement method, the metric, and our findings can help system development in the future. Our detailed analysis reveals that (1) the software-based systems are more efficient than the web-based systems for bitrate usage; (2) the web-based systems are more tolerant to insufficient bandwidth conditions than the software-based systems; (3) the studied RTC systems currently are not designed for the scenario of sharing dynamic videos because of their low-frame-rate strategy, which limits the usage; and (4) decreasing graphical quality is more likely to be recognized than decreasing the frame rate. Frame rate adjustment for rate adaptation can be considered.