A vision for contextualized evaluation of remote collaboration supported by AR

Abstract

Remote collaboration using Augmented Reality (AR) has potential to support physically distributed team-members that need to achieve a common goal by increasing knowledge retention, improving understanding and awareness of the problem and its context. In this vein, the path to achieve usable, realistic and impactful solutions must entail an explicit understanding regarding how remote collaboration occurs through AR and how it may help contribute to a more effective work effort. Thus, characterization and evaluation of the collaborative process is paramount, but a particularly challenging endeavor, due to the multitude of aspects that define the collaboration effort. In this context, the work presented here contributes with a critical analysis, discussing current evaluation efforts, identifying limitations and opportunities. Then, we outline a conceptual framework to support researchers in conducting evaluations in a more structured manner. To instrument this vision, an evaluation toolkit is proposed to support contextual data collection and analysis in such scenarios and obtain an additional perspective on selected dimensions of collaboration. We illustrate the usefulness and versatility of the toolkit through a case study on remote maintenance, comparing two distinct methods: sharing of video and AR-based annotations. Last, we discuss the results obtained, showing the proposed vision allows to have an additional level of insights to better understand what happened, eliciting a more complete characterization of the work effort.