A Corneal Surface Reflections-Based Intelligent System for Lifelogging Applications

Abstract

Corneal Surface Reflections, or reflections on our eye-surface, have been shown as a valid and more socially acceptable source of information for passive lifelogging applications by prior work. However, automatic analysis of corneal surface reflections from a single RGB camera to support passive lifelogging is not extensively investigated in prior work. To address this, we developed a synthetic and self-supervised learning-based two-stage pipeline of deep learning models to detect objects in these reflections. Our prototype only consists a single RGB camera looking into the eye. We collected data from different users in uncontrolled environments using the prototype and trained our system to detect multiple classes of objects present in a typical office environment. We then evaluated our model in partially-controlled and in-the-wild scenarios. In addition, based on the findings from a follow up user study and prior work, we discuss strengths and weaknesses of our system and using corneal surface reflections for passive lifelogging. Finally, we opensource our source codes and trained checkpoints.