Abstract
Deep neural networks (DNNs) dominate many tasks in the computer vision domain, but it is still difficult to understand and interpret the information contained within these networks. To gain better insight into how a network learns and operates, there is a strong need to visualize these complex structures, and this remains an important research direction. In this paper, we address the problem of how the interactive display of DNNs in a virtual reality (VR) setup can be used for general understanding and architectural assessment. We compiled a static library as a plugin for the Caffe framework in the Unity gaming engine. We used routines from this plugin to create and visualize a VR-based AlexNet architecture for an image classification task. Our layered interactive model allows the user to freely navigate back and forth within the network during visual exploration. To make the DNN model even more accessible, the user can select certain connections to understand the activity flow at a particular neuron. Our VR setup also allows users to hide the activation maps/filters or even interactively occlude certain features in an image in real-time. Furthermore, we added an interpretation module and reframed the Shapley values to give a deeper understanding of the different layers. Thus, this novel tool offers more direct access to network structures and results, and its immersive operation is especially instructive for both novices and experts in the field of DNNs.
Highlights
A virtual reality (VR) environment is a great way to interact with complex scientific visualizations, immense datasets, and complicated 3D structures [1]
We address the problem of how the interactive display of Deep neural networks (DNNs) in a virtual reality (VR) setup can be used for general understanding and architectural assessment
We developed a dynamic link library for the Caffe framework of deep neural networks, which was used as a plugin for the Unity gaming engine
Summary
A virtual reality (VR) environment is a great way to interact with complex scientific visualizations, immense datasets, and complicated 3D structures [1]. It is believed that a user can learn and explore multifaceted structures more by navigating around them and exploring different characteristics for better understanding [1,6] In this regard, DNNs have gained much popularity in the past decade due to the availability of faster computing devices that can minimize their computationally expensive training times. DNN visualizations have been restricted to 2D visualization of images, which is not sufficient if we want to visualize the internal representations and interpretations of the network [1,5,6] Despite their tremendous performance in many computer vision tasks, these models are still considered to be “black-boxes” and require more transparency and interpretation to understand their decision-making process. Earlier works regarding visualizations have focused on expert users with an in-depth understanding of DNNs and were specially tailored for developers in this field [7,9,10]
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
