Caffe2Unity: Immersive Visualization and Interpretation of Deep Neural Networks

Aisha Aamir,Minija Tamosiunaite,Florentin Wörgötter

doi:10.3390/electronics11010083

Aisha Aamir, Minija Tamosiunaite + Show 1 more

Open Access

https://doi.org/10.3390/electronics11010083

Copy DOI

Journal: Electronics	Publication Date: Dec 28, 2021
Citations: 7	License type: CC BY 4.0

Affiliation: University of Göttingen

Abstract

Deep neural networks (DNNs) dominate many tasks in the computer vision domain, but it is still difficult to understand and interpret the information contained within these networks. To gain better insight into how a network learns and operates, there is a strong need to visualize these complex structures, and this remains an important research direction. In this paper, we address the problem of how the interactive display of DNNs in a virtual reality (VR) setup can be used for general understanding and architectural assessment. We compiled a static library as a plugin for the Caffe framework in the Unity gaming engine. We used routines from this plugin to create and visualize a VR-based AlexNet architecture for an image classification task. Our layered interactive model allows the user to freely navigate back and forth within the network during visual exploration. To make the DNN model even more accessible, the user can select certain connections to understand the activity flow at a particular neuron. Our VR setup also allows users to hide the activation maps/filters or even interactively occlude certain features in an image in real-time. Furthermore, we added an interpretation module and reframed the Shapley values to give a deeper understanding of the different layers. Thus, this novel tool offers more direct access to network structures and results, and its immersive operation is especially instructive for both novices and experts in the field of DNNs.

Highlights

A virtual reality (VR) environment is a great way to interact with complex scientific visualizations, immense datasets, and complicated 3D structures [1]
We address the problem of how the interactive display of Deep neural networks (DNNs) in a virtual reality (VR) setup can be used for general understanding and architectural assessment
We developed a dynamic link library for the Caffe framework of deep neural networks, which was used as a plugin for the Unity gaming engine

Summary

Introduction

A virtual reality (VR) environment is a great way to interact with complex scientific visualizations, immense datasets, and complicated 3D structures [1]. It is believed that a user can learn and explore multifaceted structures more by navigating around them and exploring different characteristics for better understanding [1,6] In this regard, DNNs have gained much popularity in the past decade due to the availability of faster computing devices that can minimize their computationally expensive training times. DNN visualizations have been restricted to 2D visualization of images, which is not sufficient if we want to visualize the internal representations and interpretations of the network [1,5,6] Despite their tremendous performance in many computer vision tasks, these models are still considered to be “black-boxes” and require more transparency and interpretation to understand their decision-making process. Earlier works regarding visualizations have focused on expert users with an in-depth understanding of DNNs and were specially tailored for developers in this field [7,9,10]

Objectives

Methods

Results

Conclusion