A virtual reality interface for the immersive manipulation of live microscopic systems

Stefano Ferretti,Roberto Di Leonardo,Silvio Bianchi,Giacomo Frangipane,Giacomo Frangipane,Roberto Di Leonardo

doi:10.1038/s41598-021-87004-5

Stefano Ferretti, Roberto Di Leonardo + Show 4 more

Open Access

https://doi.org/10.1038/s41598-021-87004-5

Copy DOI

Abstract

For more than three centuries we have been watching and studying microscopic phenomena behind a microscope. We discovered that cells live in a physical environment whose predominant factors are no longer those of our scale and for which we lack a direct experience and consequently a deep intuition. Here we demonstrate a new instrument which, by integrating holographic and virtual reality technologies, allows the user to be completely immersed in a dynamic virtual world which is a simultaneous replica of a real system under the microscope. We use holographic microscopy for fast 3D imaging and real-time rendering on a virtual reality headset. At the same time, hand tracking data is used to dynamically generate holographic optical traps that can be used as virtual projections of the user hands to interactively grab and manipulate ensembles of microparticles or living motile cells.

Highlights

For more than three centuries we have been watching and studying microscopic phenomena behind a microscope
In the case of holographic trapping, a collection of laser spots is created in 3D space by modulating the light field on the Spatial Light Modulator (SLM) plane, while, in holographic microscopy, a volumetric light distribution is obtained from a 2D interference pattern recorded by the camera
The SLM shapes the wavefront of a collimated laser beam in such a way that, after propagation through a microscope objective, an array of focal spots is generated in a 3D volume around the focal plane

Summary

Introduction

For more than three centuries we have been watching and studying microscopic phenomena behind a microscope. By connecting a haptic device to the scanning tip of a STM and using a virtual reality headset to render topographic reconstructions, the user could either “fly” over the surface or “feel” the surface under her hand or deposit gold mounds over it[19] Following this pioneering work, the field of telenanorobotics has shifted towards the use of the Atomic Force Microscope (AFM)[20] which offers the advantages of being more suitable for a wider range of samples, including biological cells and macromolecules, and to provide direct force measurements for tactile feedback. Novel approach to tele-micromanipulation that is fully based on optical tools for simultaneous 3D visualization and real-time manipulation of micro-systems through an immersive Virtual Reality interface Through this instrument we can be virtually “shrunk ” by a million times and sent inside a sealed microscope slide where we can grab swimming cells on the fly and use our hands to interactively arrange colloidal particles in precise 3D geometric configurations. Further developments in computational microscopy will allow direct interaction with more complex objects like larger eukaryotic cells or complex micromachines[25] which could be assembled and operated in an immersive environment

Methods

Results

Conclusion