Diffractive Optical Neural Networks Research Articles

Reconfigurable binary diffractive optical neural network based on chalcogenide phase change material Ge2Sb2Se4Te1

Diffractive optical neural networks (DONNs) possess unique advantages such as light-speed computing, low energy consumption, and parallel processing, which have obtained increasing attention in recent years. However, once conventional DONNs are fabricated, their function remains fixed, which greatly limits the applications of DONNs. Thus, we propose a reconfigurable DONN framework based on a repeatable and non-volatile phase change material Ge2Sb2Se4Te1(GSST). By utilizing phase modulation units made of GSST to form the network's neurons, we can flexibly switch the functions of the DONN. Meanwhile, we apply a binary training algorithm to train the DONN weights to binary values of 0 and π, which is beneficial for simplifying the design and fabrication of DONN while reducing errors during physical implementation. Furthermore, the reconfigurable binary DONN has been trained as a handwritten digit classifier and a fashion product classifier to validate the feasibility of the framework. This work provides an efficient and flexible control mechanism for reconfigurable DONNs, with potential applications in various complex tasks.

Multimodal deep learning using on-chip diffractive optics with in situ training capability

Multimodal deep learning plays a pivotal role in supporting the processing and learning of diverse data types within the realm of artificial intelligence generated content (AIGC). However, most photonic neuromorphic processors for deep learning can only handle a single data modality (either vision or audio) due to the lack of abundant parameter training in optical domain. Here, we propose and demonstrate a trainable diffractive optical neural network (TDONN) chip based on on-chip diffractive optics with massive tunable elements to address these constraints. The TDONN chip includes one input layer, five hidden layers, and one output layer, and only one forward propagation is required to obtain the inference results without frequent optical-electrical conversion. The customized stochastic gradient descent algorithm and the drop-out mechanism are developed for photonic neurons to realize in situ training and fast convergence in the optical domain. The TDONN chip achieves a potential throughput of 217.6 tera-operations per second (TOPS) with high computing density (447.7 TOPS/mm2), high system-level energy efficiency (7.28 TOPS/W), and low optical latency (30.2 ps). The TDONN chip has successfully implemented four-class classification in different modalities (vision, audio, and touch) and achieve 85.7% accuracy on multimodal test sets. Our work opens up a new avenue for multimodal deep learning with integrated photonic processors, providing a potential solution for low-power AI large models using photonic technology.

Role of spatial coherence in diffractive optical neural networks.

Diffractive optical neural networks (DONNs) have emerged as a promising optical hardware platform for ultra-fast and energy-efficient signal processing for machine learning tasks, particularly in computer vision. Previous experimental demonstrations of DONNs have only been performed using coherent light. However, many real-world DONN applications require consideration of the spatial coherence properties of the optical signals. Here, we study the role of spatial coherence in DONN operation and performance. We propose a numerical approach to efficiently simulate DONNs under incoherent and partially coherent input illumination and discuss the corresponding computational complexity. As a demonstration, we train and evaluate simulated DONNs on the MNIST dataset of handwritten digits to process light with varying spatial coherence.

All-optically untangling light propagation through multimode fibers

When light propagates through a complex medium, such as a multimode optical fiber (MMF), the spatial information it carries is scrambled. In this work we experimentally demonstrate an all-optical strategy to unscramble this light again. We first create a digital model capturing the way light has been scattered, and then use this model to inverse-design and build a complementary optical system–which we call an optical inverter–that reverses this scattering process. Our implementation of this concept is based on multi-plane light conversion, and can also be understood as a physical matrix pre-conditioner, or a linear diffractive optical neural network. We present three design strategies allowing different aspects of device performance to be prioritized. We experimentally demonstrate a prototype optical inverter capable of simultaneously unscrambling up to 30 spatial modes that have propagated through a 1 m long MMF, and show how this promises near instantaneous incoherent imaging, without the need for any beam scanning or computational processing. We also demonstrate the reconfigurable nature of this prototype, allowing it to adapt and deliver a new optical transformation if the MMF it is matched to changes configuration. Our work represents a step towards a new way to see through scattering media. Beyond imaging, this concept may also have applications to the fields of optical communications, optical computing and quantum photonics.

A surface-normal photodetector as nonlinear activation function in diffractive optical neural networks

Optical neural networks (ONNs) enable high speed, parallel, and energy efficient processing compared to their conventional digital electronic counterparts. However, realizing large scale ONN systems is an open problem. Among various integrated and non-integrated ONNs, free-space diffractive ONNs benefit from a large number of pixels of spatial light modulators to realize millions of neurons. However, a significant fraction of computation time and energy is consumed by the nonlinear activation function that is typically implemented using a camera sensor. Here, we propose a novel surface-normal photodetector (SNPD) with an optical-in–electrical-out (O–E) nonlinear response to replace the camera sensor that enables about three orders of magnitude faster (5.7 µs response time) and more energy efficient (less than 10 nW/pixel) response. Direct efficient vertical optical coupling, polarization insensitivity, inherent nonlinearity with no control electronics, low optical power requirements, and the possibility of implementing large scale arrays make the SNPD a promising O–E nonlinear activation function for diffractive ONNs. To show the applicability of the proposed neural nonlinearity, successful classification simulations of the MNIST and Fashion MNIST datasets using the measured response of SNPD with accuracy comparable to that of an ideal ReLU function are demonstrated.

Scientific Computing with Diffractive Optical Neural Networks

Diffractive optical neural networks (DONNs) are emerging as high‐throughput and energy‐efficient hardware platforms to perform all‐optical machine learning (ML) in machine vision systems. However, the current demonstrated applications of DONNs are largely image classification tasks, which undermine the prospect of developing and utilizing such hardware for other ML applications. Herein, the deployment of an all‐optical reconfigurable DONNs system for scientific computing is demonstrated numerically and experimentally, including guiding two‐dimensional quantum material synthesis, predicting the properties of two‐dimensional quantum materials and small molecular cancer drugs, predicting the device response of nanopatterned integrated photonic power splitters, and the dynamic stabilization of an inverted pendulum with reinforcement learning. Despite a large variety of input data structures, a universal feature engineering approach is developed to convert categorical input features to images that can be processed in the DONNs system. The results open up new opportunities for employing DONNs systems for a broad range of ML applications.

C-DONN: compact diffractive optical neural network with deep learning regression.

A new method to improve the integration level of an on-chip diffractive optical neural network (DONN) is proposed based on a standard silicon-on-insulator (SOI) platform. The metaline, which represents a hidden layer in the integrated on-chip DONN, is composed of subwavelength silica slots, providing a large computation capacity. However, the physical propagation process of light in the subwavelength metalinses generally requires an approximate characterization using slot groups and extra length between adjacent layers, which limits further improvements of the integration of on-chip DONN. In this work, a deep mapping regression model (DMRM) is proposed to characterize the process of light propagation in the metalines. This method improves the integration level of on-chip DONN to over 60,000 and elimnates the need for approximate conditions. Based on this theory, a compact-DONN (C-DONN) is exploited and benchmarked on the Iris plants dataset to verify the performance, yielding a testing accuracy of 93.3%. This method provides a potential solution for future large-scale on-chip integration.

Machine learning enables the design of a bidirectional focusing diffractive lens.

Machine learning can efficiently empower the inverse design of cascaded diffractive optical elements. In this work, we explore the inverse design of a bidirectional focusing diffractive lens in a cascaded configuration through the diffractive optical neural network (DONN) machine learning method. The bidirectional focusing diffractive lens consists of two on-axially cascaded multi-level diffractive lenses. Each lens consists of concentric rings with equal widths and varying heights. The height of each concentric ring is optimized as part of the design algorithm. The diffractive lens has a focal length f+ as light propagates in the forward (Z+) direction. As light propagates in the backward (Z-) direction, the focal length changes to f-. The designed lens is fabricated through a two-photon polymerization 3D printing technique. The proposed design is polarization insensitive and miniature and can be readily applied in future functional optical imaging systems.

Parallelized and Cascadable Optical Logic Operations by Few-Layer Diffractive Optical Neural Network

Optical computing has gained much attention due to its high speed, low energy consumption, and the fact that it is naturally parallelizable and multiplexable, etc. Single-bit optical logic gates based on a four-hidden-layer diffractive optical neural network (DONN) have been demonstrated with paired apertures. Here, we show a parallel-logic operation strategy based on two-hidden-layer DONN, showcasing their efficiency by multiple-bit (up to 16-bit) optical logic (e.g., NAND) operations. In addition, we demonstrate how NAND-DONN units can be utilized to achieve NOR and AND operations by flipping and cascading the DONN.

Photonic machine learning with on-chip diffractive optics

Machine learning technologies have been extensively applied in high-performance information-processing fields. However, the computation rate of existing hardware is severely circumscribed by conventional Von Neumann architecture. Photonic approaches have demonstrated extraordinary potential for executing deep learning processes that involve complex calculations. In this work, an on-chip diffractive optical neural network (DONN) based on a silicon-on-insulator platform is proposed to perform machine learning tasks with high integration and low power consumption characteristics. To validate the proposed DONN, we fabricated 1-hidden-layer and 3-hidden-layer on-chip DONNs with footprints of 0.15 mm2 and 0.3 mm2 and experimentally verified their performance on the classification task of the Iris plants dataset, yielding accuracies of 86.7% and 90%, respectively. Furthermore, a 3-hidden-layer on-chip DONN is fabricated to classify the Modified National Institute of Standards and Technology handwritten digit images. The proposed passive on-chip DONN provides a potential solution for accelerating future artificial intelligence hardware with enhanced performance.

Effects of interlayer reflection and interpixel interaction in diffractive optical neural networks.

Multilayer diffractive optical neural networks (DONNs) can perform machine learning (ML) tasks at the speed of light with low energy consumption. Decreasing the number of diffractive layers can reduce inevitable material and diffraction losses to improve system performance, and incorporating compact devices can reduce the system footprint. However, current analytical DONN models cannot accurately describe such physical systems. Here we show the ever-ignored effects of interlayer reflection and interpixel interaction on the deployment performance of DONNs through full-wave electromagnetic simulations and terahertz (THz) experiments. We demonstrate that the drop of handwritten digit classification accuracy due to reflection is negligible with conventional low-index THz polymer materials, while it can be substantial with high-index materials. We further show that one- and few-layer DONN systems can achieve high classification accuracy, but there is a trade-off between accuracy and model-system matching rate because of the fast-varying spatial distribution of optical responses in diffractive masks. Deep DONNs can break down such a trade-off because of reduced mask spatial complexity. Our results suggest that new accurate and trainable DONN models are needed to advance the development and deployment of compact DONN systems for sophisticated ML tasks.

Integrated diffractive optical neural network with space-time interleaving

Integrated diffractive optical neural network with space-time interleaving

Realization of optical logic gates using on-chip diffractive optical neural networks

Optical computing is highly desired as a potential strategy for circumventing the performance limitations of semiconductor-based electronic devices and circuits. Optical logic gates are considered as fundamental building blocks for optical computation and they enable logic functions to be performed extremely quickly without the generation of heat and crosstalk. Here, we discuss the design of a multi-functional optical logic gate based on an on-chip diffractive optical neural network that can perform AND, NOT and OR logic operations at the wavelength of 1.55 µm. The wavelength-independent operation of the multi-functional logic gate at seven wavelengths (over a bandwidth of 60 nm) is also studied which paves the way for wavelength division multiplexed parallel computation. This simple, highly-integrable, low-loss, energy-efficient and broadband optical logic gate provides a path for the development of high-speed on-chip nanophotonic processors for future optical computing applications.

Partitionable High-Efficiency Multilayer Diffractive Optical Neural Network.

A partitionable adaptive multilayer diffractive optical neural network is constructed to address setup issues in multilayer diffractive optical neural network systems and the difficulty of flexibly changing the number of layers and input data size. When the diffractive devices are partitioned properly, a multilayer diffractive optical neural network can be constructed quickly and flexibly without readjusting the optical path, and the number of optical devices, which increases linearly with the number of network layers, can be avoided while preventing the energy loss during propagation where the beam energy decays exponentially with the number of layers. This architecture can be extended to construct distinct optical neural networks for different diffraction devices in various spectral bands. The accuracy values of 89.1% and 81.0% are experimentally evaluated for MNIST database and MNIST fashion database and show that the classification performance of the proposed optical neural network reaches state-of-the-art levels.

Physics‐Aware Machine Learning and Adversarial Attack in Complex‐Valued Reconfigurable Diffractive All‐Optical Neural Network

AbstractDiffractive optical neural networks have shown promising advantages over electronic circuits for accelerating modern machine learning (ML) algorithms. However, it is challenging to achieve fully programmable all‐optical implementation and rapid hardware deployment. Here, a large‐scale, cost‐effective, complex‐valued, and reconfigurable diffractive all‐optical neural networks system in the visible range is demonstrated based on cascaded transmissive twisted nematic liquid crystal spatial light modulators. The employment of categorical reparameterization technique creates a physics‐aware training framework for the fast and accurate deployment of computer‐trained models onto optical hardware. Such a full stack of hardware and software enables not only the experimental demonstration of classifying handwritten digits in standard datasets, but also theoretical analysis and experimental verification of physics‐aware adversarial attacks onto the system, which are generated from a complex‐valued gradient‐based algorithm. The detailed adversarial robustness comparison with conventional multiple layer perceptrons and convolutional neural networks features a distinct statistical adversarial property in diffractive optical neural networks. The developed full stack of software and hardware provides new opportunities of employing diffractive optics in a variety of ML tasks and in the research on optical adversarial ML.

Orbital angular momentum optical communications enhanced by artificial intelligence

Angular momentum of light can be divided into spin angular momentum and orbital angular momentum (OAM). Due to the theoretically unlimited orthogonal states, the physical dimension of OAM provides a potential solution to boost the information capacity. The OAM multiplexing and modulation techniques have been implemented to meet the continuous growth of bandwidth requirements, resulting in the concept of OAM optical communication. However, the performances of the traditional optical OAM detection techniques degrade seriously in the practical application of OAM optical communications. Thanks to the powerful data analysis advantages, the cutting-edge machine learning (ML) algorithms have been widely used in the field of image processing, laying the technical foundation for OAM recognition. This paper reviews the recent advances on OAM optical communications that are enhanced by ML methods. More than the traditional OAM detection methods, the OAM demodulation methods based on multiple network architectures, including the support vector machine, self-organizing map, feed-forward neural network, convolutional neural network, and diffractive deep optical neural network (D2NN), have been summarized. We also discuss the development of the spiking neural network and on-chip D2NN, opening a possible way to facilitate the future ultra-low power and ultra-fast OAM demodulation technology.

Graphene plasmonic spatial light modulator for reconfigurable diffractive optical neural networks.

Terahertz (THz) diffractive optical neural networks (DONNs) highlight a new route toward intelligent THz imaging, where the image capture and classification happen simultaneously. However, the state-of-the-art implementation mostly relies on passive components and thus the functionalities are limited. The reconfigurability can be achieved through spatial light modulators (SLMs), while it is not clear what device specifications are required and how challenging the associated device implementation is. Here, we show that a complex-valued modulation with a π/2 phase modulation in an active reflective graphene-plasmonics-based SLM can be employed for realizing the reconfigurability in THz DONNs. By coupling the plasmonic resonance in graphene nanoribbons with the reflected Fabry-Pérot (F-P) mode from a back reflector, we achieve a minor amplitude modulation of large reflection and a substantial π/2 phase modulation. Furthermore, the constructed reconfigurable reflective THz DONNs consisting of designed SLMs demonstrate >94.0% validation accuracy of the MNIST dataset. The results suggest that the relaxation of requirements on the specifications of SLMs should significantly simplify and enable varieties of SLM designs for versatile DONN functionalities.

Fault tolerance and noise immunity in freespace diffractive optical neural networks

Free-space diffractive optical networks are a class of trainable optical media that are currently being explored as a novel hardware platform for neural engines. The training phase of such systems is usually performed in a computer and the learned weights are then transferred onto optical hardware (‘ex-situ training’). Although this process of weight transfer has many practical advantages, it is often accompanied by performance degrading faults in the fabricated hardware. Being analog systems, these engines are also subject to performance degradation due to noises in the inputs and during optoelectronic conversion. Considering diffractive optical networks trained for image classification tasks on standard datasets, we numerically study the performance degradation arising out of weight faults and injected noises and methods to ameliorate these effects. Training regimens based on intentional fault and noise injection during the training phase are only found marginally successful at imparting fault tolerance or noise immunity. We propose an alternative training regimen using gradient based regularization terms in the training objective that are found to impart some degree of fault tolerance and noise immunity in comparison to injection based training regimen.

On-chip photonic diffractive optical neural network based on a spatial domain electromagnetic propagation model.

An integrated physical diffractive optical neural network (DONN) is proposed based on a standard silicon-on-insulator (SOI) substrate. This DONN has compact structure and can realize the function of machine learning with whole-passive fully-optical manners. The DONN structure is designed by the spatial domain electromagnetic propagation model, and the approximate process of the neuron value mapping is optimized well to guarantee the consistence between the pre-trained neuron value and the SOI integration implementation. This model can better ensure the manufacturability and the scale of the on-chip neural network, which can be used to guide the design and manufacturing of the real chip. The performance of our DONN is numerically demonstrated on the prototypical machine learning task of prediction of coronary heart disease from the UCI Heart Disease Dataset, and accuracy comparable to the state-of-the-art is achieved.

Ensemble learning of diffractive optical networks

A plethora of research advances have emerged in the fields of optics and photonics that benefit from harnessing the power of machine learning. Specifically, there has been a revival of interest in optical computing hardware due to its potential advantages for machine learning tasks in terms of parallelization, power efficiency and computation speed. Diffractive deep neural networks (D2NNs) form such an optical computing framework that benefits from deep learning-based design of successive diffractive layers to all-optically process information as the input light diffracts through these passive layers. D2NNs have demonstrated success in various tasks, including object classification, the spectral encoding of information, optical pulse shaping and imaging. Here, we substantially improve the inference performance of diffractive optical networks using feature engineering and ensemble learning. After independently training 1252 D2NNs that were diversely engineered with a variety of passive input filters, we applied a pruning algorithm to select an optimized ensemble of D2NNs that collectively improved the image classification accuracy. Through this pruning, we numerically demonstrated that ensembles of N = 14 and N = 30 D2NNs achieve blind testing accuracies of 61.14 ± 0.23% and 62.13 ± 0.05%, respectively, on the classification of CIFAR-10 test images, providing an inference improvement of >16% compared to the average performance of the individual D2NNs within each ensemble. These results constitute the highest inference accuracies achieved to date by any diffractive optical neural network design on the same dataset and might provide a significant leap to extend the application space of diffractive optical image classification and machine vision systems.