Janus meta-imager: asymmetric image transmission and transformation enabled by diffractive neural networks

Diffractive deep neural networks have been introduced earlier as an optical machine learning framework that uses task-specific diffractive surfaces designed by deep learning to all-optically perform inference, achieving promising performance for object classification and imaging. Here we demonstrate systematic improvements in diffractive optical neural networks based on a differential measurement technique that mitigates the non-negativity constraint of light intensity. In this scheme, each class is assigned to a separate pair of photodetectors, behind a diffractive network, and the class inference is made by maximizing the normalized signal difference between the detector pairs. Moreover, by utilizing the inherent parallelization capability of optical systems, we reduced the signal coupling between the positive and negative detectors of each class by dividing their optical path into two jointly-trained diffractive neural networks that work in parallel. We further made use of this parallelization approach, and divided individual classes among multiple jointly-trained differential diffractive neural networks. Using this class-specific differential detection in jointly-optimized diffractive networks, our simulations achieved testing accuracies of 98.52%, 91.48% and 50.82% for MNIST, Fashion-MNIST and grayscale CIFAR-10 datasets, respectively. Similar to ensemble methods practiced in machine learning, we also independently-optimized multiple differential diffractive networks that optically project their light onto a common detector plane, and achieved testing accuracies of 98.59%, 91.06% and 51.44% for MNIST, Fashion-MNIST and grayscale CIFAR-10, respectively. Through these systematic advances in designing diffractive neural networks, the reported classification accuracies set the state-of-the-art for an all-optical neural network design.

The rapid development of artificial intelligence has stimulated the interest of the novel designs of photonic neural networks (PNNs) because of the high speed, low energy consumption and parallelism nature of the light. Based on optical holographic technology, a kind of three-dimensional PNNs, diffractive neural networks (DNNs), have demonstrated their superb performance in parallel two-dimensional data processing. DNNs are composed of multi-layer cascaded holographic plates. Relying on the diffraction of the incident light, each pixel in every layer can be connected with multiple pixels in the next layer to mimic the architecture of the biological nervous system. Important applications, such as image recognition, optical logic operation, and image reconstruction, have been realized on DNNs with high operation efficiency. However, in most of the reported works, the layers of DNNs are spatially separated with a large size of centimeter-scale, which greatly limits the on-chip integration of DNNs. In this work, we reported a green-light bilayer integrated DNNs. The two layers of the DNNs were integrated on the double sides of a quartz wafer respectively by lithography followed by dry etching. Based on the theory of diffraction, the DNNs were trained with a size of millimeter-scale. When the DNNs work, the incident optical signal first passes through the 1st layer of the DNNs, then diffracts inside the quartz wafer, and finally emitted out from the 2nd layer of the DNNs on the backside. Handwritten digital recognition of 0~1 (89 % accuracy) or 0~9 (65% accuracy) was successfully realized. The high stability of quartz provides the basis for the long-term reliable operation of DNNs. The manufacturing of the DNNs is compatible with the mature semiconductor manufacturing technology, which provides a feasible route for the macro fabrication of DNNs.

The modern-day resurgence of machine learning has encouraged researchers to revisit older problem spaces from a new perspective. One promising avenue has been implementing deep neural networks to aid in the simulation of physical systems. In the field of optics, densely connected neural networks able to mimic wave propagation have recently been constructed. These diffractive deep neural networks (D2NN) not only offer new insights into wave propagation, but provide a novel tool for investigating and discovering multi-functional diffractive elements. In this paper, we derive an efficient GPU-friendly D2NN methodology based on Rayleigh-Sommerfeld diffraction. We then use the implementation to virtually forge cascades of optical phase masks subject to different beam steering conditions. The input and output conditions we use to train each D2NN instance is based on commercial electro-optic modulated waveguide systems to encourage experimental follow-on. In total, we analyze the beam steering efficacy of 27 individual D2NN instances which explore different permutations of input sources, mask cascades, and output steering targets.

The formulation and training of unitary neural networks is the basis of an active modulation diffractive deep neural network. In this Letter, an optical random phase DropConnect is implemented on an optical weight to manipulate a jillion of optical connections in the form of massively parallel sub-networks, in which a micro-phase assumed as an essential ingredient is drilled into Bernoulli holes to enable training convergence, and malposed deflections of the geometrical phase ray are reformulated constantly in epochs, allowing for enhancement of statistical inference. Optically, the random micro-phase-shift acts like a random phase sparse griddle with respect to values and positions, and is operated in the optical path of a projective imaging system. We investigate the performance of the full-drilling and part-drilling phenomena. In general, random micro-phase-shift part-drilling outperforms its full-drilling counterpart both in the training and inference since there are more possible recombinations of geometrical ray deflections induced by random phase DropConnect.

We demonstrate significant improvements in the inference accuracy of diffractive optical neural networks and report that a five-layer, phase-only (or amplitude/phase) modulation diffractive network can achieve 97.18% (97.81%) and 89.13% (89.32%) blind-testing accuracy for MNIST and Fashion-MNIST datasets, respectively. Moreover, the integration of diffractive neural networks with electronic deep neural networks is investigated. Using a single fully-connected layer on the electronic part and a five-layer, phase-only diffractive neural network at the optical front-end, we achieved blind-testing accuracies of 98.71% and 90.04% for MNIST and Fashion-MNIST datasets, respectively, despite a >7.8-fold reduction in the number of pixels at the opto-electronic sensor-array.

Artificial intelligence is a technology for simulating and extending human intelligence and has rapidly altered many aspects of modern society. Optical processors, which compute with photons instead of electrons, can fundamentally accelerate the development of artificial intelligence by offering substantially improved computing performance. Photonic approaches for demonstrating artificial neural networks as one of the most widely used frameworks in artificial intelligence, show extraordinary potential to achieve brain-inspired information processing at the speed of light. Recently, all-optical diffractive deep neural networks have been created that are based on passive structures and can perform complicated functions designed by computer-based neural networks. However, existing passive diffractive deep neural networks are not reconfigurable and once is fabricated, its function is fixed. This work reports an on-chip programmable deep diffractive neural network (OPD2NN), in which the optical neurons are built with Sb2Se3 phase change material, making the network reconfigurable and non-volatile. Using numerical simulations, the performance of the OPD2NN is benchmarked on two machine learning tasks that are learning a multifunctional (AND-OR-NOT) logic gate and classification of images of handwritten digits from the MNIST dataset and the obtained results are validated using FDTD simulations by a commercially available full-wave electromagnetic solver. Both numerical and FDTD simulations indicate that a five-hidden-layer OPD2NN with a footprint of 30μmx150μm can appropriately handle (AND-OR-NOT) logic functions. For handwritten digits (0-1-2-3) classification, a three-hidden-layer OPD2NN with a footprint of 60μmx105μm can achieve numerical testing accuracy of 91.5% on the test dataset and the FDTD verification results show 73% matching with numerical testing results.

All-optical diffractive neural networks have attracted extensive attention due to their characteristics of high parallelism, high processing speed, and low energy consumption. However, most existing DNN frameworks are designed for single-task operations and lack the flexibility to handle multiple tasks within an artificial intelligence (AI) system. Here, we propose polarization-multiplexed diffractive neural network (PMDNN) based on liquid crystals for multi-task classification. By incorporating liquid crystal into the neural network in the form of Jones matrix and encoding multi-task inputs into multiple polarization channels, the neurons are endowed with polarization modulation capabilities. This approach extends multi-task processing capacity and enhances the polarization multiplexing capability of liquid crystals without complex structural design. As a demonstration, a 3-task PMDNN was designed to perform classification on the MNIST, Fashion-MNIST, and KMNIST datasets. Consistent simulation and experimental results verify the effectiveness of the proposed network framework. Furthermore, we have designed multi-task PMDNNs for more task classification. These results demonstrate that the proposed framework maintains minimal inter-task crosstalk under increasing task complexity. The proposed PMDNN architecture enhances the flexibility of diffractive neural networks, paving the way for the realization of ultra-fast, low-power, and multi-task integration AI systems.

The diffractive deep neural network is a novel network model that applies the principles of diffraction to neural networks, enabling machine learning tasks to be performed through optical principles. In this paper, a fully optical authentication model is developed using the diffractive deep neural network. The model utilizes terahertz light for propagation and combines it with a self-calibration single-pixel imaging model to construct a comprehensive optical authentication system with faster authentication speed. The proposed system filters the authentication images, establishes an optical connection with the Fourier zero-frequency response of the illumination pattern, and introduces the signal-to-noise ratio as a criterion for batch image authentication. Computer simulations demonstrate the fast speed and strong automation performance of the proposed optical authentication system, suggesting broad prospects for the combined application of diffractive deep neural networks and optical systems.

The development of new storage media to meet the demands for diverse information storage scenarios is a great challenge. Here, a series of lanthanide‐based luminescent organogels with ultrastrong mechanical performance and outstanding plasticity are developed for patterned information storage and encryption applications. The organogels possessing outstanding mechanical properties and tunable luminescent colors are prepared by electrostatic and coordinative interactions between natural DNA, synthetic ligands, and rare earth (RE) ions. The organogel‐REs can be stretched by 180 times and show an ultrastrong breaking strength of 80 MPa. A series of applications with both information storage and encryption, such as self‐information pattern, quick response (QR) code, and barcode, are successfully demonstrated by the organogel‐REs. The developed information storage systems have various advantages of good processability, high stretchability, excellent stability, and versatile design of information patterns. Therefore, the organogel‐RE‐based information storage systems are suitable for applications under different scenarios, such as flexible devices under repeating rude operations. The advancements will enable the design and development of luminescent organogel‐REs as information storage and encryption media for various scenarios.

Machine learning has made remarkable achievements in image classification in recent years. It has also inspired the emergence of optical machine learning frameworks as an effective combination of optics and artificial intelligence. Diffractive deep neural network (D2NN) is a recently proposed optical neural network framework, which is designed and trained by computers, and then fabricated by the usage of 3D printing technology. It can recognize handwritten digital dataset and fashion product dataset with only light source, and the recognition effect is comparable to that of electronic neural network. We propose a diffractive light neural network structure is improved on the basis of the traditional diffractive deep neural network (D2NN), providing a multi-channel design model, while the distribution of neuron nodes on the network layer is adjusted in such a way that different network layers correspond to incoming light waves of different frequencies. The neuron nodes on the network layer are arranged in a concentric circle pattern to reduce the parameters generated during network training, and in the output layer we get channels of different frequencies and then merge these channels to obtain the final results. It is experimentally verified that the diffractive optical neural network with multiple channels has a large improvement of 95.44% recognition accuracy compared to the 89.92% recognition accuracy of the traditional D2NN for recognition of 10 different alphabetic image datasets. The proposed network improves the recognition accuracy by increasing the number of channels while reducing the parameters, which provides a guidance to design optical networks with reduced cost and improved efficiency.

We introduce a differential measurement scheme in diffractive neural networks, where object-classes are assigned to separate opto-electronic detector pairs and the class-inference is made based on the maximum normalized differential signal. This scheme enables diffractive networks to achieve blind-testing accuracies of 98.54% and 48.51% for MNIST and CIFAR-10 datasets, respectively. These accuracies improve to 98.52% and 50.82%, when differential detection is combined with the joint-training of parallel diffractive networks, with each specializing on a separate object-class. Finally, we report independently-trained diffractive networks that project their output-light onto a common plane to achieve 98.59% and 51.44%, for the same datasets, respectively.

Optical neural networks can effectively address hardware constraints and parallel computing efficiency issues inherent in electronic neural networks. However, the inability to implement convolutional neural networks at the all-optical level remains a hurdle. In this work, we propose an optical diffractive convolutional neural network (ODCNN) that is capable of performing image processing tasks in computer vision at the speed of light. We explore the application of the 4f system and the diffractive deep neural network (D2NN) in neural networks. ODCNN is then simulated by combining the 4f system as an optical convolutional layer and the diffractive networks. We also examine the potential impact of nonlinear optical materials on this network. Numerical simulation results show that the addition of convolutional layers and nonlinear functions improves the classification accuracy of the network. We believe that the proposed ODCNN model can be the basic architecture for building optical convolutional networks.

In this Letter, we propose an all-optical diffractive deep neural network modeling method based on nonlinear optical materials. First, the nonlinear optical properties of graphene and zinc selenide (ZnSe) are analyzed. Then the optical limiting effect function corresponding to the saturation absorption coefficient of the nonlinear optical materials is fitted. The optical limiting effect function is taken as the nonlinear activation function of the neural network. Finally, the all-optical diffractive neural network model based on nonlinear materials is established. The numerical simulation results show that the model can effectively improve the nonlinear representation ability of the all-optical diffractive neural network. It provides a theoretical support for the further realization of a photonic artificial intelligence chip based on nonlinear optical materials.

Structural color caused by thin‐film interference is widespread and simple in nature. Many researchers have already showed the reversible color change of structural color system, but it is difficult to regulate dynamically with tunable intensity and viewable angle. Herein, a dynamic structural color platform is reported by combining spontaneous thin film interference and wrinkling phenomenon in nature. This robust yet low cost strategy enables large‐scale and spatially arbitrary preparation of a uniform structural color surface through the whole visible spectral range. Furthermore, the prepared isotropic and anisotropic structural colors on disordered and ordered wrinkled thin films (WTFs) exhibit different optical properties that can be precisely and reversibly regulated by ultraviolet (UV) light and near‐infrared (NIR) light irradiation. This exquisite and tunable structural color platform may find applications in information storage, smart display, anticounterfeiting and encryption.

An all‐optical diffractive neural network (DNN), as the fusion of optics and artificial intelligence, utilizes light‐based computational architecture, providing potential beyond Moore's law limitations due to their low energy consumption and high parallel processing speed. However, existing DNN frameworks face limitations in realizing spatially assembled architecture (i.e., by combining two or more independent physical layers together) to create a Lego‐like reconfigurable DNN that can generate additional functionalities. In this work, the modular programming concept is introduced to develop the modular diffractive neural networks (MDNNs) using the cascaded metasurfaces where each metasurface enables respective functionalities while the cascaded metasurfaces possess additional functionalities. The MDNNs showcase great advantages in flexibility and reconfigurability for multitask functionalities. When working independently, two metasurface‐based modules can function as classifiers of handwritten letters and fashion products, respectively. When the two modules are assembled with different spatial orders, the additional functions of the handwritten digits classifier and imager can be obtained. Moreover, the MDNNs can be designed to mimic an architecture for high‐capacity encrypted communication. This flexible and multiplexed MDNNs framework shows great potential and paves the new way for all‐optical computation with multifunctional integration, massively parallel processing, and all‐optical information security.