Upconversion Imaging Research Articles

Upconversion imaging through multimode fibers based on deep learning

In this paper, to the best of our knowledge, we first realize imaging of an object passing through multimode fiber (MMF) by using upconversion imaging technique and deep learning. It is shown that as infrared object passes through MMF and/or scattering medium, the infrared speckle is obtained, which cannot be detected by CCD working in visible range. Using upconversion imaging technique based on second harmonic generation, we can shift the infrared speckle pattern to the visible one, which can be recorded by CCD working in visible range. We employ a network model, named pix2pix, which is built based on the generative adversarial network (GAN), to realize image reconstruction. It is found that the trained network can reconstruct image from the visible speckle with high quality, in which the visible speckle is obtained by the upconversed imaging technique. The combination of the upconversion imaging and deep learning is found to be feasible for extending the imaging wavelength and recovering images from speckle with high fidelity.

Infrared-to-Visible Upconversion Devices

Infrared imaging plays remarkable roles in various fields including military, biomedicine, aerospace, and artificial intelligence. However, traditional infrared imaging systems have plenty of disadvantages such as large volume, high cost, and complex fabrication process. Emerging infrared upconversion imaging devices can directly convert low-energy infrared photons into high-energy visible light photons, thus they are promising to accomplish pixel-less high-resolution infrared imaging at low cost. In this paper, recent advances and progress of infrared-to-visible upconversion devices are summarized. We further offer the main limitations of upconversion technology and the challenges that need to be addressed for the future development of infrared upconverters.

Wide-field mid-infrared single-photon upconversion imaging

Frequency upconversion technique, where the infrared signal is nonlinearly translated into the visible band to leverage the silicon sensors, offers a promising alternation for the mid-infrared (MIR) imaging. However, the intrinsic field of view (FOV) is typically limited by the phase-matching condition, thus imposing a remaining challenge to promote subsequent applications. Here, we demonstrate a wide-field upconversion imaging based on the aperiodic quasi-phase-matching configuration. The acceptance angle is significantly expanded to about 30°, over tenfold larger than that with the periodical poling crystal. The extended FOV is realized in one shot without the need of parameter scanning or post-processing. Consequently, a fast snapshot allows to facilitate high-speed imaging at a frame rate up to 216 kHz. Alternatively, single-photon imaging at room temperature is permitted due to the substantially suppressed background noise by the spectro-temporal filtering. Furthermore, we have implemented high-resolution time-of-flight 3D imaging based on the picosecond optical gating. These presented MIR imaging features with wide field, fast speed, and high sensitivity might stimulate immediate applications, such as non-destructive defect inspection, in-vivo biomedical examination, and high-speed volumetric tomography.

Conformal frequency conversion for arbitrary vectorial structured light

Vectorial structured light with spatially varying amplitude, phase, and polarization is reshaping many areas of modern optics, including nonlinear optics, as diverse parametric processes can be used to explore interactions between such complex vector fields, extending the frontiers of optics to new physical phenomena. However, the most basic nonlinear application (i.e., frequency conversion), still remains challenging for vectorial structured light since parametric processes are polarization dependent, leading to a change in the spatial topological structure of signals. In this work, to break this fundamental limit, we propose a conformal frequency conversion scheme that allows the full spatial structure of vectorial structured light to be maintained in the conversion. We systematically examine its spatial polarization independence based on nondegenerate sum-frequency generation with type-0 phase matching. This proof-of-principle demonstration paves the way for a wide range of applications that require conformal frequency conversion, and, particularly, to implement frequency interfaces with multimodal communications channels, high-dimensional quantum states, and polarization-resolved upconversion imaging.

Halide Perovskite Nanocrystals with Enhanced Water Stability for Upconversion Imaging in a Living Cell.

Halide perovskite nanomaterials are widely used in optoelectronics and photonics due to their outstanding luminescent properties, whereas their strong multiphoton absorption makes them prospective for bioimaging. Nonetheless, instability of perovskites in aqueous solutions is an important limitation that prevents their application in biology and medicine. Here, we demonstrate fluorescence and upconversion imaging in living cells by employing CsPbBr3 nanocrystals (NCs) that show an improved water-resistance (at least for 24 h) after their coating as individual particles with various silica-based shells. The obtained phTEOS-TMOS@CsPbBr3 NCs possess high quality, which we confirm with high-resolution transmission and scanning transmission electron microscopy, X-ray diffraction analysis, Fourier-transform infrared and energy-dispersive X-ray spectroscopies, as well as with fluorescence optical microscopy. The developed platform can make the halide perovskite NCs suitable for various bioimaging applications.

Mapping Drug-Induced Neuropathy through In-Situ Motor Protein Tracking and Machine Learning.

Chemotherapy can induce toxicity in the central and peripheral nervous systems and result in chronic adverse reactions that impede continuous treatment and reduce patient quality of life. There is a current lack of research to predict, identify, and offset drug-induced neurotoxicity. Rapid and accurate assessment of potential neuropathy is crucial for cost-effective diagnosis and treatment. Here we report dynamic near-infrared upconversion imaging that allows intraneuronal transport to be traced in real time with millisecond resolution, but without photobleaching or blinking. Drug-induced neurotoxicity can be screened prior to phenotyping, on the basis of subtle abnormalities of kinetic characteristics in intraneuronal transport. Moreover, we demonstrate that combining the upconverting nanoplatform with machine learning offers a powerful tool for mapping chemotherapy-induced peripheral neuropathy and assessing drug-induced neurotoxicity.

Frequency upconversion imaging based on Hadamard coding

A frequency upconversion imaging based on Hadamard coding is presented to remove the distorting effect on condition that the pump beam is tightly focused to optimize the conversion efficiency. The distortion caused by the convolution between the object field and the pump field is ascribed to the point spread function effect. In order to remove the blurring in an upconversion imaging system optimized by tight focused pump, the object is encoded by measurement matrices and the corresponding intensity of the converted field is measured. Thus the intensity distribution of the object can be calculated accurately by the measurements and the measurement matrix. The signal-to-noise ratio (SNR) is improved by employing the Hadamard matrix since the intensity of measured converted signal is far larger than the intensity of each pixel. The experimental results show the proposed method removes the distorting effect caused by the convolution. The converted image still has sharp edges on condition that the conversion efficiency is optimized by tight focusing the pump beam.

Investigation of Mid-Infrared Broadband Second-Harmonic Generation in Non-Oxide Nonlinear Optic Crystals

The mid-infrared (mid-IR) continuum generation based on broadband second harmonic generation (SHG) (or difference frequency generation) is of great interest in a wide range of applications such as free space communications, environmental monitoring, thermal imaging, high-sensitivity metrology, gas sensing, and molecular fingerprint spectroscopy. The second-order nonlinear optic (NLO) crystals have been spotlighted as a material platform for converting the wavelengths of existing lasers into the mid-IR spectral region or for realizing tunable lasers. In particular, the spectral coverage could be extended to ~19 µm with non-oxide NLO crystals. In this paper, we theoretically and numerically investigated the broadband SHG properties of non-oxide mid-IR crystals in three categories: chalcopyrite semiconductors, defect chalcopyrite, and orthorhombic ternary chalcogenides. The technique is based on group velocity matching between interacting waves in addition to birefringent phase matching. We will describe broadband SHG characteristics in terms of beam propagation directions, spectral positions of resonance, effective nonlinearities, spatial walk-offs between interacting beams, and spectral bandwidths. The results will show that the spectral bandwidths of the fundamental wave allowed for broadband SHG to reach several hundreds of nm. The corresponding SH spectral range spans from 1758.58 to 4737.18 nm in the non-oxide crystals considered in this study. Such broadband SHG using short pulse trains can potentially be applied to frequency up-conversion imaging in the mid-IR region, in information transmission, and in nonlinear optical signal processing.

Fast frequency upconversion imaging based on quasi-phase match

Fast frequency upconversion imaging based on quasi-phase match

Low noise frequency upconversion imaging based on Hadamard coding and time-gate detection

The nonlinear frequency upconversion with the tight focused pump beam based on photon counting and Hadamard coding is presented to acquire low noise converted images. The pump beam is optimized by tight focusing to enhance its power density in nonlinear crystal. In order to reduce the distortion caused by the point spread function effect, the object is encoded by measurement matrices and the converted photons corresponding to each pattern is measured. Thus the converted image with sharp edges can be reconstructed by the measurements and the measurement matrices. The time-gated detection is applied to increase the signal-to-noise ratio(SNR) of the measurements. The experimental results show the peak of the dark noise is less than 0.34 photons/spatial/s element for an image with 64 × 64 pixels by taking into account 10 ms measurement time for each pattern. Compared with the converted image reconstructed by the measurements of the photomultiplier tube(PMT) directly, the peak signal to noise ratio (PSNR) of photon counting image increases about 30.91%.

Infrared upconversion imaging in nonlinear metasurfaces

Infrared imaging is a crucial technique in a multitude of applications, including night vision, autonomous vehicles navigation, optical tomography, and food quality control. Conventional infrared imaging technologies, however, require the use of materials like narrow-band gap semiconductors which are sensitive to thermal noise and often require cryogenic cooling. Here, we demonstrate a compact all-optical alternative to perform infrared imaging in a metasurface composed of GaAs semiconductor nanoantennas, using a nonlinear wave-mixing process. We experimentally show the up-conversion of short-wave infrared wavelengths via the coherent parametric process of sum-frequency generation. In this process, an infrared image of a target is mixed inside the metasurface with a strong pump beam, translating the image from infrared to the visible in a nanoscale ultra-thin imaging device. Our results open up new opportunities for the development of compact infrared imaging devices with applications in infrared vision and life sciences.

Orthogonal Reconstruction of Upconversion and Holographic Images for Anticounterfeiting Based on Energy Transfer

Crosstalk-free reconstruction of multiple images within a single element can greatly boost the image capacity and information security. We herein demonstrate a viable approach by integrating upconversion and holographic images into a single holographic polymer nanocomposite. The holographic image is reconstructed through photopolymerization-induced phase separation under a 460 nm laser and identifiable under room light, while the upconversion image recognizable under a 980 nm laser is photopatterned via spatially photobleaching of the dye embedded in the upconversion nanoparticle (UCNP) shell under 365 nm light. To this end, the lanthanide-doped UCNP in the core/shell/shell nanostructure of NaYF4:20%Yb3+,0.5%Tm3+@NaYF4@SiO2 is designed, and the dye, fluorescein isothiocyanate (FITC), is fixed in the outermost SiO2 shell via the amine-isothiocyanate reaction and the subsequent sol-gel reaction. Energy transfer from the core of the UCNP to FITC embedded in the shell is critical to boosting the contrast of the upconversion image, which dials the emission color from blue to yellow-green. It is also found that the upconversion image can be brightened by increasing the UCNP content while the holographic image is weakened when the UCNP content is over 15 wt %. This study paves a new way toward advanced anticounterfeiting.

Controlled synthesis and upconversion luminescence properties of highly uniform and monodisperse β-NaYF4 :Yb3+ ,Tm3+ nanocrystals.

Ytterbium, thulium (Yb3+ ,Tm3+ ) co-doped β-NaYF4 upconversion nanocrystals (U-NCs) were synthesized by a convenient hydrothermal method using sodium citrate as a capping agent. X-ray diffraction (XRD) analysis revealed that NCs phase structures were dependent on reaction time. Field-emission scanning electron microscopy (FE-SEM) and transmission electron microscopy (TEM) results showed that the morphology of U-NCs could be controlled by adjusting the concentration of sodium citrate added in the hydrothermal reaction. Moreover, the emission spectra of the U-NCs were investigated to evaluate upconversion efficiency. Strong luminescence intensity of the U-NCs was observed after tuned with optimized concentration of sodium citrate. Furthermore, the U-NCs were incubated with MCF-7 cells at 37°C for 24 h. Under irradiation of a 980 nm laser, the upconversion blue emission of the NCs in MCF-7 cells can be clearly observed through a confocal microscope with an upconversion imaging system and high quality upconversion luminescence images can be acquired. Thus, prepared β-NaYF4 :Yb3+ ,Tm3+ NCs provide us with highly efficient luminescent probes which can be applied for diverse bio-applications.

Highly Ordered 2D-Assemblies of Phase-Segregated Block Molecules for Upconverted Linearly Polarized Emission.

Materials based on the laminar ordering of self-assembled molecules have a unique potential for applications requiring efficient energy migration through densely packed chromophores. Here, employing molecular assemblies of coil-rod-coil block molecules for triplet-triplet annihilation upconversion (TTA-UC) based on triplet energy migration with linearly polarized emission is reported. By covalently attaching discrete-length oligodimethylsiloxane (oDMS) to 9,10-diphenylanthracene (DPA), highly ordered 2D crystalline DPA sheets separated by oDMS layers are obtained. Transparent films of this material doped with small amounts of triplet sensitizer PtII octaethylporphyrin show air-stable TTA-UC under non-coherent excitation. Upon annealing, an increase in TTA-UC up to two orders of magnitude is observed originating from both an improved molecular ordering of DPA and an increased dispersion of the sensitizer. The molecular alignment in millimeter-sized domains leads to upconverted linearly polarized emission without alignment layers. By using a novel technique, upconversion imaging microscopy, the TTA-UC intensity is spatially resolved on a micrometer scale to visually demonstrate the importance of molecular dispersion of sensitizer molecules for efficient TTA-UC. The reported results are promising for anti-counterfeiting and 3D night-vision applications, but also exemplify the potential of discrete oligodimethylsiloxane functionalized chromophores for highly aligned and densely packed molecular materials.

Multicolor Time‐Resolved Upconversion Imaging by Adiabatic Sum Frequency Conversion

AbstractUpconversion imaging, where mid‐infrared (IR) photons are converted to visible and near‐IR photons via a nonlinear crystal and detected on cheap and high‐performance silicon detectors, is an appealing method to address the limitations of thermal sensors that are expensive, often require cooling, and suffer from both limited spectral response and limited spatial resolution as well as poor sensitivity. However, phase matching severely limits the spectral bandwidth of this technique, therefore requiring serial acquisitions in order to cover a large spectrum. Here, a novel upconversion imaging scheme covering the mid‐IR based on adiabatic frequency conversion is introduced. The study presents mid‐IR multicolor imaging and demonstrates simultaneous imaging on a complementary metal–oxide–semiconductor (CMOS) camera of radiation spanning a spectrum from 2 to 4 µm. This approach being coherent and ultrafast in essence, spectrally resolved spatiotemporal imaging is further demonstrated that allows spatially distinguishing the temporal evolution of spectral components.

A Multimodal Theranostic Nanoformulation That Dramatically Enhances Docetaxel Efficacy Against Castration Resistant Prostate Cancer

In this work, a multifunctional hierarchical nanoformulation composed of biodegradable chitosan (CS) coated poly (lactic-co-glycolic acid) (PLGA) nanocarriers loaded with docetaxel (Doc) and interleukin-8 (IL-8) small interfering RNA (siRNA) electrostatically bound to upconversion nanoparticles (UCNPs), is developed to treat castration-resistant prostate cancer (CRPC). This theranostic nanoformulation facilitates simultaneous delivery of chemotherapy and gene therapy, as well as a bimodal optical and magnetic resonance imaging agent that could enable image-guided combination therapy. Poly-d-lysine coated NaYF4; Yb20%, Er2%@NaYF4; Gd50% core@shell UCNPs are effective siRNA transfection agents, and Er3+ doping provides upconversion imaging capabilities, while Gd3+ doping enables magnetic resonance contrast enhancement. These properties are maintained upon encapsulation in PLGA-CS. PLGA-CS nanocarriers containing Doc and UCNP-siRNA are 235 ± 5 nm with a zeta potential of +17 ± 4 meV, and have a high Doc encapsulation efficiency of 57 ± 6%. Compared to free Doc, this PLGA-CS nanoformulation containing Doc and UCNP-siRNA exhibits a dramatic decrease in IC50 of ~14,000 fold (p < 0.001) through combination therapy in human PC-3 prostate cancer cells. This biocompatible, multimodal, theranostic nanoformulation demonstrates paradigm-shifting enhancement in anticancer activity over free Doc, with unique potential for use in image-guided combination therapy to treat CRPC.

Parametric upconversion of Ince-Gaussian modes.

The Ince-Gaussian (IG) mode, a recently discovered type of structured Gaussian beam, corresponds to eigenfunctions of the paraxial wave equation in elliptical coordinates. This propagation-invariant mode is of significance in various domains, in particular, its nonlinear transformation; however, there have been few relevant studies to date. In this Letter, we report the parametric upconversion of IG modes and associated full-field selection rule for the first time, to the best of our knowledge. We demonstrate that IG signals can be perfectly upconverted by a flattop-beam pump; in contrast, significant mode distortion occurred when using the most common Gaussian pump. Particular attention was given to the origin of the distortion, i.e., radial-mode degeneration induced by the sum-frequency generation excited by a Gaussian pump. This proof-of-principle demonstration has great significance in relevant areas, such as high-dimensional quantum frequency interfacing and upconversion imaging.

Design of Up-Conversion Fluorescence Imaging Device Based on Inverted Fluorescence Microscope

Fluorescence imaging is a widely used technique for microscopic observation in basic biomedical research. Currently, most of the commercial fluorescence microscopes are based on the design of down-conversion fluorescence reagent. In this paper, an up-conversion fluorescence imaging device based on inverted fluorescence microscope is designed. The device uses 980nm near-infrared as the excitation source and the inverted fluorescence microscope as the platform to transform its internal optical system and apply it to up-conversion fluorescence imaging. Through a large number of experimental studies, the stability and linearity of the equipment were verified, and the relationship between incubation time and intake of Hela cells was explored. The experimental results showed that when the laser power was 1. 5W and UCNPs@ DSPE-PE concentration was 0. 05mg/ml, the uptake of Hela cells reached the saturation state.

Upconversion dark-field imaging with extended field of view at video frame rate.

Dark-field imaging is a well-known optical method for obtaining edge-enhanced images of objects containing steep gradients in either amplitude or phase. Edge enhancement is commonly achieved by using a small physical obstruction in the center of the Fourier plane of a 4f imaging setup. By blocking the low spatial frequencies in the center of the Fourier plane, only the higher spatial frequencies from the object reach the image plane. In this work, simultaneous optical image processing (i.e., dark-field imaging) and sum-frequency generation are performed by placing a periodically poled lithium niobate crystal in the Fourier plane of a 4f setup and using a 1575 nm upconversion pump beam with a dark core. We demonstrate upconversion dark-field (UDF) imaging in which edge-enhanced images are obtained at the upconverted wavelength ($\sim{630}\;{\rm nm}$∼630nm) as a result of infrared object illumination ($\sim 1\;\unicode{x00B5}{\rm m}$∼1µm). Furthermore, we experimentally confirm that UDF imaging can be extended from collinear to noncollinear interactions between the signal and pump. The presented system allows for adjustment of the spatial cutoff frequency while maximizing the power conversion efficiency. Simultaneous collinear and noncollinear upconversion imaging of phase objects is demonstrated using broadband 1 µm illumination with a spectral bandwidth of 6 nm-resulting in a UDF imaging system with an enhanced field of view at video frame rates.

Temporal Multiplexed in Vivo Upconversion Imaging.

Upconversion nanoparticles (UCNPs), typically converting near-infrared (NIR) light into visible luminescence, are promising for bioimaging applications. However, optical multiplexed in vivo upconversion experiments have long been hampered by the exceptional rarity of available luminescence bands in UCNPs that can penetrate deep in tissues. Herein, we describe an approach to accomplish multiplexed upconversion in vivo imaging through time-domain discrimination of tissue-penetrating NIR luminescence at 808 nm (from thulium ions) with a multitude of distinct lifetimes. A tetradomain nanostructure design enables one to regulate energy migration and upconverting processes within confined nanoscopic domains in defined ways, thus yielding high quantum yield upconversion luminescence (maximum ≈ 6.1%, 0.11 W/cm2) with precisely controlled lifetimes that span 2 orders of magnitude (from 78 to 2157 μs). Importantly, intravenous and subcutaneous administration of aqueous form UCNPs into a Kunming mouse demonstrates high-contrast lifetime-colored imaging of them in liver and two abdomen subcutis. Moreover, optical patterns of these UCNPs allow multicolour presentation of a series of deciphered images that are hued with precisely defined lifetimes. The described temporal multiplexed upconversion approach, demonstrated in in vivo imaging and multilevel anticounterfeiting, has implications for high-throughput biosensing, volumetric displays, and diagnosis and therapy.