Camera Chip Research Articles

CMOS-level mega-pixel organic camera chips made of photocrosslinked photovoltaic nanocells

CMOS-level mega-pixel organic camera chips made of photocrosslinked photovoltaic nanocells

[formula omitted]: A versatile approach for unsupervised segmentation of calcium imaging data

Recent advances in calcium imaging, including the development of fast and sensitive genetically encoded indicators, high-resolution camera chips for wide-field imaging, and resonant scanning mirrors in laser scanning microscopy, have notably improved the temporal and spatial resolution of functional imaging analysis. Nonetheless, the variability of imaging approaches and brain structures challenges the development of versatile and reliable segmentation methods. Standard techniques, such as manual selection of regions of interest or machine learning solutions, often fall short due to either user bias, non-transferability among systems, or computational demand. To overcome these issues, we developed CalciSeg, a data-driven and reproducible approach for unsupervised functional calcium imaging data segmentation. CalciSeg addresses the challenges associated with brain structure variability and user bias by offering a computationally efficient solution for automatic image segmentation based on two parameters: regions’ size limits and number of refinement iterations. We evaluated CalciSeg efficacy on datasets of varied complexity, different insect species (locusts, bees, and cockroaches), and imaging systems (wide-field, confocal, and multiphoton), showing the robustness and generality of our approach. Finally, the user-friendly nature and open-source availability of CalciSeg facilitate the integration of this algorithm into existing analysis pipelines.

Photovoltaic nanocells for high-performance large-scale-integrated organic phototransistors.

A high-performance large-scale-integrated organic phototransistor needs a semiconductor layer that maintains its photoelectric conversion ability well during high-resolution pixelization. However, lacking a precise design for the nanoscale structure, a trade-off between photoelectric performance and device miniaturization greatly limits the success in commercial application. Here we demonstrate a photovoltaic-nanocell enhancement strategy, which overcomes the trade-off and enables high-performance organic phototransistors at a level beyond large-scale integration. Embedding a core-shell photovoltaic nanocell based on perovskite quantum dots in a photocrosslinkable organic semiconductor, ultralarge-scale-integrated (>221 units) imaging chips are manufactured using photolithography. 27 million pixels are interconnected and the pixel density is 3.1 × 106 units cm-2, at least two orders of magnitude higher than in existing organic imaging chips and equivalent to the latest commercial full-frame complementary metal-oxide-semiconductor camera chips. The embedded photovoltaic nanocells induce an in situ photogating modulation and enable photoresponsivity and detectivity of 6.8 × 106 A W-1 and 1.1 × 1013 Jones (at 1 Hz), respectively, achieving the highest values of organic imaging chips at large-scale or higher integration. In addition, a very-large-scale-integrated (>216 units) stretchable biomimetic retina based on photovoltaic nanocells is manufactured for neuromorphic imaging recognition with not only resolution but also photoresponsivity and power consumption approaching those of the biological counterpart.

Real-time and Intelligent Moving Targets Tracking based on UAV Remote Sensing Video Camera and Brain-like Computing Chips

Abstract. Moving target tracking technology based on Unmanned Aerial Vehicles (UAV) is widely used in many fields such as automatic inspection and emergency response. The existing moving target tracking methods usually have the problems of large computation and low tracking efficiency. Limited by the computing power of the UAV platform, real-time tracking and analysis of multiple targets based on the video data collected by UAV platform is a difficult task. In this paper, we proposed a novel Target Specific Filtering Tracking with Memory (TSFMTrack) method designed for UAV-based real-time tracking tasks, which involves a Tracklet Filtering Module (TFM) for capturing object appearance features and a Tracklet Matching Module (TMM) for bounding box association in each frame. By experimental comparison with other State-Of-The-Art (SOTA) methods on popular MOT and UAV tracking datasets, the TSFMTrack have shown obvious advantages in accuracy, computational efficiency and reliability. Furthermore, we deployed the TSFMTrack on the brain-inspired chip Lynchip KA200, the experimental results have shown that the TSFMTrack is effective on edge computational platform and suitable for UAV real-time tracking tasks.

High-throughput single-molecule localization microscopy: Potential clinical applications.

High-throughput single-molecule localization microscopy: Potential clinical applications.

Integrating IR4.0 Technology in Engineering Education: Robotic Arm Remote Lab

The education sector, in general, is constantly trying to integrate the current fourth industrial revolution known as Industry Revolution 4.0 (IR4.0) into its curriculum. In efforts to do that, this capstone project was assigned to a group of students doing their Bachelor's in Engineering to effectively create and design a virtual or remote lab that would further help students everywhere to be more engaged and involved in the learning process. In this regard, this paper introduces the initiative being carried out by the UOW Malaysia KDU University College (UOWMKDU) to implement IR 4.0 in various modules being taught, in terms of projects or assignments. Therefore, a remote lab was created based on the learning outcomes of a subject available in the module. The lab includes a robotic arm manipulated through an established interface developer Blynk, being simultaneously observed through a website built by a student in the School of Engineering, UOWMKDU. The movement of therobotic arm is viewed through the website by incorporating an inexpensive camera chip into the design. Also, this paper discusses the advantages of using those components, in-depth information regarding the components and their usage, and a detailed methodology that could be used as a guideline for future students interested in creating their robotic arm. It highlights the objectives of the lab, procedures, and challenges that were faced during the period of assembling the prototype and how those challenges were overcome to produce a wellfunctioning robotic arm. Keywords : education sector; IR4.0; remote lab; robotic arm; virtual lab

Ultracompact Orbital Angular Momentum Sorter on a CMOS Chip

On-chip integrated orbital angular momentum (OAM) sorting is of great importance in tackling the severe challenge of exponential growth in data traffic. Despite the continuous success, current demultiplexing techniques either scarify efficiency dramatically or lose the compactness of a system. Here we experimentally demonstrate an ultracompact OAM sorter using TiO2 metasurfaces integrated onto a complementary metal-oxide-semiconductor (CMOS) camera. By utilizing the propagation phases, we transfer the unitary transformation theory in bulky systems into two TiO2 metasurfaces, responsible for the functions of log-polar transformation and fan-out beam copying and focusing as well as the functions of phase correction and Fourier transform. The flatform metasurface doublet enables one to integrate the OAM sorter onto a camera chip. Consequently, OAM beams with topological charges of m = -3 to 3 were separated by a CMOS camera with an average crosstalk of -6.43 dB. This approach shall shed light on next-generation OAM modes processing.

Simultaneous Interference Reflection and Total Internal Reflection Fluorescence Microscopy for Imaging Dynamic Microtubules and Associated Proteins.

Several techniques have been employed for the direct visualization of cytoskeletal filaments and their associated proteins. Total-internal-reflection-fluorescence (TIRF) microscopy has a high signal-to-background ratio, but it suffers from photobleaching and photodamage of the fluorescent proteins. Label-free techniques such as interference reflection microscopy (IRM) and interferometric scattering microscopy (iSCAT) circumvent the problem of photobleaching but cannot readily visualize single molecules. This paper presents a protocol for combining IRM with a commercial TIRF microscope for the simultaneous imaging of microtubule-associated proteins (MAPs) and dynamic microtubules in vitro. This protocol allows for high-speed observation of MAPs interacting with dynamic microtubules. This improves on existing two-color TIRF setups by eliminating both the need for microtubule labeling and the need for several additional optical components, such as a second excitation laser. Both channels are imaged on the same camera chip to avoid image registration and frame synchronization problems. This setup is demonstrated by visualizing single kinesin molecules walking on dynamic microtubules.

Simultaneous Interference Reflection and Total Internal Reflection Fluorescence Microscopy for Imaging Dynamic Microtubules and Associated Proteins

Several techniques have been employed for the direct visualization of cytoskeletal filaments and their associated proteins. Total-internal-reflection-fluorescence (TIRF) microscopy has a high signal-to-background ratio, but it suffers from photobleaching and photodamage of the fluorescent proteins. Label-free techniques such as interference reflection microscopy (IRM) and interferometric scattering microscopy (iSCAT) circumvent the problem of photobleaching but cannot readily visualize single molecules. This paper presents a protocol for combining IRM with a commercial TIRF microscope for the simultaneous imaging of microtubule-associated proteins (MAPs) and dynamic microtubules in vitro. This protocol allows for high-speed observation of MAPs interacting with dynamic microtubules. This improves on existing two-color TIRF setups by eliminating both the need for microtubule labeling and the need for several additional optical components, such as a second excitation laser. Both channels are imaged on the same camera chip to avoid image registration and frame synchronization problems. This setup is demonstrated by visualizing single kinesin molecules walking on dynamic microtubules.

Optical Equivalence Ratio Measurement of a Dual Fuel Burner for Natural Gas and Kerosene

A measurement technique for determination of the global and local equivalence ratios from the flame chemiluminescence for a swirl-stabilized lean premixed combustion of natural gas and kerosene is presented. First, we conducted spectrally resolved chemiluminescence studies using an imaging spectrometer to correlate the ratio of individual chemiluminescence signals to the equivalence ratio. Flame spectra were recorded for a multitude of different lean operating conditions for natural gas and kerosene combustion. The spectra show that, without background correction, the CH*/CO2* ratios for both natural gas and kerosene combustion exhibited a monotonic relationship to the equivalence ratio in the investigated range. Subsequently, bandpass-filtered images of CH* and CO2* chemiluminescence were acquired simultaneously on one camera chip using an image doubler to investigate the local relationship of the CH*/CO2* ratio with the equivalence ratio. The ratio images corroborate the monotonic relationship of the CH*/CO2* ratio to the equivalence ratio. Furthermore, the ratio was found to be influenced by the local reaction zone temperature. The presented technique allows high temporal resolution determination of the local equivalence ratio in lean premixed natural gas and kerosene flames and can thus be applied to quantify equivalence ratio oscillations during unstable combustion.

A Simple Integratable Silicon Photodetector Covering the Short‐Wave Infrared and Optical Communication Bands at 1.3 and 1.55 μm

Silicon photodetectors and camera chips are ubiquitous in the visible region of the spectrum and the near‐infrared. However, their large bandgap prevents their use at longer, technologically very important, wavelengths. The band‐edge‐modified rare‐earth detectors in silicon demonstrated here, fabricated using standard silicon technology tools, offer a route to these wavelengths and room‐temperature operation. Ion implantation of the usual n‐ and p‐type dopants, arsenic and boron, into commercial silicon wafers is used to fabricate a standard silicon p–n junction. Further implants of the rare earths, either Ce or Yb, are made into the depletion region to enable the extended long‐wavelength response. Detectivities up to 1.47 × 108 cm Hz1/2 W−1 and 2.89 × 107 cm Hz1/2 W−1 at 1.3 and 1.55 μm have been obtained with these simple planar devices. If integrated as lateral detectors in a silicon photonics platform, responsivities as good as standard silicon photodetectors in the visible and near‐infrared will be achievable.

Experimental investigation of equivalence ratio fluctuations in a lean premixed kerosene combustor

In this study a measurement technique for determination of equivalence ratio fluctuations from flame chemiluminescence in a kerosene-fuelled lean premixed combustor under atmospheric conditions is presented. Firstly, fundamental investigations into the relationship between the ratio of different chemiluminescence signals and the equivalence ratio are carried out using an imaging spectrometer. The chemiluminescence intensity is recorded for a wide range of equivalence ratios and fuel mass flows during steady state operation. The spectra show that the CH*/OH* ratio depends linearly on the equivalence ratio and is independent of the mass flow in the investigated range. Moreover, the background radiation has no influence on the monotonous trend of the CH*/OH* ratio for kerosene combustion. This interesting finding opens up new possibilities for passive optical measurement of the equivalence ratio in kerosene flames. Bandpass-filtered phase-correlated images of OH* and CH* chemiluminescence of an acoustically excited flame are taken simultaneously on one camera chip using an image doubler. After distortion correction, the image pair is used to calculate the global equivalence ratio from the CH*/OH* ratio. Based on the calibration chart derived in stationary operation, phase-resolved equivalence ratio perturbations are determined during acoustic excitation. The presented technique allows a quantitative measurement of equivalence ratio fluctuations in spray combustion and can therefore provide a better understanding of the fundamental mechanisms of thermoacoustic instabilities triggered by equivalence ratio fluctuations.Graphic abstract

High-fidelity off-axis digital optical phase conjugation with transmission matrix assisted calibration.

The spatial information carried by light is scrambled when it propagates through a scattering medium, such as frosted glass, biological tissue, turbulent air, or multimode optical fibres. Digital optical phase conjugation (DOPC) is a technique that 'pre-aberrates' an illuminating wavefront to compensate for scatterer induced distortion. DOPC systems act as phase-conjugate mirrors: they require a camera to holographically record a distorted wavefront emanating from the scatterer and a spatial light modulator (SLM) to synthesize a phase conjugate of the measured wavefront, which is sent back through the scatterer thus creating a time-reversed copy of the original optical field. High-fidelity DOPC can be technically challenging to achieve as it typically requires pixel-perfect alignment between the camera and SLM. Here we describe a DOPC system in which the normally stringent alignment criteria are relaxed. In our system the SLM and camera are placed in-line in the same optical path from the sample, and the SLM is used in an off-axis configuration. This means high-precision alignment can be achieved by measurement of the transmission matrix (TM) mapping optical fields from the SLM to the camera and vice-versa, irrespective of their relative position. The TM also absorbs and removes other aberrations in the optical system, such as the curvature of the SLM and camera chips. Using our system we demonstrate high-fidelity focussing of light through two ground glass diffusers with a peak-intensity to mean-background ratio of ∼700. We provide a step-by-step guide detailing how to align this system and discuss the trade-offs with alternative configurations. We also describe how our setup can be used as a 'single-pixel camera' based DOPC system, offering potential for DOPC at wavelengths in which cameras are not available or are prohibitively expensive.

Geodesic Length Measurement in Medical Images: Effect of the Discretization by the Camera Chip and Quantitative Assessment of Error Reduction Methods

After interventions such as bypass surgeries the vascular function is checked qualitatively and remotely by observing the blood dynamics inside the vessel via Fluorescence Angiography. This state-of-the-art method has to be improved by introducing a quantitatively measured blood flow. Previous approaches show that the measured blood flow cannot be easily calibrated against a gold standard reference. In order to systematically address the possible sources of error, we investigated the error in geodesic length measurement caused by spatial discretization on the camera chip. We used an in-silico vessel segmentation model based on mathematical functions as a ground truth for the length of vessel-like anatomical structures in the continuous space. Discretization errors for the chosen models were determined in a typical magnitude of 6%. Since this length error would propagate to an unacceptable error in blood flow measurement, counteractions need to be developed. Therefore, different methods for the centerline extraction and spatial interpolation have been tested and compared against their performance in reducing the discretization error in length measurement by re-continualization. In conclusion, the discretization error is reduced by the re-continualization of the centerline to an acceptable range. The discretization error is dependent on the complexity of the centerline and this dependency is also reduced. Thereby the centerline extraction by erosion in combination with the piecewise Bézier curve fitting performs best by reducing the error to 2.7% with an acceptable computational time.

Adding Flexible Instrumentation to a Curved Videolaryngoscope: A Novel Tool for Laryngeal Surgery.

Transoral surgery of the larynx with rigid instruments is not always possible. This may result in insufficient therapy or in an increased need for open surgery. For these patients, alternative surgical systems are needed. Here, we demonstrate a curved prototype for laryngeal surgery equipped with flexible instruments. Pre-clinical user study in an ex vivo porcine laryngeal model. The prototype was built from established medical devices, namely a hyperangulated videolaryngoscope and modified flexible instruments as well as three-dimensional printed parts. Feasibility of laryngeal manipulation was evaluated in a user study (n = 19) with a porcine ex vivo laryngeal model. Using three different visualization technologies, the participants performed various fine motor skills tasks and rated the usability of the system on a 5-point Likert scale. Exposure, accessibility, and manipulation of important laryngeal structures were always possible using the new prototype. The participants needed considerably less time (mean, 96.4 seconds ± 6.4 seconds vs. 111.5 seconds ± 4.5 seconds, P = .18), reported significantly better general impression (mean score 3.0 vs. 3.8, P = .041) and significantly lower user head and neck strain (2.6 vs. 1.7, P = .022) using a 40-inch television screen as compared to a standard videolaryngoscope monitor. The results indicate that our curved prototype and large monitor visualization may provide a cost-effective minimally invasive alternative for difficult laryngeal exposure. Its special advantages include avoiding the need for a straight line of sight and a simple and cost-effective construction. The system could be further improved through advances in camera chip technology and smaller instruments. Laryngoscope, 131:E561-E568, 2021.

A single-beam 3DoF homodyne interferometer

In this paper, a camera-based 3DoF homodyne interferometer is proposed to measure displacements and rotations with precision below nanometre/microradian level. The interferometer uses a single beam, has a simple and compact layout and is easy to adjust. A spatial interference pattern is introduced by a tilted reference mirror and recorded by a camera chip. The variation of the spacing and orientation of the fringes is correlated with the yaw and pitch angular rotation of the moving mirror, while the phase correlates with its displacement. The noise level and resolution of displacement and angle measurements were determined to be below 5 pm/√Hz and 5 nrad/√Hz above 1 Hz. Periodic nonlinearities of the system were mainly caused by multi-reflections between optical surfaces in the system. By choosing particular components and improving their adjustment, the displacement nonlinearity was reduced to be below 0.1 nm, and angle nonlinearities achieved sub-microradian level.

Optical range determination of clinical proton beams in water. A comparison with standard measurement methods

As recently discovered, water emits a weak luminescence when it is irradiated with protons even with energies below the Cerenkov light threshold. In this work it was investigated if this phenomenon could be exploited for range measurements in proton therapy. A measurement setup based on a scientific CMOS camera that can be operated under normal room light was built and tested in a proof-of-principle experiment at the West German Proton Therapy Center, Essen. The luminescence depth profiles were analyzed to obtain the range information and the method was compared with ionization chamber based depth dose measurements. The noise caused by scattered radiation hitting the camera chip could be removed with a simple threshold-based median filter. The influence of Cerenkov radiation produced by delta electrons was analyzed by FLUKA simulations and it was shown that it does not affect the range measurements. It could be shown that the luminescence method is as fast as the multi-layer ionization chamber measurement (a few seconds) but with a higher depth resolution that is comparable with the Bragg peak chamber method. The proton ranges determined with the luminescence method agree with the reference methods better than 0.2% over the whole energy range 100–226MeV. The sensitivity of the method regarding detectable range shifts was tested. It was shown, that energy shifts of 0.5MeV (at 151MeV), leading to a range shift of ∼0.9mm, were clearly detectable.

Spikes to Pixels: Camera Chips for Large-scale Electrophysiology

Implanted neural probes are among the most important techniques in both fundamental and clinical neuroscience. Despite great successes and promise, neural electrodes are technically limited by their scalability. A recent study by Obaid et al. demonstrated an innovative way to greatly scale up the channel count and density of neural electrode arrays.

Massively parallel microwire arrays integrated with CMOS chips for neural recording.

Multi-channel electrical recordings of neural activity in the brain is an increasingly powerful method revealing new aspects of neural communication, computation, and prosthetics. However, while planar silicon-based CMOS devices in conventional electronics scale rapidly, neural interface devices have not kept pace. Here, we present a new strategy to interface silicon-based chips with three-dimensional microwire arrays, providing the link between rapidly-developing electronics and high density neural interfaces. The system consists of a bundle of microwires mated to large-scale microelectrode arrays, such as camera chips. This system has excellent recording performance, demonstrated via single unit and local-field potential recordings in isolated retina and in the motor cortex or striatum of awake moving mice. The modular design enables a variety of microwire types and sizes to be integrated with different types of pixel arrays, connecting the rapid progress of commercial multiplexing, digitisation and data acquisition hardware together with a three-dimensional neural interface.

Parallel Fourier ptychographic microscopy for high-throughput screening with 96 cameras (96 Eyes)

We report the implementation of a parallel microscopy system (96 Eyes) that is capable of simultaneous imaging of all wells on a 96-well plate. The optical system consists of 96 microscopy units, where each unit is made out of a four element objective, made through a molded injection process, and a low cost CMOS camera chip. By illuminating the sample with angle varying light and applying Fourier Ptychography, we can improve the effective brightfield imaging numerical aperture of the objectives from 0.23 to 0.3, and extend the depth of field from ±5 μm to ±15 μm. The use of Fourier Ptychography additionally allows us to computationally correct the objectives’ aberrations out of the rendered images, and provides us with the ability to render phase images. The 96 Eyes acquires raw data at a rate of 0.7 frame per second (all wells) and the data are processed with 4 cores of graphical processing units (GPUs; GK210, Nvidia Tesla K80, USA). The system is also capable of fluorescence imaging (excitation = 465 nm, emission = 510 nm) at the native resolution of the objectives. We demonstrate the capability of this system by imaging S1P1-eGFP-Human bone osteosarcoma epithelial (U2OS) cells.