Real-time Multispectral Imaging Research Articles

Spectral imaging enables contrast agent–free real-time ischemia monitoring in laparoscopic surgery

Laparoscopic surgery has evolved as a key technique for cancer diagnosis and therapy. While characterization of the tissue perfusion is crucial in various procedures, such as partial nephrectomy, doing so by means of visual inspection remains highly challenging. We developed a laparoscopic real-time multispectral imaging system featuring a compact and lightweight multispectral camera and the possibility to complement the conventional surgical view of the patient with functional information at a video rate of 25 Hz. To enable contrast agent-free ischemia monitoring during laparoscopic partial nephrectomy, we phrase the problem of ischemia detection as an out-of-distribution detection problem that does not rely on data from any other patient and uses an ensemble of invertible neural networks at its core. An in-human trial demonstrates the feasibility of our approach and highlights the potential of spectral imaging combined with advanced deep learning-based analysis tools for fast, efficient, reliable, and safe functional laparoscopic imaging.

Fish Capsules: A System for High-Throughput Screening of Combinatorial Drugs.

Large‐scale screening of molecules heavily relies on phenotyping of small living organisms during preclinical development. However, deep profiling candidate therapeutics on whole animals typically requires laborious manipulations and anesthetic treatment using traditional techniques or automated tools. Here, a novel fish capsule system that combines automated zebrafish encapsulating technology and droplet microarray strategy for in vivo functional screening of mono/polytherapies is described. This platform enables automated, rapid zebrafish orientation and immobilization in agarose to generate large‐scale fish capsules by using a microfluidic device. Based on the effect of discontinuous dewetting, the prompt trapping of fish capsules in the aqueous arrays is successfully demonstrate. This system provides the capability to integrate pharmaceutical treatments with real‐time multispectral microscopic imaging in a simple, pipetting‐free and highly parallel manner. Coupling with machine learning algorithms, a small library of compounds is screened and analyzed, and clues about how to exploit compound combinations as therapeutic candidates are obtained. It is believed that this proposed strategy can be readily applied to multiple fields and is especially useful in the exploration of combinatorial drugs with limited amounts of samples and resources to accelerate the identification of novel therapeutics for precision medicines.

Using physiological parameters measured by hyperspectral imaging to detect colorectal cancer.

The accurate detection of malignant tissue during colorectal surgery impacts operation outcome. The non-invasive spectral imaging combined with machine learning (ML) methods showed to be promising for tumor identification. However, large spectral range implies large computing time. To reduce the number of features, ML methods (e.g. logistic regression and convolutional neuronal network CNN) were evaluated based on four physiological tissue parameters to automatically classify cancer and healthy mucosa in resected colon tissue. A ROC AUC of 0.81 was achieved with the CNN. This study shows that the use of only specific wavelengths bands can detect cancer.Clinical Relevance- These outcomes support the possibility to automatically classify colon tumor based on physiological parameters calculated using only specific wavelength bands. Hence, future image-guided colorectal surgeries can be performed with real-time multispectral imaging.

Biomimetic multispectral curved compound eye camera for real-time multispectral imaging in an ultra-large field of view.

In this work, we demonstrate a prototype of a biomimetic multispectral curved compound eye camera (BMCCEC). In comparison with traditional multispectral imaging systems, the BMCCEC developed in this work has the distinct features of multi-spectral imaging on multiple targets in real time in an ultra-large field of view (FOV), which can be attributed to its biomimetic curved compound eye structure as well as the multispectral cluster network. Specifically, the BMCCEC has a total of 104 multispectral ommatidia and a FOV of 98°×98°, which is able to realize 7-band multispectral imaging with center wavelengths of 500 nm, 560 nm, 600 nm, 650 nm, 700 nm, 750 nm and 800 nm and a spectral resolution of 10 nm. A prototype of BMCCEC was then manufactured and multispectral imaging experiments were performed based on it. As a result, the red edge feature of the spectrum of green plants has been successfully obtained and retrieved with a good accuracy.

Ex vivo validation of a real-time multispectral endoscopic system for the detection and biopsy of bladder tumors.

BackgroundReal-time multispectral imaging (rMSI) simultaneously provides white light (WL), photodynamic diagnosis (PDD) images, and a real-time fusion of both. It may improve the detection of bladder tumors. However, rMSI has not been used for transurethral biopsy or resection so far. The aim of this ex vivo study was to test the feasibility of bladder tumor biopsies using the rMSI system and compare it to a conventional endoscopic system.MethodsA 3D printed rigid bladder phantom was equipped with small and flat (5 mm × 1 mm) mock-bladder-tumors made of silicone and fluorescent Qdots655 (Thermo Fisher Scientific, Germany). Urologists (n=15) were asked to perform a rigid cystoscopy and biopsy of all identified lesions (n=6) using a prototype rMSI system and the Image1 S system (Karl Storz, Tuttlingen). Success rate and completion time were measured. The image quality of both systems and the usability of the rMSI system according to the system usability scale (SUS) were evaluated with a task-specific questionnaire.ResultsTumor detection and biopsy rate were 100% (90/90) for the rMSI system and 98.9% (89/90) for the Image1 S system (P=0.3). The biopsy completion time did not differ significantly between the systems (P=0.48). Differentiation between healthy and suspect mucosa with the rMSI system was rated as comparable to the Image1 S system by 53% of surgeons and as better by 33% of the surgeons. The median SUS score for the rMSI system was 87.5%.ConclusionsAccurate transurethral biopsies are feasible with the rMSI system. Furthermore, the rMSI system has an excellent SUS. This study paves the way to the first in-human transurethral resections of bladder tumors (TUR-B) using rMSI technology.

Self-referencing biosensors using Fano resonance in periodic aluminium nanostructures.

Surface plasmon resonance (SPR) is an important technique for real-time and label-free detection of specific binding biomolecules. However, conventional SPR signals come from both the surface binding biomolecules and the variation in the bulk refractive index. This work demonstrates that Fano resonance in an aluminum capped nanoslit array has the ability to remove the signal of bulk refractive index changes from the SPR signal. As compared to gold nanostructures, the aluminum nanostructure provides an asymmetrical Fano resonance with clear peak and dip wavelengths. The peak wavelength is close to the grating resonance condition. The evanescent depth at the peak wavelength is up to several microns. The dip wavelength comes from the SPR effect. The evanescent depth at the dip wavelength is about 300 nm. By simultaneously measuring the shifts of peaks and the dip wavelengths, the variation in the bulk refractive index can be removed and only the biolayer thickness is measured. The finite-difference time-domain calculation shows that the 470 nm-period nanoslit array with 90 and 70 nm slit depths has the optimal thickness sensitivity. In this experiment, a simple multispectral imaging system is developed for multiple bio-interaction measurements. The measured results verify that the bulk refractive index changes can be removed and the surface biomolecular interactions can be directly obtained without the need of a reference channel.

Establishment of Real-Time Multispectral Imaging for the Detection of Bladder Cancer Using a Preclinical in Vivo Model

BACKGROUND: Emerging imaging technologies such as real-time multispectral imaging (rMSI) hold great potential for simultaneous visualization of multiple target structures using fluorophores on various tumours including bladder cancer (BC). These technologies, however, require a multi-step preclinical evaluation process, including mouse models. OBJECTIVE: To demonstrate the suitability of the new rMSI technology for the detection of premalignant lesions and malignant BC in a preclinical mouse model using contrast agents. METHODS: Tumours were induced by N-butyl-N-(4-hydroxybutyl)-nitrosamine (BBN), which is known to induce BC in rodent models. In total, 30 mice (C57BL/6) were fed with 0.1% BBN ad libitum in drinking water for up to 5 months. Bladders were excised at 3 (n = 6) and 5 months (n = 24) of treatment and incubated ex vivo with Hexaminolevulinat (HAL, Hexvix®), CD47-FITC, CD90.2-FITC or a combination of CD90.2-FITC/CD47-FITC and HAL. The bladders were analyzed by an endoscopic rMSI prototype system equipped with a spectral filter (Chroma), a 4 mm endoscope (Karl Storz) with 30° optic, a LED light source and a PC with a microcontroller board. RESULTS: 5-month treatment of mice with 0.1% BBN led to the formation of squamous carcinoma (46%, n = 11) while urothelial carcinoma was observed only in one mouse (4%, n = 1). Carcinoma in situ (CIS) was detectable in twelve out of twenty-four mice (50%, n = 12) treated for 5 month and in three out of six mice (50%, n = 3) treated for 3 months.The metabolite of HAL, protoporphyrin IX (PpIX), could be reliably and specifically detected in all of mouse bladder tumours and CIS. However, detection of the CD90.2 surface marker was less reliable, potentially due to species- or tumour-subtype specificity. CONCLUSIONS: This model offers the potential for preclinical imaging studies with combined fluorescence targets, e.g. HAL, in combination with BC-specific antibodies.

Real-time, wide-field and high-quality single snapshot imaging of optical properties with profile correction using deep learning.

The development of real-time, wide-field and quantitative diffuse optical imaging methods to visualize functional and structural biomarkers of living tissues is a pressing need for numerous clinical applications including image-guided surgery. In this context, Spatial Frequency Domain Imaging (SFDI) is an attractive method allowing for the fast estimation of optical properties using the Single Snapshot of Optical Properties (SSOP) approach. Herein, we present a novel implementation of SSOP based on a combination of deep learning network at the filtering stage and Graphics Processing Units (GPU) capable of simultaneous high visual quality image reconstruction, surface profile correction and accurate optical property (OP) extraction in real-time across large fields of view. In the most optimal implementation, the presented methodology demonstrates megapixel profile-corrected OP imaging with results comparable to that of profile-corrected SFDI, with a processing time of 18 ms and errors relative to SFDI method less than 10% in both profilometry and profile-corrected OPs. This novel processing framework lays the foundation for real-time multispectral quantitative diffuse optical imaging for surgical guidance and healthcare applications. All code and data used for this work is publicly available at www.healthphotonics.org under the resources tab.

Real-time multispectral fluorescence lifetime imaging using Single Photon Avalanche Diode arrays

Autofluorescence spectroscopy has emerged in recent years as a powerful tool to report label-free contrast between normal and diseased tissues, both in vivo and ex vivo. We report the development of an instrument employing Single Photon Avalanche Diode (SPAD) arrays to realize real-time multispectral autofluorescence lifetime imaging at a macroscopic scale using handheld single-point fibre optic probes, under bright background conditions. At the detection end, the fluorescence signal is passed through a transmission grating and both spectral and temporal information are encoded in the SPAD array. This configuration allows interrogation in the spectral range of interest in real time. Spatial information is provided by an external camera together with a guiding beam that provides a visual reference that is tracked in real-time. Through fast image processing and data analysis, fluorescence lifetime maps are augmented on white light images to provide feedback of the measurements in real-time. We validate and demonstrate the practicality of this technique in the reference fluorophores and in articular cartilage samples mimicking the degradation that occurs in osteoarthritis. Our results demonstrate that SPADs together with fibre probes can offer means to report autofluorescence spectral and lifetime contrast in real-time and thus are suitable candidates for in situ tissue diagnostics.

Spatial–Spectral–Temporal Framework for Emotion Recognition

An emotion recognition method based on multispectral imaging technology and tissue oxygen saturation (StO2) is proposed in this study. This method is called spatial-spectral-temporal adjustment convolutional neural network (SACNN). First, we use the algorithm to extract the StO2 content of an emotionally sensitive nose area through real-time multispectral imaging technology. Compared with facial expression data, StO2 data are more objective and cannot be controlled and changed artificially. Second, we construct a clustering algorithm based on the emotional state by extracting the spectral, StO2, and spatial features of the nose image to obtain accurate signals of emotionally sensitive areas. To utilize the correlation between spectral and spatial signals, we propose an adjustment-based CNN module, which reorganizes the relationship between all previous layers of the feature map, thereby making the relationship among layers close and highly quantitative. The features extracted through this method are consistent with spatial-spectral features. Third, we incorporate the extracted temporal feature signal into the long short-term memory module and finally complete the correlation between the spatial-spectral-temporal features. Experimental results show that the accuracy of the SACNN algorithm in emotional recognition reaches 90%, and the proposed method is more competitive than state-of-the-art approaches. To the best of our knowledge, this study is the first to use time-series StO2 signals for emotion recognition.

Tunable Narrow-bandpass Filter Using Blue Phase Liquid Crystal Etalon for Real-time Multi-spectral Imaging Systems

Tunable Narrow-bandpass Filter Using Blue Phase Liquid Crystal Etalon for Real-time Multi-spectral Imaging Systems

Tunable Narrow-bandpass Filter Using Blue Phase Liquid Crystal Etalon for Real-time Multi-spectral Imaging Systems

Tunable Narrow-bandpass Filter Using Blue Phase Liquid Crystal Etalon for Real-time Multi-spectral Imaging Systems

Multiparametric Cystoscopy for Detection of Bladder Cancer Using Real-time Multispectral Imaging

Background: Various imaging modalities can be used in addition to white light (WL) to improve detection of bladder cancer (BC). Objective: To use real-time multispectral imaging (rMSI) during urethrocystoscopy to combine different imaging modalities to achieve multiparametric cystoscopy (MPC). Design, setting, and participants: The rMSI system consisted of a camera with a spectral filter, a multi-LED light source, a microcontroller, and a computer for display and data acquisition. MSI with this system was achieved via temporal multiplexing. Surgical procedure: MPC was performed in ten patients with a diagnosed bladder tumor. Measurements: We gathered evidence to prove the feasibility of our approach. In addition, experienced urologists performed post-interventional evaluation of images of individual lesions. Images were independently rated in a semiquantitative manner for each modality. A statistical model was built for pairwise comparisons across modalities. Results and limitations: Overall, 31 lesions were detected using the rMSI set-up. Histopathology revealed malignancy in 27 lesions. All lesions could be visualized simultaneously in five modalities: WL, enhanced vascular contrast (EVC), blue light fluorescence, protoporphyrin IX fluorescence, and autofluorescence. EVC and photodynamic diagnosis images were merged in real time into one MP image. Using the recorded images, two observers identified all malignant lesions via MPC, whereas the single modalities did not arouse substantial suspicion for some lesions. The MP images of malignant lesions were rated significantly more suspicious than the images from single imaging modalities. Conclusions: We demonstrated for the first time the application of rMSI in endourology and we established MPC for detection of BC. This approach allows existing imaging modalities to be combined, and it may significantly improve the detection of bladder cancer. Patient summary: Real-time multispectral imaging was successfully used to combine different imaging aids for more comprehensive illustration of bladder tumors for surgeons. In the future, this technique may allow better detection of bladder tumors and more complete endoscopic resection in cases of cancer.

The development of real-time multispectral imaging for the diagnostics of bladder cancer

The performance of white light (WL) cystoscopy in the diagnostics of bladder cancer can be optimized by the use of modern imaging modalities, such as photodynamic diagnostics (PDD) and narrow band imaging (NBI). Real-time multispectral imaging (rMSI) enables simultaneous imaging of reflectance and fluorescence modalities in multiple spectral bands. We created amultiparametric cystoscopy image by digital overlapping of several modalities, e.g. WL, enhanced vascular contrast (EVC), raw fluorescence mode, protoporphyrinIX and autofluorescence (AF). The technical development and the subsequent clinical implementation of rMSI required astructured preclinical evaluation process, including both ex vivo and in vivo trials before the technology can be applied in patients. This review article presents the phases of testing, validation and the first clinical application of rMSI in urological endoscopy.

Towards real-time multispectral endoscopic imaging for cardiac lesion quality assessment.

Atrial fibrillation (Afib) can lead to life threatening conditions such as heart failure and stroke. During Afib treatment, clinicians aim to repress unusual electrical activity by electrically isolating the pulmonary veins (PV) from the left atrium (LA) using radiofrequency ablation. However, current clinical tools are limited in reliably assessing transmurality of the ablation lesions and detecting the presence of gaps within ablation lines, which can warrant repeat procedures. In this study, we developed an endoscopic multispectral reflectance imaging (eMSI) system for enhanced discrimination of tissue treatment at the PV junction. The system enables direct visualization of cardiac lesions through an endoscope at acquisition rates up to 25 Hz. Five narrowband, high-power LEDs were used to illuminate the sample (450, 530, 625, 810 and 940nm) and combinatory parameters were calculated based on their relative reflectance. A stitching algorithm was employed to generate large field-of-view, multispectral mosaics of the ablated PV junction from individual eMSI images. A total of 79 lesions from 15 swine hearts were imaged, ex vivo. Statistical analysis of the acquired five spectral data sets and ratiometric maps revealed significant differences between transmural lesions, non-transmural lesions around the venoatrial junctions, unablated posterior wall of left atrium tissue, and pulmonary vein (p < 0.0001). A pixel-based quadratic discriminant analysis classifier was applied to distinguish four tissue types: PV, untreated LA, non-transmural and transmural lesions. We demonstrate tissue type classification accuracies of 80.2% and 92.1% for non-transmural and transmural lesions, and 95.0% and 92.8% for PV and untreated LA sites, respectively. These findings showcase the potential of eMSI for lesion validation and may help to improve AFib treatment efficacy.

PT213 - Development of a rodent model for preclinical evaluation of multiple contrast agents and real-time multispectral imaging in bladder cancer

PT213 - Development of a rodent model for preclinical evaluation of multiple contrast agents and real-time multispectral imaging in bladder cancer

Performance analysis of LAPAN-A3 data acquisition using X-band receiver at Rancabungur ground station

As an experimental remote sensing microsatellite, LAPAN-A3/IPB satellite mission produces at least 10 GB payload data daily, mainly comes from multispectral imager payload with the addition of digital matrix imager, Automatic Identification System (AIS) and scientific earth magnetometer. To improve the satellite data reception capability, LAPAN had already installed dual X-band and S-band receiver on Rancabungur ground station. This research aims to analyze the performance of LAPAN-A3 payload data reception by using X-band receiver from Rancabungur ground station. The assessment consists of determining daily payload data transmission capability and realtime multispectral image acquisition performance, as well as distortion occurs on the received data caused by signal interferences. Based on the first four months of the X-band receiver operation, it is found that the receiver has similar data reception performance compare to the X-band receiver on Kongsberg ground station currently leased for LAPAN-A3 operation, i.e. receiving about 7 GB payload data in high elevation satellite pass. The X-band receiver could also receive realtime multispectral image acquisition when the satellite passes western part of Indonesia in the morning, producing up to 10 GB of multispectral image in single satellite pass. However, the data reception performance often degraded by signal interferences from ground station neighborhood, both quantitative and qualitatively. Based on these performances, a better LAPAN-A3 satellite mission planning can be developed and systematic interferences detection can be done based on the received data. This allows LAPAN-A3 satellite to produce more payload data with better quality.

Conceptual design and proof-of-principle testing of the real-time multispectral imaging system MANTIS

The Multispectral Advanced Narrowband Tokamak Imaging System (MANTIS) is proposed to resolve the steep temperature and density gradients in the scrape-off layer of tokamaks in real-time. The initial design is to deliver two-dimensional distributions of key plasma parameters of the TCV tokamak to a real-time control system in order to enable novel control strategies, while providing new insights into power exhaust physics in the full offline analysis. This paper presents the conceptual system design, the mechanical and optical design of a prototype that was built to assess the optical performance, and the results of the first proof-of-principle tests of the prototype. These demonstrate a central resolving power of 50-46 line pairs per millimeter (CTF50) in the first four channels. For the additional channels, the sharpness is a factor two worse for the odd channels (likely affected by sub-optimal alignment), while the even channels continue the trend observed for the first four channels of 3% degradation per channel. This is explained by the self-cancellation of off-axis aberrations, which is an attractive property of the chosen optical design. The results show that at least a 10-channel real-time multispectral imaging system is feasible.

用于实时微型多光谱荧光成像方法的二维点阵式多通道窄带滤镜

用于实时微型多光谱荧光成像方法的二维点阵式多通道窄带滤镜

基于多通道窄带滤光片和彩色图像传感器的实时多光谱成像系统

基于多通道窄带滤光片和彩色图像传感器的实时多光谱成像系统