Spectral Camera Research Articles

High-Quality Multispectral Image Reconstruction for the Spectral Camera Based on Ghost Imaging via Sparsity Constraints Using CoT-Unet

To solve the problem of poor quality in ghost imaging via sparsity constraints (GISC) multispectral image reconstruction with correlation operations and compressed sensing algorithms under low sampling rate detection conditions, we propose an end-to-end deep-learning-based method. Based on the U-Net, Res2Net-SE-Conv is employed instead of convolutional blocks to extract local and global image features at a more fine-grained level while adaptively adjusting the channel feature response. The two-dimensional contextual transformer is constructed to fully use contextual correlation information to enhance the effectiveness of feature representations. We employ the two-dimensional contextual transformer in the decoder part, dubbed CoT-Unet, to reconstruct the desired 3D cube. The results show that compared with U-Net, TSA-Net based on spatial-spectral self-attention, the PSNR of reconstructed images by the CoT-Unet is improved by 5 dB and 3 dB, respectively, SSIM is improved by 0.23 and 0.07, and SAM is decreased by 0.06 and 0.58. Compared with conventional algorithms such as DGI and CS, our method significantly improves the quality of reconstructed images. Furthermore, the comparison results at 10%, 20%, and 30% sampling rates show that our approach has the best quality in reconstructing GISC multispectral images at low sampling rates.

Fully non-invasive measurement of protein content in soybean based on spectral characteristics of the pod

Cultivation and breeding of legumes as protein sources in the human or animal diet could benefit from accurate, rapid, and non-invasive measurements of protein content. A study was conducted into the feasibility of a fully non-invasive, in vivo protein measurement methodology applied to soybean (Glycine max. L.). The proposed methodology works by recording spectral images of the soybean pods in the visible and near-infrared (Vis-NIR), a rule-based segmentation approach, and partial least squares (PLS) regression to predict the crude protein content of the beans contained within the imaged pods. Using all 150 channels of the spectral camera, a model could be calibrated with a mean absolute precision error (MAPE) of 4.8 % (R2 = 0.92). Applying a tailored feature elimination approach to select only eight spectral bands and degrading the spectral resolution to 25 nm yields a model with a MAPE of 6.0 % (R2 = 0.88), indicating the potential for multispectral cameras in this application.

Multimodal close range hyperspectral imaging combined with multiblock sequential predictive modelling for fresh produce analysis

Multimodal measurements are increasingly becoming common in the domain of spectral sensing and imaging for fresh produce. Often multiple sensors are expected to carry complementary information which allows precise estimation of responses. In this study, a novel case of multimodal hyperspectral imaging is described where two different spectral cameras working in the complementary spectral ranges were integrated into a fully standalone system for spectral imaging for fresh produce analysis. Furthermore, a comparative analysis of different multiblock predictive modelling approaches for fusing data from these two complementary spectral cameras is demonstrated. Both multiblock latent space and multiblock variable selection approaches to identify key variables of interest was examined and compared with the analysis carried out on individual data blocks. Prediction of the soluble solids content in grapes was used to demonstrate the application. The presented approach can increase the applications of multimodal hyperspectral imaging for non-destructive analysis.

Compact and ultracompact spectral imagers: technology and applications in biomedical imaging.

Spectral imaging, which includes hyperspectral and multispectral imaging, can provide images in numerous wavelength bands within and beyond the visible light spectrum. Emerging technologies that enable compact, portable spectral imaging cameras can facilitate new applications in biomedical imaging. With this review paper, researchers will (1)understand the technological trends of upcoming spectral cameras, (2)understand new specific applications that portable spectral imaging unlocked, and (3)evaluate proper spectral imaging systems for their specific applications. We performed a comprehensive literature review in three databases (Scopus, PubMed, and Web of Science). We included only fully realized systems with definable dimensions. To best accommodate many different definitions of "compact," we included a table of dimensions and weights for systems that met our definition. There is a wide variety of contributions from industry, academic, and hobbyist spaces. A variety of new engineering approaches, such as Fabry-Perot interferometers, spectrally resolved detector array (mosaic array), microelectro-mechanical systems, 3D printing, light-emitting diodes, and smartphones, were used in the construction of compact spectral imaging cameras. In bioimaging applications, these compact devices were used for in vivo and ex vivo diagnosis and surgical settings. Compact and ultracompact spectral imagers are the future of spectral imaging systems. Researchers in the bioimaging fields are building systems that are low-cost, fast in acquisition time, and mobile enough to be handheld.

Improved U-Net3+ With Spatial–Spectral Transformer for Multispectral Image Reconstruction

Multispectral image reconstruction, which aims to recover a three-dimensional (3D) spatial–spectral signal from a two-dimensional measurement in a spectral camera based on ghost imaging via sparsity constraint (GISC), has been attracting much attention recently. However, faced with abundant 3D spectral data, the reconstruction quality cannot meet the visual requirements. Based on the robust data processing capability of deep learning, a novel network called SSTU-Net3+ is constructed by improving U-Net3+ with a spatial–spectral transformer (SST). To enhance the feature representation of images during reconstruction, mixed pooling modules and new convolution processes are proposed to improve the performance of the encoder and decoder, with U-Net3+ as the backbone. To boost the quality of reconstructed images, with split and concatenate (Concat) operations, we construct SST modules by exploiting both spatial and spectral correlations of multispectral images to refine the spatial and spectral features. Furthermore, we employ the SST in the decoder to reconstruct the desired 3D cube. Given similar network parameters, experiments on GISC spectral imaging data show that, compared to convolutional neural network–based methods, the average peak signal-to-noise ratio of images reconstructed using SSTU-Net3+ is improved by 3%, the structural similarity is enhanced by 3%, and the spectral angle mapping is cut by 12%. Particularly, compared to differential ghost imaging and compressed sensing, the reconstruction quality of SSTU-Net3+ has been significantly improved. SSTU-Net3+ can process a large amount of 3D multispectral image data more efficiently and construct the target image more accurately than the abovementioned methods.

Local polarization properties extraction using single incident state, single-mode-fiber-based spectral domain polarization-sensitive optical coherence tomography.

We showed the local polarization properties extraction method for the single incident state, all-single-mode-fiber-based spectral domain polarization-sensitive optical coherence tomography (SD-PS-OCT) system that uses the single linear-in-wavenumber spectral camera. Polarization controllers are used in the single-mode-fiber-based SD-PS-OCT system to provide a compact structure with polarization state stability. The local polarization properties of the birefringent sample are extracted from the cumulative polarization properties iteratively. The reconstructed polarization images demonstrate the local polarization properties extraction ability of the system.

Evaluation of the potential of near infrared hyperspectral imaging for monitoring the invasive brown marmorated stink bug

The brown marmorated stink bug (BMSB), Halyomorpha halys, is an invasive insect pest of global importance that damages several crops, compromising agri-food production. Field monitoring procedures are fundamental to perform risk assessment operations, in order to promptly face crop infestations and avoid economical losses. To improve pest management, spectral cameras mounted on Unmanned Aerial Vehicles (UAVs) and other Internet of Things (IoT) devices, such as smart traps or unmanned ground vehicles, could be used as an innovative technology allowing fast, efficient and real-time monitoring of insect infestations.The present study consists in a preliminary evaluation at the laboratory level of Near Infrared Hyperspectral Imaging (NIR-HSI) as a possible technology to detect BMSB specimens on different vegetal backgrounds, overcoming the problem of BMSB mimicry. Hyperspectral images of BMSB were acquired in the 980–1660 nm range, considering different vegetal backgrounds selected to mimic a real field application scene. Classification models were obtained following two different chemometric approaches. The first approach was focused on modelling spectral information and selecting relevant spectral regions for discrimination by means of sparse-based variable selection coupled with Soft Partial Least Squares Discriminant Analysis (s-Soft PLS-DA) classification algorithm. The second approach was based on modelling spatial and spectral features contained in the hyperspectral images using Convolutional Neural Networks (CNN). Finally, to further improve BMSB detection ability, the two strategies were merged, considering only the spectral regions selected by s-Soft PLS-DA for CNN modelling.

Hyperspectral Multispectral Image Fusion via Fast Matrix Truncated Singular Value Decomposition

Recently, methods for obtaining a high spatial resolution hyperspectral image (HR-HSI) by fusing a low spatial resolution hyperspectral image (LR-HSI) and high spatial resolution multispectral image (HR-MSI) have become increasingly popular. However, most fusion methods require knowing the point spread function (PSF) or the spectral response function (SRF) in advance, which are uncertain and thus limit the practicability of these fusion methods. To solve this problem, we propose a fast fusion method based on the matrix truncated singular value decomposition (FTMSVD) without using the SRF, in which our first finding about the similarity between the HR-HSI and HR-MSI is utilized after matrix truncated singular value decomposition (TMSVD). We tested the FTMSVD method on two simulated data sets, Pavia University and CAVE, and a real data set wherein the remote sensing images are generated by two different spectral cameras, Sentinel 2 and Hyperion. The advantages of FTMSVD method are demonstrated by the experimental results for all data sets. Compared with the state-of-the-art non-blind methods, our proposed method can achieve more effective fusion results while reducing the fusing time to less than 1% of such methods; moreover, our proposed method can improve the PSNR value by up to 16 dB compared with the state-of-the-art blind methods.

The Glacial Meltwater Turbidity Algorithm (GaMTA): Adaptation of single-band algorithm retrieving turbidity to satellite and UAV dataset from highly glaciated Antarctic region

The growing need for marine polar observations, especially using remote sensing data, necessitates the adaptation of existing algorithms to local conditions. Five satellite imageries acquired by MSI and OLI sensors from 2020 and 2021 depicting Admiralty Bay in King George Island and sixteen multispectral maps created using data from Unoccupied Aerial Vehicles (UAV) flights with a spectral camera over the cove near Zalewski Glacier were analysed. Data were supplemented by field measurements of turbidity. The Glacial Meltwater Turbidity Algorithm (GaMTA) was proposed and compared with well-known semianalytical, single-band algorithms for turbidity retrieval based on spectral imageries of heavily glaciated areas. Comparative analysis showed the ineffectiveness of algorithms applied to medium-resolution satellite images in the area of small glacial coves with high and spatially varying surface water turbidity (R = 0.59-0.61, RMSE = 4.45-7.46), in contrast to open water with low turbidity (R = 0.79-0.88, RMSE = 2.61-3.08). The results from the use of high-resolution UAV data showed the applicability of this algorithm to aerial images (R = 0.96-0.97, RMSE = 5.75-7.57). The use of GaMTA, which is adapted to local conditions, produces better results than the other algorithms, but validation on a larger dataset is needed.

Space debris spectroscopy: Specular reflections at LEO regime

Space debris, the dysfunctional human-made objects on Low Earth Orbits (LEO), occasionally irradiate bright specular flares or glints which can be detected by a ground-observer. These events are reflections of sunlight from flat, highly reflective surface elements and their spectra represent the ability of the material to reflect sunlight in specific regions of wavelengths. Spectral features then define satellite’s exterior properties, mainly the colour of the surface material. Presented here are the reduction techniques and reflectance spectra of LEO space debris acquired by slit-less spectral cameras AMOS-Spec–HR. Cameras are a part of world-wide distributed network All-sky Meteor Orbit System(AMOS), designed for meteor observation, and operated by the Comenius University in Bratislava. During spectroscopic observations from 2016 to 2021, AMOS-Spec–HR harnessed numerous debris recordings and by data mining of its archives, dozens of spectra of flares and glints were retrieved. According to their properties, reflectance spectra can be organized into distinct categories: Category I – Convex Spectra (includes spectra of solar panels), Category II – Spectra Increasing towards Red Wavelength Region (common for spectra of golden polyimide Kapton foil, the surface material of Multi-Layer Insulation), and Category III – Flat, Straight-lined Spectra (typically includes spectra of aluminium, or other silvery alloys and metals). Defined spectral types were confronted and correlated with laboratory spectra of reflective aerospace materials obtained from literature.

Coffee Crops Analysis Using UAVs Equipped with Multispectral Cameras

The use of Unmanned Aerial Vehicles (UAVs) equipped with spectral cameras has increased in recent years, especially in the agricultural sector, because it allows farmers and researchers to analyze the state of a crop, i.e., health, nutrients, growth, epidemics, among other parameters. In Colombia, the coffee production sector faces several challenges, such as the need to increase the productivity, the yield, and the quality of coffee. This work estimated the health status of a Castilla variety crop located in San Joaquín, Tambo, Cauca to support the decision-making of coffee growers. For this, chlorophyll data were measured in the field with the CCM-200 plus device, multispectral images were captured with the MAPIR SURVEY 3 camera airborne on a SOLO 3DR UAV, and synthetic data were generated to increase the data set. Six vegetation indices were set, which—together with the chlorophyll values—were modeled through the implementation of simple and multiple linear regressions, decision trees, vector machines, random forests, and k-nearest neighbors. The model with the best performance and the lowest mean square error was disorder with the support vector machine. Likewise, the best performance indices in the models were CVI, GNDVI, and GCI, which are widely used in agriculture to estimate the chlorophyll of plants.

Research on the deep learning-based exposure invariant spectral reconstruction method.

The surface spectral reflectance of an object is the key factor for high-fidelity color reproduction and material analysis, and spectral acquisition is the basis of its applications. Based on the theoretical imaging model of a digital camera, the spectral reflectance of any pixels in the image can be obtained through spectral reconstruction technology. This technology can avoid the application limitations of spectral cameras in open scenarios and obtain high spatial resolution multispectral images. However, the current spectral reconstruction algorithms are sensitive to the exposure variant of the test images. That is, when the exposure of the test image is different from that of the training image, the reconstructed spectral curve of the test object will deviate from the real spectral to varying degrees, which will lead to the spectral data of the target object being accurately reconstructed. This article proposes an optimized method for spectral reconstruction based on data augmentation and attention mechanisms using the current deep learning-based spectral reconstruction framework. The proposed method is exposure invariant and will adapt to the open environment in which the light is easily changed and the illumination is non-uniform. Thus, the robustness and reconstruction accuracy of the spectral reconstruction model in practical applications are improved. The experiments show that the proposed method can accurately reconstruct the shape of the spectral reflectance curve of the test object under different test exposure levels. And the spectral reconstruction error of our method at different exposure levels is significantly lower than that of the existing methods, which verifies the proposed method's effectiveness and superiority.

Crosstalk elimination by rearranging thin-film filters.

Patterned thin-film Fabry-Pérot filters are used to develop compact spectral cameras. Recent articles report on crosstalk in such devices, raising concerns regarding spectral and spatial resolution. It has been suggested that light entering a filter might spill over to neighboring filters but this has not yet been analyzed in detail. The proposed mechanism in this Letter is that the Fabry-Pérot filters act as coupled waveguides that can propagate crosstalk above the pixel array. The results show that the crosstalk can be asymmetric, enabling elimination by rearranging the filters on the sensor. Interestingly, the logical strategy where crosstalk to all neighbors is eliminated appears suboptimal due to additional resonances. These findings reveal untapped opportunities for developing better sensors and a corresponding need for further systematic investigations.

A survey on computational spectral reconstruction methods from RGB to hyperspectral imaging

Hyperspectral imaging enables many versatile applications for its competence in capturing abundant spatial and spectral information, which is crucial for identifying substances. However, the devices for acquiring hyperspectral images are typically expensive and very complicated, hindering the promotion of their application in consumer electronics, such as daily food inspection and point-of-care medical screening, etc. Recently, many computational spectral imaging methods have been proposed by directly reconstructing the hyperspectral information from widely available RGB images. These reconstruction methods can exclude the usage of burdensome spectral camera hardware while keeping a high spectral resolution and imaging performance. We present a thorough investigation of more than 25 state-of-the-art spectral reconstruction methods which are categorized as prior-based and data-driven methods. Simulations on open-source datasets show that prior-based methods are more suitable for rare data situations, while data-driven methods can unleash the full potential of deep learning in big data cases. We have identified current challenges faced by those methods (e.g., loss function, spectral accuracy, data generalization) and summarized a few trends for future work. With the rapid expansion in datasets and the advent of more advanced neural networks, learnable methods with fine feature representation abilities are very promising. This comprehensive review can serve as a fruitful reference source for peer researchers, thus paving the way for the development of computational hyperspectral imaging.

Assessment of Injury by Four Major Pests in Soybean Plants Using Hyperspectral Proximal Imaging

Arthropod pests are among the major problems in soybean production and regular field sampling is required as a basis for decision-making for control. However, traditional sampling methods are laborious and time-consuming. Therefore, our goal is to evaluate hyperspectral remote sensing as a tool to establish reflectance patterns from soybean plants infested by various densities of two species of stinkbugs (Euschistus heros and Diceraeus melacanthus (Hemiptera: Pentatomidae)) and two species of caterpillars (Spodoptera eridania and Chrysodeixis includens (Lepidoptera: Noctuidae)). Bioassays were carried out in greenhouses with potted plants placed in cages with 5 plants infested with 0, 2, 5, and 10 insects. Plants were classified according to their reflectance, based on the acquisition of spectral data before and after infestation, using a hyperspectral push-broom spectral camera. Infestation by stinkbugs did not cause significative differences in the reflectance patterns of infested or non-infested plants. In contrast, caterpillars caused changes in the reflectance patterns, which were classified using a deep-learning approach based on a multilayer perceptron artificial neural network. High accuracies were achieved when the models classified low (0 + 2) or high (5 + 10) infestation and presence or absence of insects. This study provides an initial assessment to apply a non-invasive detection method to monitor caterpillars in soybean before causing economic damage.

Zoom optical system design for acousto-optic tunable filter camera assisted by programming multidimensional analysis.

A medium infrared 12× continuous zoom optical system serving for an acousto-optic tunable filter spectral camera has been designed covering a 25-300 mm zooming range. Instead of relying on past experience for roughly determining the initial optical structure, a simulation programming based on Gaussian principle has been composed via MATLAB to accurately calculate the initial designing parameters, which is confirmed to be extremely close to the optimized results using Zemax. The relative design results have been multidimensionally analyzed in detail, which offers fresh thinking for future zoom optical design covering broadband operation wave band.

A survey on IoT-based smart agriculture to reduce vegetable and fruit waste

Agriculture automation is a top focus and developing area for a number of countries right now. We are seeing a surge in demand for Internet of Things (IoT) in various industries these days. One of the most essential applications of IoT is agriculture. Today, we notice that the world’s population is quickly rising, and an agro product plays a critical part in this population’s existence. We are conscious of the fact that resources are limited. If we continue to farm in the traditional manner, it will be extremely difficult for the rising population to survive. Due to improper fertilizers, quantity of water, chemicals and huge amount of pesticides decreases the fertility of land. There is a need for smart agriculture to monitor all these factors which affect the fertility of soil. When the fertility is decreasing it will impact on the growth of fruits and vegetables. In this paper we are going to study how we can manage our resources through Internet of Things, multispectral camera, hyper spectral camera, and thermal camera and RGB camera. Plant diseases, pesticide control, weed control, proper irrigation, and water management are all problems in agriculture that can be readily solved with the various automated and control approaches stated above.

Chlorophyll detector development based on snapshot-mosaic multispectral image sensing and field wheat canopy processing

To achieve rapid nondestructive detection of chlorophyll content in crops in a field environment, we designed a new lightweight device based on a snapshot-mosaic imaging sensor given the sensitive properties of chlorophyll in the red-edge to near-infrared band (700–900 nm) and the advantages of imaging spectroscopy in environmental noise separation. The hardware part of the device includes a spectral camera based on a snapshot-mosaic sensor, a main control unit, a network module, a storage module, a power supply module, and a display and remote-control device. The software part was written using the Qt library and the C++ programming language. It includes the connection and initialization of the sensor, the control of the exposure time, the display of the image in real time, the acquisition and storage, and the calculation of the reflectance. The sensor was tested and calibrated to evaluate the sensor performance. Experimental results show that the different channels of the sensor have an excellent linear relationship for light intensity changes and can be used to measure reflected radiation from crop leaves in a field environment after calibration. At the same time, field experiments on wheat were conducted. A dark channel filtering method for 25 channels of the developed device was proposed to effectively eliminate the scattered light interference from the crop canopy in the spectral images. The background interference was effectively eliminated by background segmentation. Partial least squares regression (PLSR) method was used for modeling after preprocessing and band screening of the spectral data. The model achieved high accuracy with RC2 of 0.87, RV2 of 0.79, and RMSE of 3.94 mg/L. As a result, the system combined the proposed model with the developed device can effectively eliminate the interference of soil background and scattered light caused by the complex structure of crop canopy, and improve the accuracy of diagnosis. So that the system has potential in the crop growth variation analysis.

A low-dispersion spectral video camera for observing lunar impact flashes

An impact of a meteoroid on the lunar surface at speeds exceeding several kilometers per second generates a light flash generally less than 0.1 s in duration. We made a simple spectral video camera for observing the lunar impact flashes and monitored the waxing crescent Moon’s non-sunlit surface from Oct. 2016 to May 2017. We detected ten flash candidates though there was no report of simultaneous detections by other observers. We obtained low-dispersion spectra in visible wavelengths for nine of them. Six of them show spectra similar to those of the flashes observed during the Geminids meteor activity in Dec. 2018 by the same camera. The spectra are continuous and red. Blackbody spectra fitted to them show temperatures around 3000 K. On the other hand, three of them show continuous blue spectra. Blackbody spectra fitted to them show temperatures of more than 6000 K. Specular reflection of sunlight by space debris might lead to these flashes. However, the impact of a low-density meteoroid not against the fine lunar regolith but solid lunar rocks could cause blue flashes. In this paper, we give full details of the camera and the analytical procedures of the videos. We also discuss recommendations for future spectral observations.Graphical

Design and Geometrical Optimization of Nozzle to Reduce Noise for Jet Engine

A scientific payload is used for deep space observations. It consists of several components, which are used to observe and detect the objects and its details. Such payloads are used to collect data for tin-filer research and understanding about the universe and to know its hidden mysteries. This research is conducted using several observatory instruments like, X-ray spectroscope, hyper spectral image camera, laser ranging instruments, etc. In this project. a mechanical configuration of X-ray spectroscope is designed considering all the design parameters and constraints. Boeing recently tested a scaled variable area jet nozzle capable of a 20% area change Shape Memory Alloy actuators were used to position 12 interlocking panels at the nozzle exit. A closed loop control system was used to maintain a range of constant diameters with varying flow conditions and to vary the diameter under constant flow conditions. Acoustic data by sideline microphones and flow field measurements at several cross-sections using PIV was collected at each condition. In this paper the variable area nozzle's design is described. The effect of the nozzle's diameter on acoustic performance is presented flow field data is shown including the effects of the joints between the interlocking panels.