Line Scan Sensor Research Articles

On-the-go table grape ripeness estimation via proximal snapshot hyperspectral imaging

The monitoring of grapes for ripeness estimation is a practice that enables fruit harvesting at the optimal time. Hyperspectral Imaging (HSI) represents a non-destructive and high-throughput alternative to traditional laboratory analyses. Current literature approaches perform hyperspectral measurements using line scan sensors or low-resolution static snapshot cameras, which hinder a fast per-bunch ripeness characterization. We propose a framework for on-the-go collection and processing of proximal snapshot hyperspectral images to estimate single bunch ripeness parameters. Focusing on table grapes (Vitis vinifera L. cv. Red Globe), we collected images under natural illumination with a hyperspectral camera (500–900 nm) mounted on a moving vehicle in an experimental block sited in Piacenza, Italy. We investigated images collected in August and September 2021 representing two ripening stages. The composition of the imaged grape bunches was determined through laboratory chemical analyses to predict Total Soluble Solids (TSS) and anthocyanin concentration. The images were pre-processed via multimodal image registration to correct the unalignment of bands due to the vehicle motion, and the single bunches were automatically identified on false RGB images through a Mask Region-Convolutional Neural Network (Mask R-CNN) instance segmentation network. The mean spectra of the bunches were used as input features of a Partial Least Squares Regression (PLSR) model to predict the chemical parameters at single bunch and whole vine scales. The regression model of TSS had an R2 (10-fold nested cross-validation) of 0.75 and 0.85 on a per-bunch and per-vine basis, respectively. The regression model of anthocyanin had an R2 of 0.68 and 0.49 on a per-bunch and per-vine basis, respectively. The results suggest the potential of using snapshot hyperspectral images for high-throughput analysis of a per-bunch grape ripeness estimation. The method described in this study could give valuable information to improve grape ripening monitoring and management of harvest operations and even allow for precise and automated robotic harvesting.

Development and validation of a 64-channel ROIC prototype for SWIR line scan sensor applications

This paper introduces the development and validation of a 64-channel readout integrated circuit (ROIC) prototype, specifically engineered for short-wave infrared (SWIR) line scan sensors. The design of the prototype undergoes various evaluation through comprehensive silicon-level testing, ensuring its robust performance across a variety of operational modes. Key features such as capacitive transimpedance amplifier (CTIA) gain control and sensitivity control are examined, demonstrating the prototype's ability to handle different input currents and capacitance values with precision. Fabricated with 0.18-μm CMOS technology, the ROIC is tailored for integration with Indium Gallium Arsenide (InGaAs) pixels, facilitating high-resolution imaging. The prototype consumes 26.55 mW with A 3.3 V power supply. The fabricated chip show that the total random noise (RN) level is 128 μVrms and column fixed pattern noise (FPN) is 0.16 mVrms

Performance evaluation of CNN and R-CNN based line by line analysis algorithms for fibre placement defect classification

The Automated Fibre Placement process is commonly used in aerospace for the manufacturing of structural components, but requires a subsequent inspection to meet the corresponding safety requirements. In order to improve this mostly manual inspection step, machine learning methods for the interpretation of 2D surface images are being increasingly utilised in research. Depending on the manufacturing process, a very long time can elapse between the appearance of a manufacturing defect and its recognition. Hence, in this paper Convolutional and Recurrent Neural Network techniques are presented that allow a line-by-line analysis of the incoming height profile scans of a Laser Line Scan Sensor as a 1D signal, which enables a direct reaction to a defect, even if only one or a few individual height profiles of the defect have been recorded. The combination of Convolutional and Recurrent Neural Network structures is particularly beneficial for this application. The investigations in this paper are especially interesting for developers of automated inspection systems in composite engineering.

Contaminant detection using a CZT photon counting detector with TDI image reconstruction

Food x-ray inspection systems are designed to detect unwanted physical contaminants in packaged food to maintain a high level of food safety for consumers. Modern day x-ray inspection systems often utilize line scan sensors to detect these physical contaminants but are limited to single or dual energies. However, by using a photon counting detector (PCD), a new generation of food inspection systems capable of acquiring images at more than two energy bins could improve discrimination between low density contaminants. In this work, five type of contaminants were embedded in an acrylic phantom and imaged using a cadmium zinc telluride (CZT) PCD with a pixel pitch of 330 μm. A set of images were acquired while the phantom was stationary, and another set of images were acquired while the phantom was moving to mimic the movement of a conveyor belt. Image quality was assessed by evaluating the contrast-to-noise ratio (CNR) for each set of images. For imaging times larger than 25 ms, the results showed that the moving phantom data set yielded larger CNR values compared to a stationary phantom. While conventional x-ray inspections often utilize line scan sensors, we report that physical contaminant detection is possible with a CZT PCD x-ray imaging system.

In-process adaptive dimension correction strategy for laser aided additive manufacturing using laser line scanning

Additive manufacturing (AM) technologies have seen rapid growth in the past decade. Achieving high-quality consistency and accuracy remains a challenge in the fabrication of large-format metallic parts using the directed energy deposition AM processes. An efficient dimension correction strategy is required to prevent build failure during the AM process. In this paper, a laser line scanning sensor was integrated into a robot-based laser aided additive manufacturing (LAAM) system to realise the on-machine measurement of the part geometry. With this system, an in-process adaptive dimension correction strategy was proposed. The dimensional deviations in the intermediate layers could be corrected immediately after they were detected during the LAAM process, thus avoiding excessive dimensional deviation leading to build failure. A tool-path generation process for dimension correction was proposed which did not rely on traditional time-consuming CAD reconstruction. Only 3D point cloud was used directly, enabling the quick response of the LAAM system in restoring the dimensional accuracy. The deviated surface could be automatically filled up, and the subsequent deposition processes were resumed after each cycle of the dimension correction. To facilitate the proposed dimension correction strategy, a Robot Operating System (ROS)-based software platform was developed. Experimental comparisons between the part built with and without correction were conducted. The results demonstrated a significant improvement in dimensional accuracy when the correction strategy was applied.

RGB-D fusion models for construction and demolition waste detection

The development of urbanization has brought a large amount of construction and demolition waste (CDW), which occupy land and cause adverse ecological effects. To effectively solve the negative impact of CDW, it needs to be recycled. Accurate waste classification is key to successful waste management. However, the current waste classification methods mainly use color images to classify, which cannot meet the needs of accurate classification. This paper built an RGB-depth (RGB-D) detection platform, using a color camera and a laser line-scanning sensor to collect RGB images and depth images. In order to use RGB images and depth images for feature fusion more effectively, this paper proposed three fusion models: RGB-D concat、RGB-D Ci-add and RGB-D Ci-concat. All these models based on an instance segmentation network called mask region convolutional neural network (Mask R-CNN), which can accurately segment the contours of each object while classifying them. The experimental results show that the mAPs of the RGB-D Ci-add / concat model are 1.33% to 1.72% higher than those of the RGB model, and the classification accuracy is 1.92% ∼ 2.27% higher.In addition, all the proposed models can meet the real-time requirement of online detection. Due to the excellent comprehensive performance of the RGB-D Ci-concat model, it can be regarded as the final detection model of the robot, which can improve the sorting efficiency of CDW further.

Optical Material Characterisation of Prepreg CFRP for Improved Composite Inspection

Automated fibre layup techniques are often applied for the production of complex structural components. In order to ensure a sufficient component quality, a subsequent visual inspection is necessary, especially in the aerospace industry. The use of automated optical inspection systems can reduce the inspection effort by up to 50 %. Laser line scan sensors, which capture the topology of the surface, are particularly advantageous for this purpose. These sensors project a laser beam at an angle onto the surface and detect its position via a camera. The optical properties of the observed surface potentially have a great influence on the quality of the recorded data. This is especially relevant for dark or highly scattering materials such as Carbon Fiber Reinforced Plastics (CFRP). For this reason, in this study we investigate the optical reflection and transmission properties of the commonly used Hexel HexPly 8552 IM7 prepreg CFRP in detail. Therefore, we utilise a Gonioreflectometer to investigate such optical characteristics of the material with respect to different fibre orientations, illumination directions and detection angles. In this way, specific scattering information of the material in the hemispherical space are recorded. The major novelty of this research are the findings about the scattering behaviour of the fibre composite material which can be used as a more precise input for the methods of image data quality assessment from our previous research and thus is particularly valuable for developers and users of camera based inspection systems for CFRP components.

Synthetic image data augmentation for fibre layup inspection processes: Techniques to enhance the data set

In the aerospace industry, the Automated Fiber Placement process is an established method for producing composite parts. Nowadays the required visual inspection, subsequent to this process, typically takes up to 50% of the total manufacturing time and the inspection quality strongly depends on the inspector. A Deep Learning based classification of manufacturing defects is a possibility to improve the process efficiency and accuracy. However, these techniques require several hundreds or thousands of training data samples. Acquiring this huge amount of data is difficult and time consuming in a real world manufacturing process. Thus, an approach for augmenting a smaller number of defect images for the training of a neural network classifier is presented. Five traditional methods and eight deep learning approaches are theoretically assessed according to the literature. The selected conditional Deep Convolutional Generative Adversarial Network and Geometrical Transformation techniques are investigated in detail, with regard to the diversity and realism of the synthetic images. Between 22 and 166 laser line scan sensor images per defect class from six common fiber placement inspection cases are utilised for tests. The GAN-Train GAN-Test method was applied for the validation. The studies demonstrated that a conditional Deep Convolutional Generative Adversarial Network combined with a previous Geometrical Transformation is well suited to generate a large realistic data set from less than 50 actual input images. The presented network architecture and the associated training weights can serve as a basis for applying the demonstrated approach to other fibre layup inspection images.

Surface Texture Measurement on Complex Geometry Using Dual-Scan Positioning Strategy

In this paper, a surface measurement method based on dual-scan positioning strategy is presented to address the challenges of irregular surface patterns and complex geometries. A confocal sensor with an internal scanning mechanism was used in this study. By synchronizing the local scan, enabled by the internal actuator in the confocal sensor, and the global scans, enabled by external positioners, the developed system was able to perform noncontact line scan and area scan. Thus, this system was able to measure both surface roughness and surface uniformity. Unlike laboratory surface measurement equipment, the proposed system is reconfigurable for in situ measurement and able to scan free-form surfaces with a proper stand-off distance and approaching angle. For long-travel line scan, which is needed for rough surfaces, a surface form tracing algorithm was developed to ensure that the data were always captured within the sensing range of the confocal sensor. It was experimentally verified that in a scanning length of 100 mm, where the surface fluctuation in vertical direction is around 10 mm, the system was able to perform accurate surface measurement. For area scan, XY coordinates provided by the lateral positioning system and the Z coordinate captured by the confocal sensor were plotted into one coordinate system for 3D reconstruction. A coherence scanning interferometer and a confocal microscope were employed as the reference measurement systems to verify the performance of the proposed system in a scanning area of 1 mm by 1 mm. Experimental data showed that the proposed system was able to achieve comparable accuracy with laboratory systems. The measurement deviation was within 0.1 µm. Because line scan mechanisms are widely used in sensor design, the presented work can be generalized to expand the applications of line scan sensors.

Multi-Sensor Mapping for Low Contrast, Quasi-Dynamic, Large Objects

This paper proposes a systems level solution for addressing the problem of mapping large moving targets with slow but complicated dynamics. Our approach utilizes the complementary nature of multiple sensing modalities. While this work is applicable to other domains we focus our efforts on mapping rotating and translating icebergs. Our solution involves a rigidly coupled combination of a line scan sensor - a subsurface multibeam sonar, with an area scan sensor - an optical camera. This allows the system to exploit the optical camera information to perform iceberg relative navigation which can directly be used by the multibeam sonar to map the iceberg underwater. This paper details the algorithm required to compute the scale of the navigation solution and corrections to find iceberg centric navigation and thus an accurate iceberg reconstruction. This approach is successfully demonstrated on real world iceberg data collected during the 2018 Sermilik campaign in Eastern Greenland. Due to the availability of iceberg mounted GPS observations during this research expedition we could also groundtruth our navigation and thus our systems level mapping efforts.

Interferometer-in-Spectrometer for High-Resolution Optical Coherence Tomography

Ultrahigh-resolution optical coherence tomography provides an axial resolution of 1-2 μm for resolving cellular structures of biological tissues critical for the diagnosis of diseases. However, it requires a relatively large spectral bandwidth which is not supported by the key components of the imaging system. We propose a novel spectral-domain OCT design, termed interferometer-in-spectrometer, which is able to compensate the bandwidth limitations of the grating and the line scan sensor by spectral shaping without compromising the signal intensity and adding the system cost. The advantage of axial resolution and ranging depth over the standard design is experimentally validated using the standard testing method and fresh swine cornea ex vivo. Moreover, opportunities that opened up by this new scheme for improving the performances of spectral-domain OCT are also discussed.

A novel approach for sensor based sorting performance determination

Sensor-based sorters execute an “accept-reject” task to selectively remove the target particles from a material stream. Depending on the intended use, defining the separation performance of a Sensor-based sorting (SBS) operation has critical importance on evaluation of the output product qualities. SBS systems are dominated by line-scan sensors, within which image acquisition via machine vision is practicable as the material is sensed during transportation on a belt or during free fall. Current favorable tools for performance evaluation use numerous parameters such as particle size, throughput, mass % to be rejected, accurate identification of the particles, and even particle shape to determine efficiency, performance, and accuracy of the sorting process in mining/mineral processing industry. However, performance of the operation is also affected by mechanical and physical factors along with the correct identification of materials. Hence, it is necessary to ensure and monitor that all accurately identified particles are reported to the correct product stream, either accept or reject. The aim of this study is to investigate the performance of a real-time Near-infrared (NIR)-sorter with a novel perspective, and compare the results with favorable and known previous approaches. Within the experimental studies, accept and reject products of an industrial scale NIR-sorter used for borate sorting were collected, and sieved to three distinct size fractions. Images of individual particles were acquired with and without optical filters, and boron contents were analyzed, respectively. Resulting data was used to determine the material specific partition coefficients of the sorting operation for different particle size classes. Combined overall and actual separations reported fairly average figures of 0.105 Ep (Ecart probability) and 0.730 RV50 (cut-reflectance value) for the separation. Evaluation of the overall findings showed that presented approach execute reliable performance figures applicable to ore sorting applications.

Hyperspectral imaging for textile sorting in the visible–near infrared range

Recycling of textile materials is becoming important due to the increasing amount of textile waste and its large environmental impact. The Resyntex project aims at dealing with this textile waste by enabling its chemical recycling. To do so, pure textile materials and blends need to be sorted first. In this paper we evaluate the suitability of hyperspectral imaging for pure and blend textile sorting. We also test the discrimination capacity between denim and non-denim textile, since this is required prior to the de-colouration processes. For this purpose, we use a line-scan sensor in the 450–950 nm range, since its cost, compactness and speed characteristics make it suitable for industrial deployment. To deal with the strong colour interference of the textile a hierarchical classification approach is proposed. The results on the available sample set show promising discrimination potential for material discrimination as well as for denim versus non-denim detection.

Focus on analytical equipment, sensors, and instrumentation

Focus on analytical equipment, sensors, and instrumentation

A Multi-View Stereo Measurement System Based on a Laser Scanner for Fine Workpieces.

A new solution to the high-quality 3D reverse modeling problem of complex surfaces for fine workpieces is presented using a laser line-scanning sensor. Due to registration errors, measurement errors, deformations, etc., a fast and accurate method is important in machine vision measurement. This paper builds a convenient and economic multi-view stereo (MVS) measurement system based on a linear stage and a rotary stage to reconstruct the measured object surface completely and accurately. In the proposed technique, the linear stage is used to generate the trigger signal and synchronize the laser sensor scanning; the rotary stage is used to rotate the object and obtain multi-view point cloud data, and then the multi-view point cloud data are registered and integrated into a 3D model. The measurement results show a measurement accuracy of 0.075 mm for a 360° reconstruction in 34 s, and some evaluation experiments were carried out to demonstrate the validity and practicability of the proposed technique.

Flexible calibration method for line-scan cameras using a stereo target with hollow stripes

With the advantages of ultrahigh resolution and acquisition rate, stereo line-scan sensors are gradually being developed for three-dimensional (3D) measurements. Calibration is crucial for line-scan cameras. However, viewing targets of line-scan cameras from different angles cause eccentricity errors. This paper presents a flexible calibration method for line-scan cameras to solve the eccentricity error problem. The calibration quality is improved by the accurate extraction of one-dimensional (1D) points on line-scan images, and the corresponding 3D points in space. Through the analysis of the eccentricity error in 1D image point extraction when utilizing targets with solid stripes, a stereo target with hollow stripes is designed. An algorithm is described to calculate the 3D points of intersection between the centerlines of hollow lines on the target and the viewing plane of the camera. A robust optimization is proposed to enhance the stability of parameter estimation. Experiments are performed to compare the re-projection error between the cross-ratio-based method and proposed method. A simulation is also performed to estimate the combined uncertainty. The comparative results of the experiments verify that the proposed method is accurate, low cost, and easy to implement.

Improved spatial resolution of luminescence images acquired with a silicon line scanning camera

Luminescence imaging is currently being used to provide spatially resolved defect in high volume silicon solar cell production. One option to obtain the high throughput required for on the fly detection is the use a silicon line scan cameras. However, when using a silicon based camera, the spatial resolution is reduced as a result of the weakly absorbed light scattering within the camera's chip. This paper address this issue by applying deconvolution from a measured point spread function. This paper extends the methods for determining the point spread function of a silicon area camera to a line scan camera with charge transfer. The improvement in resolution is quantified in the Fourier domain and in spatial domain on an image of a multicrystalline silicon brick. It is found that light spreading beyond the active sensor area is significant in line scan sensors, but can be corrected for through normalization of the point spread function. The application of this method improves the raw data, allowing effective detection of the spatial resolution of defects in manufacturing.

Stereo line-scan sensor calibration for 3D shape measurement

The stereo line-scan sensor opens up new potentialities for 3D measurement owing to the ultra-high resolution and acquisition rate. Calibration is a crucial key technology for a stereo line-scan sensor. This paper presents a precise calibration method for the stereo line-scan sensor. Several reference points are installed onto the sensor's body as an intermediary. The calibration turns into a two-step process: calibrating the cameras in the laboratory prior to measurement and locating the sensor in an actual measurement field. A mobile apparatus that comprises a planar pattern and extra reference points is designed. By incorporating the apparatus in combination with an auxiliary instrument, an optimal calibration configuration is created by placing the apparatus into multiple positions. A robust algorithm is proposed to enhance the stability of the parameter estimation. The quality of the calibration method is experimentally tested, and the performance is further investigated. Experimental results demonstrate that the proposed method offers a practical solution to calibrate a stereo line-scan sensor for 3D shape measurement.

Line-scan system for continuous hand authentication

An increasing number of heavy machinery and vehicles have come into service, giving rise to a significant concern over protecting these high-security systems from misuse. Conventionally, authentication performed merely at the initial login may not be sufficient for detecting intruders throughout the operating session. To address this critical security flaw, a line-scan continuous hand authentication system with the appearance of an operating rod is proposed. Given that the operating rod is occupied throughout the operating period, it can be a possible solution for unobtrusively recording the personal characteristics for continuous monitoring. The ergonomics in the physiological and psychological aspects are fully considered. Under the shape constraints, a highly integrated line-scan sensor, a controller unit, and a gear motor with encoder are utilized. This system is suitable for both the desktop and embedded platforms with a universal serial bus interface. The volume of the proposed system is smaller than 15% of current multispectral area-based camera systems. Based on experiments on a database with 4000 images from 200 volunteers, a competitive equal error rate of 0.1179% is achieved, which is far more accurate than the state-of-the-art continuous authentication systems using other modalities.

A Dual Image Sensor Approach for Automated, High Resolution, Region-of-Interest Imaging in a 96-Well Plate

Phenotypic screens are performed on model organisms to understand how chemical compounds affect biological systems. High-throughput devices are needed to efficiently gather large volumes of data for these screens. Recently we developed a low cost high-throughput imaging solution to capture the effects of chemicals on the egg-laying behaviour of C. elegans. We present a second-generation prototype capable of high-resolution (0.2 µm/px) imaging of individual worms and their internal structures. It features an automatic real time segmentation of C. elegans worms and is compatible with 96-well plates. In order to maximize efficiency in data collection, we developed an approach for simultaneous dual imaging through a single objective. A low-resolution (15 µm/px) arm continuously collects images of an entire well using a CCD camera (1280 px by 1024 px). The images are continuously fed into a Python image algorithm that tracks the movement of the worms in regions of interest (ROIs). Using this information, high resolution images of the worms are captured using a linescan camera (8000 px). This solution provides automated high-resolution imaging minimizing acquisition time by eliminating the need for multiple objectives, allowing image collection with a continuously moving system (high speed linescan camera) and minimizing the amount of stitching (large linescan sensor). The scan time of a 96-well plate is proportional to the number of ROIs, which is significantly less than imaging an entire plate. The high-resolution images can be fed into an image processing pipeline that will be able to classify the effects of chemicals on the internal structure of the worms. Simultaneous acquisition of high and low resolution data can be used to correlate deformations of internal structure with changes in mobility. This low cost solution provides users with an automated approach to obtain high-resolution images and mobility statistics of C. elegans.