Microscopic Vision System Research Articles

Digitalization method of microassembly space consisting of trans-scale microparts and microgripper jaws used for digital microassembly

Because of small depth of field and field of view of microscopic vision system in microassembly system, the trans-scale microparts and microgripper jaws can not be observed and described simultaneously, which limits microassembly of trans-scale microparts. Focusing on simultaneously obtaining global morphologies and local features of trans-scale microparts and microgripper jaws in microassembly space with high precision and realizing digital microassembly, this paper presents a principle of digitalization of microassembly space that can transform the microassembly space containing trans-scale microparts and microgripper jaws into digital microassembly space expressed by 3D point cloud. By using the obtained digital microassembly space, the trans-scale microparts and microgripper jaws in microassembly space can be observed and described, their global morphologies and local features can be simultaneously obtained and digitally measured with high accuracy, and their positions and orientations can be estimated for realizing their digital microassembly.

Depth estimation method of surface of micropart in microassembly space based on microscopic vision tomographic scanning images.

Three-dimensional (3D) morphology of microparts has an important influence on performance of microassembly system that mainly assembles microparts in millimetre and micron scale. Because 3D morphology of microparts cannot be accurately obtained by conventional microscopic vision system, a depth estimation method of surface of micropart in microassembly space based on microscopic vision tomographic scanning (MVTS) images is proposed in this paper. The proposed method uses the positions of pixels with the largest focus values in MVTS image to construct the isodepth contours of surface of micropart and obtains the depth values of micropart's surface at the positions of MVTS by assigning depth values to corresponding isodepth contours. The MVTS images are obtained by MVTS and pixels with the largest focus values in MVTS image are obtained by focus measurement of MVTS images of micropart in microassembly space. On these bases, 3D spatial interpolation method is applied to map depth value of space between adjacent isodepth contours and to obtain depth values of all surface of micropart. Simulation experiments are carried out to verify the proposed method by generating simulated MVTS image array from two simulation objects, and the influence parameters of the proposed method are analysed. In established experimental setup of microassembly that can realise MVTS, experimental verification for the proposed depth estimation method are carried out by using cone cavity and end jaws of microgripper. 3D morphologies of depth maps of cone cavity and end jaws of microgripper are registered with their respective CAD models using iterative nearest point registration algorithm to quantify accuracy of depth estimation. The research results show that 3D morphology of micropart can be obtained by the proposed method and has better accuracy than those by conventional shape from focus method. This method provides a new way to obtain the morphology of microparts and lays a foundation for improving the accuracy and efficiency of gripping, alignment and approaching microparts in microassembly systems.

Microscope vision system based on micro laser line scanning for characterizing microscale topography.

A microscope vision system to characterize a microscale surface via micro laser line projection is presented. The characterization is performed by means of surface descriptors, which include the root mean square, kurtosis, skewness, homogeneity, entropy, contrast, and correlation. These descriptors are computed from surface irregularities, which are retrieved by means of the micro laser line projection. The characterization is carried out by an optical microscope system on which a CCD camera and a 36 µm laser line are attached. Thus, the microscope vision system projects the micro laser line on the surface, and the CCD camera captures the line reflection, which provides the surface contour. The contour dimension is computed via Bezier networks by means of the micro laser line coordinates. Thus, the surface descriptors are computed by means of the surface contour to accomplish the characterization. The proposed characterization improves the accuracy of the optical microscope imaging systems, which characterize the microscale surface by means of the gray-level intensity. Thus, the capability of the characterization via micro laser line projection is established by means of the descriptors' accuracy. This contribution is corroborated by characterizing metal and paper surfaces.

Adaptive Kinematic Mapping Based on Chebyshev Interpolation: Application to Flexure-Jointed Micromanipulator Control

For flexure-jointed mechanisms and manipulators, accurate kinematic models typically involve complex nonlinear descriptions of elastic behavior and are ill-suited for real-time control computations. An alternative approach is to use multidimensional function approximation methods, with optimization via online kinematic identification and calibration. In this article, a function interpolation approach is proposed based on Chebyshev approximation theory for near-optimal minimization of maximum positioning errors. The method allows fast calibration procedures using a small number of data points. The implementation involves a correction mapping that operates on command input variables before an approximate inverse kinematic model is applied. An adaptation algorithm is further proposed that can be used to update and refine mappings: 1) in a localized space for improved precision for the current task, or 2) globally by using calibration points chosen to match the Chebyshev nodes of the overall workspace. Results are shown for simulation of a flexure-jointed X–Y motion stage and for experiments on a X–Y–Z micromanipulation platform with Delta-type parallel kinematics. For the experiments, direct measurement of platform position was achieved using a microscope vision system. The proposed method gave order-of-magnitude improvements in positioning accuracy compared with the pseudo-rigid-body modeling approach and was found to out-perform direct visual servoing when operating with similar image capture rates.

Precision 3-D Motion Tracking for Binocular Microscopic Vision System

In this paper, a three-dimensional (3-D) motion tracking method is proposed for binocular microscopic vision system to precisely record the motion trajectories of millimeter size objects in the Cartesian space. Primarily two fundamental problems are solved. The first problem arises from the limited depth of field (DOF) of microscope. Considering the motion of the objects, the existing autofocusing methods requiring sequential images either in focus or defocus are not workable. Therefore, a one-shot prior autofocusing approach is desired, which needs to take the motion tendency of objects into account. Besides, the autofocusing process always lags behind the motion of objects, and there inevitably will be prediction deviation on the motion tendency of objects. This leads to the second problem to estimate the 3-D motion states from defocused images. In this paper, we first explain the defocusing process from the perspective of S-Transform, based on which the Bayesian inference inspired method to estimate the depth from defocus from one single image is derived thereafter. Afterwards, a motion states, including both the position and velocity, estimation approach is developed within the Kalman filter framework. The above two aspects mutually supply the necessary information for each other to be functional to accurately realize the DOF tracking and motion tracking of moving objects. Experiments were well-designed to validate the effectiveness of the proposed method, and experiments result showed a tracking precision of 3 μ m was achieved.

Application of convolutional neural network to acquisition of clear images for objects with large vertical size in stereo light microscope vision system.

For an object with large vertical size that exceeds the certain depth of a stereo light microscope (SLM), its image will be blurred. To obtain clear images, we proposed an image fusion method based on the convolutional neural network (CNN) for the microscopic image sequence. The CNN was designed to discriminate clear and blurred pixels in the source images according to the neighborhood information. To train the CNN, a training set that contained correctly labeled clear and blurred images was created from an open-access database. The image sequence to be fused was aligned at first. The trained CNN was then used to measure the activity level of each pixel in the aligned source images. The fused image was obtained by taking the pixels with the highest activity levels in the source image sequence. The performance was evaluated using five microscopic image sequences. Compared with other two fusion methods, the proposed method obtained better performance in terms of both visual quality and objective assessment. It is suitable for fusion of the SLM image sequence.

Contour extraction of a laser stripe located on a microscope image from a stereo light microscope.

A stereo light microscope has a special optical structure that consists of two optical paths. With two cameras fitted on its imaging planes, a microscopic binocular vision system can be constructed, and this system can be applied in three-dimensional (3D) shape measurements of a microscopic object. A novel-shape reconstruction system is developed in this article composed of laser fringe scanning and a stereo light microscope. A laser projector emits a thin light sheet and projects it onto the microscopic object's surface. The microscopic object is placed on a micro-displacement mechanism and moved in a given direction. The entire surface of the object is scanned by the light sheet. This system captures a series of microscopic images of the laser stripe, which are used to restore the 3D shape of the object. In this article, we mainly focus on the study of the laser stripe detection method, which is derived based on the Canny rule. First, the Canny rule outputs pixels at the left and right edges of the laser stripe. Then, a subpixel edge extraction method is proposed based on polynomial fitting that outputs the center curve of the laser stripe. Finally, an edge filter is used to smooth the edge burrs, and the Hermite interpolation method is used to link the broken edges and construct continuous, smoothing edge contours. The results show that this method can effectively find the subpixel position of the laser stripe and output high-quality edge contours. The method is suitable for extracting the contours of a laser stripe located in a microscope image.

Microscopic Machine Vision Based Degradation Monitoring of Low-Voltage Electromagnetic Coil Insulation Using Ensemble Learning in a Membrane Computing Framework

In this paper, a novel microscopic machine vision system is proposed to solve a degradation monitoring problem of low-voltage electromagnetic coil insulation in practical industrial fields, where an ensemble learning approach in a compound membrane computing framework is newly introduced. This membrane computing framework is constituted by eight layers, 29 membranes, 72 objects, and 35 rules. In this framework, multiple machine learning methods, including classical pattern recognition methods and novel deep learning methods, are tested and compared. First, the most optimal feature extraction approaches are selected. Then, the selected approaches are fused together to achieve an even better monitoring performance. Third, a large number of experiments are used to evaluate and prove the usefulness and potential of the proposed system, where a mean accuracy of 61.4% is achieved on 1035 validation images of six degradation states with single state matching, and mean accuracies of 61.0% and 77.4% are achieved on 622 test images of six degradation states with single state matching and state range matching, respectively. Finally, a mechanical device is designed to apply the system to real industrial tasks.

Contour Primitives of Interest Extraction Method for Microscopic Images and Its Application on Pose Measurement

This paper proposes a suite of methods to realize high precision pose measurement in 3-D Cartesian space based on a multicamera microscopic vision system. Since it is inefficient to develop a specific image algorithm for each kind of object and the imaging condition might be unsatisfactory, we propose a method of contour primitives of interest extraction, which allows flexible reconfiguration for novel object image and owns robustness under different imaging conditions. The object is detected in grayscale image based on a template of contour primitives. Edges are extracted according to derivatives along the normal vectors of these contour primitives. The positions and directional derivatives of these edges are used for feature extraction and autofocus, respectively. The point features and line features extracted from multiview images are utilized to measure 3-D vectors and orientations, respectively, based on image Jacobian matrices. Cameras’ linear motions are considered in the imaging model, so that the measurement range is expanded beyond the limitation of microscopes’ shallow depths of field. The affine epipolar constraint and focused planes intersection constraint between cameras are applied to improve the real time performances of image feature extraction and multicamera autofocus, respectively. A series of experiments are conducted to verify the effectiveness of the proposed methods. The root mean square errors of pose measurement are evaluated as 3 ${ {\mu } }\text{m}$ in position and 0.05° in orientation, while the measurement range is about 5000 $ {\mu }\text{m}$ in position and 20° in orientation.

Pareto optimization-based trade-off between the visual resolvability and the intersection space of view of binocular microscopic vision system for microassembly systems

Visual resolvability and intersection space of view of a binocular microscopic vision system (MVS) in microassembly systems have influences on each other via analyzing theoretically. According to requirements of various microassembly tasks, it is an explicit problem how to find the best compromise between the visual resolvability of binocular MVS and its intersection space of view in three-dimensional space. A Pareto optimization-based method to make the trade-off between the visual resolvability and the intersection space of view of binocular MVS with relevant fixed parameters in microassembly systems is proposed. Through modeling and analyzing on the visual resolvability and intersection space of view of a binocular MVS, the topology and optical magnifications of two MVSs are considered as the key parameters to optimize its physical properties. A multiobjection optimization model based on a Pareto optimization method is formulated and used to find the Pareto optimal cure, namely the Pareto front, which represents the best trade-off between the visual resolvability and the intersection space of view of a binocular MVS. On these bases, the physical properties of a binocular MVS with structural parameters corresponding to the optimization solutions are numerically simulated. Accordingly, a binocular MVS with different parameters is established on an automatic microassembly system with a single manipulator with five degree-of-freedom and experimentally tested. The simulation and experimental results indicate that all of the optimal solutions, which locate on the Pareto front, are the best trade-off between the visual resolvability and the intersection space of view of a binocular MVS. According to the Pareto front, physical parameters of a binocular MVS can be chosen to get the optimal visual resolvability with a certain intersection space of view and vice versa.

Microscope self-calibration based on micro laser line imaging and soft computing algorithms

A technique to perform microscope self-calibration via micro laser line and soft computing algorithms is presented. In this technique, the microscope vision parameters are computed by means of soft computing algorithms based on laser line projection. To implement the self-calibration, a microscope vision system is constructed by means of a CCD camera and a 38 µm laser line. From this arrangement, the microscope vision parameters are represented via Bezier approximation networks, which are accomplished through the laser line position. In this procedure, a genetic algorithm determines the microscope vision parameters by means of laser line imaging. Also, the approximation networks compute the three-dimensional vision by means of the laser line position. Additionally, the soft computing algorithms re-calibrate the vision parameters when the microscope vision system is modified during the vision task. The proposed self-calibration improves accuracy of the traditional microscope calibration, which is accomplished via external references to the microscope system. The capability of the self-calibration based on soft computing algorithms is determined by means of the calibration accuracy and the micro-scale measurement error. This contribution is corroborated by an evaluation based on the accuracy of the traditional microscope calibration.

基于3-CCD显微视觉的高精度靶位姿控制

基于3-CCD显微视觉的高精度靶位姿控制

Quality detection system and method of micro-accessory based on microscopic vision

Considering that the traditional manual detection of micro-accessory has some problems, such as heavy workload, low efficiency and large artificial error, a kind of quality inspection system of micro-accessory has been designed. Micro-vision technology has been used to inspect quality, which optimizes the structure of the detection system. The stepper motor is used to drive the rotating micro-platform to transfer quarantine device and the microscopic vision system is applied to get graphic information of micro-accessory. The methods of image processing and pattern matching, the variable scale Sobel differential edge detection algorithm and the improved Zernike moments sub-pixel edge detection algorithm are combined in the system in order to achieve a more detailed and accurate edge of the defect detection. The grade at the edge of the complex signal can be achieved accurately by extracting through the proposed system, and then it can distinguish the qualified products and unqualified products with high precision recognition.

Three-dimensional microscope vision system based on micro laser line scanning and adaptive genetic algorithms

A microscope vision system to retrieve small metallic surface via micro laser line scanning and genetic algorithms is presented. In this technique, a 36µm laser line is projected on the metallic surface through a laser diode head, which is placed to a small distance away from the target. The micro laser line is captured by a CCD camera, which is attached to the microscope. The surface topography is computed by triangulation by means of the line position and microscope vision parameters. The calibration of the microscope vision system is carried out by an adaptive genetic algorithm based on the line position. In this algorithm, an objective function is constructed from the microscope geometry to determine the microscope vision parameters. Also, the genetic algorithm provides the search space to calculate the microscope vision parameters with high accuracy in fast form. This procedure avoids errors produced by the missing of references and physical measurements, which are employed by the traditional microscope vision systems. The contribution of the proposed system is corroborated by an evaluation via accuracy and speed of the traditional microscope vision systems, which retrieve micro-scale surface topography.

Study on clear stereo image pair acquisition method for small objects with big vertical size in SLM vision system.

Microscopic vision system with stereo light microscope (SLM) has been applied to surface profile measurement. If the vertical size of a small object exceeds the range of depth, its images will contain clear and fuzzy image regions. Hence, in order to obtain clear stereo images, we propose a microscopic sequence image fusion method which is suitable for SLM vision system. First, a solution to capture and align image sequence is designed, which outputs an aligning stereo images. Second, we decompose stereo image sequence by wavelet analysis theory, and obtain a series of high and low frequency coefficients with different resolutions. Then fused stereo images are output based on the high and low frequency coefficient fusion rules proposed in this article. The results show that Δw1 (Δw2 ) and ΔZ of stereo images in a sequence have linear relationship. Hence, a procedure for image alignment is necessary before image fusion. In contrast with other image fusion methods, our method can output clear fused stereo images with better performance, which is suitable for SLM vision system, and very helpful for avoiding image fuzzy caused by big vertical size of small objects.

Image distortion correction for micromanipulation system based on SLM microscopic vision.

Stereo light microscope (SLM) simulates stereo imaging principle of human eyes. Microscopic vision system based on SLM has become an important visual tool for micro measurement, micromanipulation, and microinjection. We develop a micromanipulation system based on SLM and present an image distortion correction method. We mainly correct two kinds of image distortions: lateral and vertical distortion. Distortion correction consists of two steps. First, a linear fitting algorithm for each row or column of target points is developed, and the fitting errors are calculated. If the fitting errors are smaller than a given threshold, the linear fitting results are kept and used. Otherwise polynomial fitting procedure will be used. Second, the parallelism of straight lines is corrected. The results show that a line in world coordinate frame (WCF) is not necessarily a straight line in image coordinate frame (ICF), or two parallel lines in WCF may be not parallel in ICF. Distortion correction can restore the parallel and linear relationship. For distorted left and right images, the magnitude of distortion exceeds 6 pixels and 4 pixels in the horizontal direction, and 1.2 pixels and 1.7 pixels in the vertical direction, respectively. After corrected, for left and right image, distortion can be reduced to 0.8 pixels and 0.7 pixels in the horizontal direction, and 0.96 pixels and 1.3 pixels in the vertical direction, respectively. The results show that distortion parameters obtained from the proposed method can effectively correct distorted images.

Position/Force Hybrid Control System for High Precision Aligning of Small Gripper to Ring Object

A position/force hybrid control system based on impedance control scheme is designed to align a small gripper to a special ring object. The vision information provided by microscope vision system is used as the feedback to indicate the position relationship between the gripper and the ring object. Multiple image features of the gripper and the ring object are extracted to estimate the relative positions between them. The end-effector of the gripper is tracked using the extracted features to keep the gripper moving in the field of view. The force information from the force sensor serves as the feedback to ensure that the contact force between the gripper and the ring object is limited in a small safe range. Experimental results verify the effectiveness of the proposed control strategy.

An improved surface roughness measurement method for micro-heterogeneous texture in deep hole based on gray-level co-occurrence matrix and support vector machine

Microscopic vision system has been employed to measure the surface roughness of micro-heterogeneous texture in deep hole, by virtue of frequency domain features of microscopic image and back-propagation artificial neural network optimized by genetic algorithm. However, the measurement accuracy needs to be improved for engineering application. In this paper, we propose an improved method based on microscopic vision to detect the surface roughness of R-surface in the valve. Firstly, the measurement system for the roughness of R-surface in deep hole is described. Thereafter, the surface topography images of R-surface are analyzed by the gray-level co-occurrence matrix (GLCM) method, and several features of microscopic image, which are nearly monotonic with the surface roughness, are extracted to fabricate the prediction model of the roughness of R-surface accurately. Moreover, a support vector machine (SVM) model is presented to describe the relationship of GLCM features and the actual surface roughness. Finally, experiments on measuring the surface roughness are conducted, and the experimental results indicate that the GLCM-SVM model exhibits higher accuracy and generalization ability for evaluating the microcosmic surface roughness of micro-heterogeneous texture in deep hole.

Cancelling bias induced by correlation coefficient interpolation for sub-pixel image registration

Accurate sub-pixel image registration has many applications in precision engineering and manufacturing. Due to its computational efficiency, correlation-based sub-pixel image registration is especially useful for real-time applications. This paper presents a theoretical analysis of the bias induced by correlation coefficient interpolation, when employed to achieve two-dimensional sub-pixel registration resolution. An analytical model of bias in terms of true sub-pixel image shift is first derived according to the given reference image model. It is then used to establish an accurate bias map expressed explicitly in terms of sub-pixel image shift estimated from coefficient interpolation. This bias map is readily employed to eliminate the bias error and to significantly improve measurement accuracy for real-time applications. A bias cancellation scheme was implemented and verified by means of both computer simulation and experimental results. Specifically, sub-nanometre measurement precision was experimentally demonstrated using a microscopic vision system implemented for real-time motion tracking.

Bias-error reduction in nanometer resolution microscopic motion tracking vision systems

Microscopic Motion Tracking Systems utilize sub-pixel algorithms to achieve close to nanometer precision motion measurement. This paper characterizes the bias-error of template matching image registration techniques which are commonly used to achieve this level of sub-pixel accuracy. An accurate functional expression for such bias error is derived, which shows how the error depends on image content. Furthermore, this expression is directly utilized in a bias reduction scheme that is shown to significantly reduce the bias error. The newly derived expressions and reduction schemes for bias are applied to a Microscopic Vision System that measures micro-object motion with close to nanometer resolution.