Microscopic Scenes Research Articles

A Segment-Based Algorithm for Grid Junction Corner Detection Used in Stereo Microscope Camera Calibration

Corner detection is responsible for accurate camera calibration, which is an essential task for binocular three-dimensional (3D) reconstruction. In microscopic scenes, binocular 3D reconstruction has significant potential to achieve fast and accurate measurements. However, traditional corner detectors and calibration patterns (checkerboard) performed poorly in microscopic scenes due to the non-uniform illumination and the shallow depth of field of the microscope. In this paper, we present a novel method for detecting grid junction corners based on image segmentation, offering a robust alternative to the traditional checkerboard pattern. Model fitting was utilized to obtain the coordinates at a sub-pixel level. The procedures of the proposed method were elaborated, including image segmentation, corner prediction, and model fitting. The mathematical model was established to describe the grid junction. The experiment was conducted using both synthetic and real data and the experimental result shows that this method achieves high precision and is robust to image blurring, indicating this method is suitable for microscope camera calibration.

Snapshot temporal compressive light-sheet fluorescence microscopy via deep denoising and total variation priors.

We present a snapshot temporal compressive light-sheet fluorescence microscopy system to capture high-speed microscopic scenes with a low-speed camera. A deep denoising network and total variation denoiser are incorporated into a plug-and-play framework to quickly reconstruct 20 high-speed video frames from a short-time measurement. Specifically, we can observe 1,000-frames-per-second (fps) microscopic scenes when the camera works at 50 fps to capture the measurement. The proposed method can potentially be applied to observe cell and tissue motions in thick living biological specimens.

3D Surface Reconstruction Method for Microsamples Based on Polarization Modulation Microscopy

A novel microscopic Shape-From-Polarization (SFP) 3D shape reconstruction method is proposed, in which degree of polarization is calculated using Polarized Bidirectional Reflectance Distribution Function (PBRDF) to overcome azimuthal ambiguity for both specular and diffuse polarization microscopic scenes. Based on the calculated degree of polarization, a polarization modulation microscopy oriented variational model is established to estimate surface normal vector and rebuild 3D shape of the sample with “fuzzy guidance” based on atomic force microscopy (AFM). Experiments on different microscopic samples indicate the proposed method can restore the stereoscopic morphology distribution with precision in micrometer.

Improved Image Fusion Method Based on Sparse Decomposition

In the principle of lens imaging, when we project a three-dimensional object onto a photosensitive element through a convex lens, the point intersecting the focal plane can show a clear image of the photosensitive element, and the object point far away from the focal plane presents a fuzzy image point. The imaging position is considered to be clear within the limited size of the front and back of the focal plane. Otherwise, the image is considered to be fuzzy. In microscopic scenes, an electron microscope is usually used as the shooting equipment, which can basically eliminate the factors of defocus between the lens and the object. Most of the blur is caused by the shallow depth of field of the microscope, which makes the image defocused. Based on this, this paper analyzes the causes of defocusing in a video microscope and finds out that the shallow depth of field is the main reason, so we choose the corresponding deblurring method: the multi-focus image fusion method. We proposed a new multi-focus image fusion method based on sparse representation (DWT-SR). The operation burden is reduced by decomposing multiple frequency bands, and multi-channel operation is carried out by GPU parallel operation. The running time of the algorithm is further reduced. The results indicate that the DWT-SR algorithm introduced in this paper is higher in contrast and has much more details. It also solves the problem that dictionary training sparse approximation takes a long time.

Depth Estimation Method for Monocular Camera Defocus Images in Microscopic Scenes

When using a monocular camera for detection or observation, one only obtain two-dimensional information, which is far from adequate for surgical robot manipulation and workpiece detection. Therefore, at this scale, obtaining three-dimensional information of the observed object, especially the depth information estimation of the surface points of each object, has become a key issue. This paper proposes two methods to solve the problem of depth estimation of defiant images in microscopic scenes. These are the depth estimation method of the defocused image based on a Markov random field, and the method based on geometric constraints. According to the real aperture imaging principle, the geometric constraints on the relative defocus parameters of the point spread function are derived, which improves the traditional iterative method and improves the algorithm’s efficiency.

DNF: diffractive neural field for lensless microscopic imaging.

Lensless imaging has emerged as a robust means for the observation of microscopic scenes, enabling vast applications like whole-slide imaging, wave-front detection and microfluidic on-chip imaging. Such system captures diffractive measurements in a compact optical setup without the use of optical lens, and then typically applies phase retrieval algorithms to recover the complex field of target object. However existing techniques still suffer from unsatisfactory performance with noticeable reconstruction artifacts especially when the imaging parameter is not well calibrated. Here we propose a novel unsupervised Diffractive Neural Field (DNF) method to accurately characterize the imaging physical process to best reconstruct desired complex field of the target object through very limited measurement snapshots by jointly optimizing the imaging parameter and implicit mapping between spatial coordinates and complex field. Both simulations and experiments reveal the superior performance of proposed method, having > 6 dB PSNR (Peak Signal-to-Noise Ratio) gains on synthetic data quantitatively, and clear qualitative improvement on real-world samples. The proposed DNF also promises attractive prospects in practical applications because of its ultra lightweight complexity (e.g., 50× model size reduction) and plug-to-play advantage (e.g., random measurements with a coarse parameter estimation).

Microscopic 3D reconstruction based on point cloud data generated using defocused images

Most of the 3D reconstruction requirements of microscopic scenes exist in industrial detection, and this scene requires real-time object reconstruction and can get object surface information quickly. However, this demand is challenging to obtain for micro scenarios. The reason is that the microscope’s depth of field is shallow, and it is easy to blur the image because the object’s surface is not in the focus plane. Under the video microscope, the images taken frame by frame are mostly defocused images. In the process of 3D reconstruction, a single sheet or a few 2D images are used for geometric-optical calculation, and the affine transformation is used to obtain the 3D information of the object and complete the 3D reconstruction. The feature of defocus image is that its complete information needs to be restored by a whole set of single view defocus image sequences. The defocused image cannot complete the task of affine transformation due to the lack of information. Therefore, using defocus image sequence to restore 3D information has higher processing difficulty than ordinary scenes, and the real-time performance is more difficult to guarantee. In this paper, the surface reconstruction process based on point-cloud data is studied. A Delaunay triangulation method based on plane projection and synthesis algorithm is used to complete surface fitting. Finally, the 3D reconstruction experiment of the collected image sequence is completed. The experimental results show that the reconstructed surface conforms to the surface contour information of the selected object.

A deep learning approach for real-time crash prediction using vehicle-by-vehicle data

In road safety, real-time crash prediction may play a crucial role in preventing such traffic events. However, much of the research in this line generally uses data aggregated every five or ten minutes. This article proposes a new image-inspired data architecture capable of capturing the microscopic scene of vehicular behavior. In order to achieve this, an accident-prediction model is built for a section of the Autopista Central urban highway in Santiago, Chile, based on the concatenation of multiple-input Convolutional Neural Networks, using both the aggregated standard traffic data and the proposed architecture. Different oversampling methodologies are analyzed to balance the training data, finding that the Deep Convolutional Generative Adversarial Networks technique with random undersampling presents better results when generating synthetic instances that permit maximizing the predictive performance. Computational experiments suggest that this model outperforms other traditional prediction methodologies in terms of AUC and sensitivity values over a range of false positives with greater applicability in real life.

Snapshot temporal compressive microscopy using an iterative algorithm with untrained neural networks.

We report a snapshot temporal compressive microscopy imaging system, using the idea of video compressive sensing, to capture high-speed microscopic scenes with a low-speed camera. An untrained deep neural network is used in our iterative inversion algorithm to reconstruct 20 high-speed video frames from a single compressed measurement. Specifically, using a camera working at 50 frames per second (fps) to capture the measurement, we can recover videos at 1000 fps. Our deep neural network is embedded in the inversion algorithm, and its parameters are learned simultaneously with the reconstruction.

Multifocus image fusion and depth reconstruction

The acquisition of sharp, full-focus images and the restoration of microscopic scenes require complex instrumentation and algorithms. To conveniently obtain the three-dimensional (3-D) structural information of full-focus images and high-quality microscopic scenes, we construct a zoomable 3-D microscope imaging system and propose new image fusion and depth reconstruction algorithms based on this imaging system. The image acquisition environment of the system is analyzed using interference factors such as light transmission variation and jitter, and the corresponding image preprocessing methods are discussed. We combined the new sum-modified-Laplacian (SML) and local band-limited contrast methods, which contain multiple image features but measure the image definition from different angles, then proposed a mixed-contrast factor, and combined it with the SML method to propose an image fusion method. We then proposed a depth reconstruction method based on the structural similarity of multifocus images. For cases where depth reconstruction results in large distortion with high noise, we proposed a depth value restoration method based on anisotropic diffusion to improve the 3-D reconstruction results. Experimental results show that the proposed image fusion and depth reconstruction methods exhibit excellent performance.

Image-based characterization of thrombus formation in time-lapse DIC microscopy

The characterization of thrombus formation in time-lapse DIC microscopy is of increased interest for identifying genes which account for atherothrombosis and coronary artery diseases (CADs). In particular, we are interested in large-scale studies on zebrafish, which result in large amount of data, and require automatic processing. In this work, we present an image-based solution for the automatized extraction of parameters quantifying the temporal development of thrombotic plugs. Our system is based on the joint segmentation of thrombotic and aortic regions over time. This task is made difficult by the low contrast and the high dynamic conditions observed in vivo DIC microscopic scenes. Our key idea is to perform this segmentation by distinguishing the different motion patterns in image time series rather than by solving standard image segmentation tasks in each image frame. Thus, we are able to compensate for the poor imaging conditions. We model motion patterns by energies based on the idea of dynamic textures, and regularize the model by two prior energies on the shape of the aortic region and on the topological relationship between the thrombus and the aorta. We demonstrate the performance of our segmentation algorithm by qualitative and quantitative experiments on synthetic examples as well as on real in vivo microscopic sequences.

Key technologies of light field capture for 3D reconstruction in microscopic scene

Light field capture for 3D reconstruction in microscopic scene is a very promising and useful technology, which can be extensively applied to life sciences, medicine, materials science, etc. This paper summarizes the key technologies in the evolution of microscopes for capturing 3D information, including wavefront-reconstruction, holography, fluorescence, tomography and so on. To give in-depth insights into them, detailed analyses and comparisons are provided. Finally, some future potential work in terms of light field capture and its application are discussed at length.

Application of Real-Time Photoelastic Analysis to Single Fibre Fragmentation Tests

One of our main research areas is the trans-scale modelling of damage in composite materials, which consist of a polymer matrix and carbon or glass fibres in different material combinations and geometrical arrangements. From the local stress redistribution in the fibre-matrix interphase and in the surrounding matrix material information on the parameters of microscopic damage models for composite materials can be obtained. Owing to the difficult interface characterisation based on the properties of the single material components, a photoelastic analysis of single fibre fragmentation tests is performed. In addition to the qualitative visual interpretation in polarized light, an enhanced quantitative analysis in combination with digital photoelasticity using a four image phase shifting method will be applied [1]. As the sequential capturing of images might cause incorrect results, these four pictures are grabbed simultaneously. This allows for continuous testing. Additionally, errors due to the relaxation behaviour of the matrix material can be avoided. To this, a modular optical system consisting of a variable long distance microscope and a beam dividing module proposed by [2] was developed. It allows for the simultaneous projection of four different filtered images of one microscopic scene to the four quadrants of a CCD chip. This special equipment gives the possibility to apply quantitative photoelasticity to tensile tests performed on standard testing machines. This paper explains the measurement hardware and discusses the main problems and realised solutions from picture capturing through image processing to real-time photoelastic analysis at the present state of development. Exemplary results for the qualitative analysis of selected material combinations and different manufacturing processes are shown.

Enhancing Depth of Field in LCSEM Scenes by Sharpness Map Partitioning

Large scale scanning electron microscopes (LCSEM) suffer from the problem of limited depth of field (DOF) making it difficult to inspect and image a 3-dimensional microscopic scene. Multifocus fusion is the process of unifying focal information from a set of scanned input images into one image, as one acquired under an extended DOF. Each input image has certain regions of the scene in focus, and an image partition is a region or a set of regions in an input image that fall on the same focal plane. The crux of our method is to isolate and attribute such partitions to one particular input image. A sharpness map is calculated for every input image Ii(x,y), using { } 2 / 1 2yi 2xi i ) y , x ( I ) y , x ( I ) y , x ( S + = ,where Ixi(x,y) and Iyi(x,y) are horizontal and vertical gradient maps. When the sharpness image of input image Ii(x,y) is examined with its N-1 counterparts for regions of sharper focus, one image partition, Pi(x,y) is isolated by, ) y , x ( S ) y , x ( S ) y , x ( P } i k { i i ≠ > = , for all k≠i. The union of the partitions, Pi(x,y)’s, forms the fused image space and the intersection of the partitions is the null set, i.e. the blurred sections of all the input images. The image partitions are then seamlessly mosaiced to form the fused image, from

Spectral Imaging and Pathology: Seeing More

Anatomic pathology is a visualdiscipline. Microscopic scenes ofhematoxylin and eosin-stained tissuesyield both low-power spatial clues aswell as insights on the structural andfunctional state of individual cells,thousands of which may be visible at atime. Add to this the fact that assayssuch as immunohistochemistry and insitu hybridizations can provide molecu-lar information within the architecturalcontexts, and it is surprising that digitaland quantitative imaging continues toplay no more than a small role in clini-cal practice. This is still true despiteremarkable improvements in the capa-bilities, cost, and availability of cam-eras (sensors) and computers. Anumber of factors are responsible forthis. The human eye-brain combinationis skilled at perceiving and interpretingvisual scenes and can rapidly evaluatea sample at low magnification and,with facility, switch to high power forvalidation or for further detail. To dothe same electronically may require thecollection and analysis of literally thou-sands of high-resolution digital imagesfrom an entire sample. Secondly,pathology is difficult, and each image-based interpretative system must spe-cialize in the nuances of a particularorgan and disease. Thirdly, current im-aging hardware-softwareconfigurations often require trainedoperator assistance, although graduallythe demands on the user are lessening.Finally, while subjective (“1+, 2+, 3+”)grading or scoring schemes are widelyused, there is little discernible enthusi-asm among practicing pathologists forobjective, quantitative image-based de-scriptors, at least for metrics such asnuclear grade or dysplasia. Reasons forthis include the fact that the case formany quantitative schemes have, untilrecently, not been compelling; thatmulti-institutional image analysis pro-tocols are difficult to implement; andthat recognized standards for imageanalysis have not been established.On the other hand, molecularlybased techniques are being adoptedinto pathology at an ever-increasingrate. Many of these rely on suspensionsor extracts of cells or tissues, but oth-ers, such as immunohistochemistry andfluorescence in situ hybridization, areperformed on tissue sections or isolatedcells and the results evaluated througha microscope. In contrast to conven-tional histopathology, the simplifiedreadout (absence or presence of a cer-tain color) and the (semi-) quantitativenature of the techniques (percent ofcells positive, degree of positivity,number of signals per cell) are morereadily approached through digitalimage analysis.Spectral imaging is a relativelynew technique that acquires an imagealong with an optical spectrumcaptured at every pixel. Its performanceimprovement over visual examinationand conventional grayscale or 3-colorimaging has led to the development ofnovel, qualitative, as well as quantita-tive tools for interpreting convention-ally stained pathology samples,overcoming some, although not all, ofthe obstacles outlined above. In addi-tion, spectral imaging facilitates the useand extension of molecular pathologyapproaches; in particular, permittingthe detection and analysis of multipleprobes simultaneously. Spectral imag-ing can be applied to fluorescence-based assays, but it offers particularpromise for increasing the utility ofstandard brightfield microscopy famil-iar to all pathologists.Multiplexing in BiomedicineThe genomics revolution and thearrival of truly molecular medicinehave led to demands on pathologists toprovide more information on individualpatients’ tumors. An accurate diagno-sis, perhaps accompanied by broad sta-tistical predictions of how the patientmight fare must now be supplementedwith predictive and prescriptive infor-mation specific to the individual pa-tient. While a great deal of effort isbeing applied to DNA-chip-based RNAexpression arrays for molecular profil-ing of tumors,

Chemistry by Touch

In the world of nature's architecture dominates the microscopic scene. A protein may show up as hundreds of clustered spheres, while in DNA, atoms spiral up a long helix. People appreciate chemical forms mainly with their eyes. But what if a person cannot see? Can a blind person comprehend molecular structure? Blind students have struggled with molecular structures for years in studying chemistry, says Lawrence Scadden, the National Science Foundation's program director for persons with disabilities. one of them. I'm blind. I got my Ph.D. many years ago, and taking chemistry was difficult. It's hard to grasp molecular structures without asking someone to build a model. Now, chemist William J. Skawinski, a postdoctoral research associate at the New Jersey Institute of Technology in Newark, is doing just that. Sitting at his computer workstation, he puts the finishing touches on a 150-atom molecule he has just designed. Having made the calculations for a chemical destined to bind to a receptor with lock-andkey precision, he enters information about its chemical structure and shape into his terminal. After formatting the data with his unique program, he feeds the file into a computer that controls a laser stereolithography machine. Several hours later, a plastic model of the molecule emerges.