Multi-frame Blind Deconvolution Research Articles

Solar multiobject multiframe blind deconvolution with a spatially variant convolution neural emulator

The study of astronomical phenomena through ground-based observations is always challenged by the distorting effects of Earth's atmosphere. Traditional methods of post facto image correction, essential for correcting these distortions, often rely on simplifying assumptions that limit their effectiveness, particularly in the presence of spatially variant atmospheric turbulence. Such cases are often solved by partitioning the field of view into small patches, deconvolving each patch independently, and merging all patches together. This approach is often inefficient and can produce artifacts. Recent advancements in computational techniques and the advent of deep learning offer new pathways to address these limitations. This paper introduces a novel framework leveraging a deep neural network to emulate spatially variant convolutions, offering a breakthrough in the efficiency and accuracy of astronomical image deconvolution. By training on a dataset of images convolved with spatially invariant point spread functions and validating its generalizability to spatially variant conditions, this approach presents a significant advancement over traditional methods. The convolution emulator is used as a forward model in a multiobject multiframe blind deconvolution algorithm for solar images. The emulator enables the deconvolution of solar observations across large fields of view without resorting to patch-wise mosaicking, thus avoiding the artifacts associated with such techniques. This method represents a significant computational advantage, reducing processing times by orders of magnitude.

High-contrast, High-angular-resolution Optical Speckle Imaging: Uncovering Hidden Stellar Companions

We explore the possibility of detecting very faint, very close-in stellar companions using large aperture ground-based telescopes and the technique of optical speckle imaging. We examine the state of high-angular-resolution speckle imaging and contrast levels being achieved using current speckle cameras on the Gemini 8 m telescope. We then explore the use of the modern image reconstruction technique—multiframe blind deconvolution (MFBD)—applied to speckle imaging from the Gemini 8 m telescope. We show that MFBD allows us to measure the flux ratio of the imaged stars to high accuracy and the reconstructed images yield higher precision astrometry. Both of these advances provide a large refinement in the derived astrophysical parameters compared with current Fourier techniques. MFBD image reconstructions reach contrast levels of ∼5 × 10−3, near the diffraction limit, to ∼10−4 about 1.″0 away. At these deep contrast levels with angular limits starting near the 8 m diffraction limit (∼20 mas), most stellar companions to a solar-like stars can be imaged in the optical to near-IR bandpass (320–1000 nm). “To Xanadu we go...”—adapted from S. T. Coleridge.

Improved High-resolution Fast Imager

The improved High-resolution Fast Imager (HiFI+) is a multiwavelength imaging filtergraph, which was commissioned at the GREGOR solar telescope at Observatorio del Teide, Izaña, Tenerife, Spain, in March 2022 – followed by science verification in April 2022, after which it entered routine observations. Three camera control computers with two synchronized sCMOS and CMOS cameras each provide near diffraction-limited imaging at high cadence in six wavelength bands (Ca ii H at 396.8 nm, G-band at 430.7 nm, blue continuum at 450.6 nm, narrow- and broad-band Hα at 656.3 nm, and TiO bandhead at 705.8 nm). This unique combination of photospheric and chromospheric images provides “tomographic” access to the dynamic Sun and complements spectropolarimetric observations at the GREGOR telescope. High image acquisition rates of 50 and 100 Hz facilitate image restoration, where time series of restored images have a typical cadence of 6 and 12 s, which is sufficient to resolve the dynamics of the solar photosphere and chromosphere. In principle, all imaging channels can be restored individually using the speckle masking technique or multiframe blind deconvolution (MFBD). However, images recorded strictly simultaneously in the narrow-/broad-band Hα and the G-band/blue continuum channels can be pairwise subjected to multiobject multiframe deconvolution (MOMFBD) expanding the science capabilities of HiFI+. For example, the narrow-band (FWHM = 60 nm) Halle Hα Lyot filter isolates the Hα line core, which facilitates matching chromospheric fibrils and filamentary structures to photospheric bright points. Likewise, dividing G-band by blue continuum images enhances small-scale brightenings, which are often related to small-scale magnetic fields so that their evolution can be tracked in time. A detailed description of the improved high-cadence, large-format imaging system is presented and its performance is assessed based on first-light observations.

Multi-frame blind deconvolution and phase diversity with statistical inclusion of uncorrected high-order modes

Context. Images collected with ground-based telescopes suffer blurring and distortions from turbulence in the Earth’s atmosphere. Adaptive optics (AO) can only partially compensate for these effects. Neither multi-frame blind deconvolution (MFBD) methods nor speckle techniques perfectly restore AO-compensated images to the correct power spectrum and contrast. MFBD methods can only estimate and compensate for a finite number of low-order aberrations, leaving a tail of uncorrected high-order modes. Restoration of AO-corrected data with speckle interferometry depends on calibrations of the AO corrections together with assumptions regarding the height distribution of atmospheric turbulence. Aims. We seek to develop an improvement to MFBD image restoration that combines the use of turbulence statistics to account for high-order modes in speckle interferometry with the ability of MFBD methods to sense low-order modes that can be partially corrected by AO and/or include fixed or slowly changing instrumental aberrations. Methods. We modify the MFBD image-formation model by supplementing the fitted low-order wavefront aberrations with tails of random high-order aberrations. These tails follow Kolmogorov statistics scaled to estimated or measured values of Fried’s parameter, r0, that characterize the strength of the seeing at the moment of data collection. We refer to this as statistical diversity (SD). We test the implementation of MFBD with SD with noise-free synthetic data, simulating many different values of r0 and numbers of modes corrected with AO. Results. Statistical diversity improves the contrasts and power spectra of restored images, both in accuracy and in consistency with varying r0, without penalty in processing time. Together with focus diversity (FD, or traditional phase diversity), the results are almost perfect. SD also reduces errors in the fitted wavefront parameters. MFBD with SD and FD seems to be resistant to errors of several percentage in the assumed r0 values. Conclusions. The addition of SD to MFBD methods shows great promise for improving contrasts and power spectra in restored images. Further studies with real data are merited.

SUTO-Solar Through-Turbulence Open Image Dataset.

Imaging through turbulence has been the subject of many research papers in a variety of fields, including defence, astronomy, earth observations, and medicine. The main goal of such research is usually to recover the original, undisturbed image, in which the impact of spatially dependent blurring induced by the phase modulation of the light wavefront is removed. The number of turbulence-disturbed image databases available online is small, and the datasets usually contain repeating types of ground objects (cars, buildings, ships, chessboard patterns). In this article, we present a database of solar images in widely varying turbulence conditions obtained from the SUTO-Solar patrol station recorded over a period of more than a year. The dataset contains image sequences of distinctive yet randomly selected fragments of the solar chromosphere and photosphere. Reference images have been provided with the data using computationally intensive image recovery with the latest multiframe blind deconvolution technique, which is widely accepted in solar imaging. The presented dataset will be extended in the next few years as new image sequences are routinely acquired each sunny day at the SUTO-Solar station.

TurbuGAN: An Adversarial Learning Approach to Spatially-Varying Multiframe Blind Deconvolution With Applications to Imaging Through Turbulence

We present a self-supervised and self-calibrating multi-shot approach to imaging through atmospheric turbulence, called TurbuGAN. Our approach requires no paired training data, adapts itself to the distribution of the turbulence, leverages domain-specific data priors, and can generalize from tens to thousands of measurements. We achieve such functionality through an adversarial sensing framework adapted from CryoGAN 1, which uses a discriminator network to match the distributions of captured and simulated measurements. Our framework builds on CryoGAN by (1) generalizing the forward measurement model to incorporate physically accurate and computationally efficient models for light propagation through anisoplanatic turbulence, (2) enabling adaptation to slightly misspecified forward models, and (3) leveraging domain-specific prior knowledge using pretrained generative networks, when available. We validate TurbuGAN on both computationally simulated and experimentally captured images distorted with anisoplanatic turbulence.

The Thresher: Lucky imaging without the waste

ABSTRACT In traditional lucky imaging (TLI), many consecutive images of the same scene are taken with a high frame-rate camera, and all but the sharpest images are discarded before constructing the final shift-and-add image. Here, we present an alternative image analysis pipeline – The Thresher – for these kinds of data, based on online multi-frame blind deconvolution. It makes use of all available data to obtain the best estimate of the astronomical scene in the context of reasonable computational limits; it does not require prior estimates of the point-spread functions in the images, or knowledge of point sources in the scene that could provide such estimates. Most importantly, the scene it aims to return is the optimum of a justified scalar objective based on the likelihood function. Because it uses the full set of images in the stack, The Thresher outperforms TLI in signal-to-noise ratio; as it accounts for the individual-frame PSFs, it does this without loss of angular resolution. We demonstrate the effectiveness of our algorithm on both simulated data and real Electron-Multiplying CCD images obtained at the Danish 1.54-m telescope (hosted by ESO, La Silla). We also explore the current limitations of the algorithm, and find that for the choice of image model presented here, non-linearities in flux are introduced into the returned scene. Ongoing development of the software can be viewed at https://github.com/jah1994/TheThresher.

Post-AO High-resolution Imaging Using the Kraken Multi-frame Blind Deconvolution Algorithm

Abstract In the context of extreme adaptive optics for large telescopes, we present the Kraken multi-frame blind deconvolution (MFBD) algorithm for processing high-cadence acquisitions, capable of providing a diffraction-limited estimation of the source brightness distribution. This is achieved by a data modeling of each frame in the sequence driven by the estimation of the instantaneous wave front at the entrance pupil. Under suitable physical constraints, numerical convergence is guaranteed by an iteration scheme starting from a compact MFBD, which provides a very robust initial guess that only employs a few frames. We describe the mathematics behind the process and report the high-resolution reconstruction of the spectroscopic binary α And (16.3 mas separation) acquired with the precursor of SHARK-VIS, the upcoming high-contrast camera in the visible for the Large Binocular Telescope.

Correction of atmospheric stray light in restored slit spectra

Context. A long-standing issue in solar ground-based observations has been the contamination of data due to stray light, which is particularly relevant in inversions of spectropolarimetric data. Aims. We aim to build on a statistical method of correcting stray-light contamination due to residual high-order aberrations and apply it to ground-based slit spectra. Methods. The observations were obtained at the Swedish Solar Telescope, and restored using the multi-frame blind deconvolution restoration procedure. Using the statistical properties of seeing, we created artificially degraded synthetic images generated from magneto-hydrodynamic simulations. We then compared the synthetic data with the observations to derive estimates of the amount of the residual stray light in the observations. In the final step, the slit spectra were deconvolved with a stray-light point spread function to remove the residual stray light from the observations. Results. The RMS granulation contrasts of the deconvolved spectra were found to increase to approximately 12.5%, from 9%. Spectral lines, on average, were found to become deeper in the granules and shallower in the inter-granular lanes, indicating systematic changes to gradients in temperature. The deconvolution was also found to increase the redshifts and blueshifts of spectral lines, suggesting that the velocities of granulation in the solar photosphere are higher than had previously been observed.

Atmospheric turbulence characterization through multiframe blind deconvolution.

Characterization of turbulence in the atmosphere and mitigation of its effects in optical systems are important capabilities in both commercial and military applications. We present an image processing approach that jointly characterizes the magnitude of turbulence in the atmosphere and mitigates the adverse effects imposed on optical imaging systems. The magnitude of turbulence is measured indirectly through a series of image frames in terms of the atmospheric coherence length. We believe the results demonstrate the utility of the approach on both simulated and experimental data.

Rotated rectangular aperture imaging through multi-frame blind deconvolution with Hyper-Laplacian priors.

Rotated rectangular aperture imaging has many advantages in large aperture telephoto systems due to its lower cost and lower complexity. This technology makes it possible to build super large aperture telescopes. In this paper, we combine the ideas of deblurring with rotated rectangular aperture imaging and propose an image synthesis algorithm based on multi-frame deconvolution. In the specific reconstruction process, Hyper-Laplacian priors and sparse priors are used, and an effective solution is developed. The simulation and real shooting experiments show that our algorithm has excellent performance in visual effect and objective evaluation. The synthetic images are significantly sharper than the results of the existing methods.

Learning to do multiframe wavefront sensing unsupervised: Applications to blind deconvolution

Context. Observations from ground-based telescopes are severely perturbed by the presence of the Earth’s atmosphere. The use of adaptive optics techniques has allowed us to partly overcome this limitation. However, image-selection or post-facto image-reconstruction methods applied to bursts of short-exposure images are routinely needed to reach the diffraction limit. Deep learning has recently been proposed as an efficient way to accelerate these image reconstructions. Currently, these deep neural networks are trained with supervision, meaning that either standard deconvolution algorithms need to be applied a priori or complex simulations of the solar magneto-convection need to be carried out to generate the training sets. Aims. Our aim here is to propose a general unsupervised training scheme that allows multiframe blind deconvolution deep learning systems to be trained with observations only. The approach can be applied for the correction of point-like as well as extended objects. Methods. Leveraging the linear image formation theory and a probabilistic approach to the blind deconvolution problem produces a physically motivated loss function. Optimization of this loss function allows end-to-end training of a machine learning model composed of three neural networks. Results. As examples, we apply this procedure to the deconvolution of stellar data from the FastCam instrument and to solar extended data from the Swedish Solar Telescope. The analysis demonstrates that the proposed neural model can be successfully trained without supervision using observations only. It provides estimations of the instantaneous wavefronts, from which a corrected image can be found using standard deconvolution techniques. The network model is roughly three orders of magnitude faster than applying standard deconvolution based on optimization and shows potential to be used on real-time at the telescope.

Improving closely spaced dim object detection through multiframe blind deconvolution of near stellar neighbourhoods

A new iterative algorithm is proposed to improve the detection of dim stellar objects that are in the neighbourhood of a bright object, using short-exposure images. This method separates data functions into the primary bright object function, the neighbourhood system function, and the background function. This approach uses the principles of the Expectation-Maximization algorithm with the Gerchberg-Saxton phase retrieval algorithm to overcome the image degradation caused by the photon counting noise from the charge-coupled devices and the turbulent atmospheric conditions. The performance of this new neighbourhood system algorithm is compared with that of the multiframe blind deconvolution algorithm, using laboratory data and computer-simulated data. This paper provides an improved technique to image closely spaced dim objects.

Machine learning for quality assessment of ground-based optical images of satellites

Astronomical images collected by ground-based telescopes suffer from degradation and perturbations attributed to atmospheric turbulence. We investigate the application of convolutional neural networks (CNNs) to ground-based satellite imaging to address two existing problems. First, the multiframe blind deconvolution (MFBD) algorithms that can extract well-resolved images from these degraded data frames are computationally expensive, requiring supercomputing infrastructure for relatively fast performance and currently cannot be done in real time. Because of this, it is difficult to optimize collection parameters to maximize the likelihood of producing a resolved image with MFBD. Second, the space-object National Imagery Interpretability Rating Scale (SNIIRS) allows human analysts to provide a quantitative score of image quality based on identification of target features. It is naturally difficult to automate this scoring process, not only because the scale is based on identifiable features but also because the images may be in an almost-resolved image quality regime that is difficult to handle for traditional computer vision techniques. For both applications, we present our results using CNNs on data collected at the Maui Space Surveillance Site as well as a new synthetic dataset we introduce containing over a million SNIIRS rated pairings of perturbed and pristine ground-based satellite images.

Real-time image restoration for space-object imaging.

Military applications such as optical space surveillance and civilian applications such as astronomical imaging often require adaptive optics to compensate images of distant objects that are dynamically blurred by atmospheric turbulence. Many factors prevent adaptive optics (AO) from restoring a fully diffraction-limited image quality. Accordingly, restoration methods such as blind deconvolution and contrast enhancement are applied to further improve such imagery. Sometimes, the restoration must take place with low-latency and real-time frame rates because video imagery needs to be viewed promptly. This paper describes a procedure for conducting multi-frame blind deconvolution on experimental AO-compensated imagery in real time. In the procedure, registration and windowing enabled deconvolution, and subsequent enhancements improved the visibility of object features for visual assessment. This process features "multi-frame online blind deconvolution," which is a modification of thepreviously published "online blind deconvolution." This modified algorithm jointly processes multiple frames simultaneously, making it a true multi-frame, blind deconvolution method. The new method was tested on simulated and experimental imagery. The full procedure was implemented on a workstation with a low-end graphics processing unit, and timing tests were evaluated to estimate execution times.

High-Fidelity Imaging Using Compact Multi-Frame Blind Deconvolution

Multi-frame blind deconvolution (MFBD) has been a cornerstone for ground-based space situational awareness of near-Earth satellites since the early 2000’s. In 2011 a variation of the classic MFBD algorithm was introduced that required solving for fewer variables than in the classic algorithm, but which still used all the available data to constrain the solution. The initial application of the new approach, referred to as compact multi-frame blind deconvolution (CMFBD), was found to be significantly faster than MFBD, and showed an indication that it may be able to provide restorations of higher quality, i.e. fewer artifacts. Since its introduction, the CMFBD approach has become the foundation of several MFBD-based algorithms that have been developed for applications such as high-accuracy wave front sensing from image plane data, and imaging through strong turbulence: both of which contribute to space situational awareness by increasing the area of sky available for surveillance. Here we show that the performance of the CMFBD approach can be improved through the addition of a new ”internal consistency” constraint on the estimated point-spread functions.

Multi‐frame blind deconvolution of atmospheric turbulence degraded images with mixed noise models

This paper proposes a mixed noise model and uses the multi-frame blind deconvolution to restore the images of space objects under the Bayesian inference framework. To minimize the cost function, an algorithm based on iterative recursion was proposed. In addition, three limited bandwidth constraints of the point spread functions were imposed into the solution process to avoid converging to local minima. Experimental results show that the proposed algorithm can effectively restore the turbulence degraded images and alleviate the distortion caused by the noise.

Randomized apertures: high resolution imaging in far field.

We explore opportunities afforded by an extremely large telescope design comprised of ill-figured randomly varying subapertures. The veracity of this approach is demonstrated with a laboratory scaled system whereby we reconstruct a white light binary point source separated by 2.5 times the diffraction limit. With an inherently unknown varying random point spread function, the measured speckle images require a restoration framework that combine support vector machine based lucky imaging and non-negative matrix factorization based multiframe blind deconvolution. To further validate the approach, we model the experimental system to explore sub-diffraction-limited performance, and an object comprised of multiple point sources.

Parallel blind deconvolution reconstruction using phase diversity for three-dimensional imaging

Phase diversity (PD) is the special case of multiframe blind deconvolution algorithm, where an object is observed with a stack intentionally defocused image. The known defocused information between phases is used to estimate the object and aberration function. This paper presents a parallel three-dimensional (3-D) wavefront aberration estimation using blind deconvolution. We used multiplane PD for widefield fluorescence microscopy images of 3-D objects. Parallel multiplane PD yields a speedup factor of 4.33 with respect to the counterpart sequential algorithm for an image size of 256×256. We used the outcome of the PD algorithm to generate 3-D point cloud data from images from one view.

The Second Flight of the Sunrise Balloon-borne Solar Observatory: Overview of Instrument Updates, the Flight, the Data, and First Results

Abstract The Sunrise balloon-borne solar observatory, consisting of a 1 m aperture telescope that provides a stabilized image to a UV filter imager and an imaging vector polarimeter, carried out its second science flight in 2013 June. It provided observations of parts of active regions at high spatial resolution, including the first high-resolution images in the Mg ii k line. The obtained data are of very high quality, with the best UV images reaching the diffraction limit of the telescope at 3000 Å after Multi-Frame Blind Deconvolution reconstruction accounting for phase-diversity information. Here a brief update is given of the instruments and the data reduction techniques, which includes an inversion of the polarimetric data. Mainly those aspects that evolved compared with the first flight are described. A tabular overview of the observations is given. In addition, an example time series of a part of the emerging active region NOAA AR 11768 observed relatively close to disk center is described and discussed in some detail. The observations cover the pores in the trailing polarity of the active region, as well as the polarity inversion line where flux emergence was ongoing and a small flare-like brightening occurred in the course of the time series. The pores are found to contain magnetic field strengths ranging up to 2500 G, and while large pores are clearly darker and cooler than the quiet Sun in all layers of the photosphere, the temperature and brightness of small pores approach or even exceed those of the quiet Sun in the upper photosphere.